From Faults to Features: Pretraining to Learn Robust Representations against Sensor Failures
Authors: Jens U. Brandt
Noah C. Pütz
Marcus Greiff
Thomas Jonathan Lew
John Subosits
Marc Hilbert
Thomas Bartz-Beielstein
Venue: NeurIPS 2025 poster ()
Keywords: Robustness
Pretraining
Masking
Self-Supervised Learning
Representation Learning
Sensor Failures
Primary area: deep_learning
TL;DR: A pretraining scheme to learn representations that are robust to diverse sensor failures.

Abstract: Machine learning models play a key role in safety-critical applications, such as autonomous vehicles and advanced driver assistance systems, where their robustness during inference is essential to ensure reliable operation. Sensor faults, however, can corrupt input signals, potentially leading to severe model failures that compromise reliability. In this context, pretraining emerges as a powerful approach for learning expressive representations applicable to various downstream tasks. Among existing techniques, masking represents a promising direction for learning representations that are robust to corrupted input data. In this work, we extend this concept by specifically targeting robustness to sensor outages during pretraining. We propose a self-supervised masking scheme that simulates common sensor failures and explicitly trains the model to recover the original signal. We demonstrate that the resulting representations significantly improve the robustness of predictions to seen and unseen sensor failures on a vehicle dynamics dataset, maintaining
strong downstream performance under both nominal and various fault conditions. As a practical application, we deploy the method on a modified Lexus LC 500 and show that the pretrained model successfully operates as a substitute for a physical sensor in a closed-loop control system. In this autonomous racing application, a supervised baseline trained without sensor failures may cause the vehicle to leave the track. In contrast, a model trained using the proposed masking scheme enables reliable racing performance in the presence of sensor failures.

From Faults to Features: Pretraining to Learn Robust Representations against Sensor Failures
Jens U. Brandt Noah C. Puetz
Mareus Greiff?
Thomas Lew?
John Subosits²
Marc Hilbert
Thomas Bartz-Beielstein'
TH Köln
Toyota Research Institute
Toyota Gazoo Racing Europe
Abstract
Machine learning models play key role in safety-critical applications, such tonomous vehicles and advanced driver assistance systems, where their robustness during inference is essential to ensure reliable operation. Sensor faults, however, can corrupt input signals, potentially Icading to severe model failures that com- promise reliability. In this context, pretraining emerges powerful approach for learning expressive representations applicable to various downstream tasks. Among existing techniques, masking represents promising direction for learning representations that are robust to corrupted input data. In this work, we extend this concept by specifically targeting robustness sensor outages during pretraining. We propose self-supervised masking scheme that simulates common sensor fail- ures and explicitly trains the model to recover the original signal. We demonstrate that the resulting representations significantly improve the robustness of predic- tions seen and unseen sensor failures on a vehicle dynamics dataset, maintaining strong downstream performance under both nominal and various fault conditions. As a practical application, we deploy the method on modified Lexus LC 500 and show that the pretrained model successfully operates as substitute for physical sensor closed-loop control system. this autonomous racing application, a supervised baseline trained without sensor failures may cause the vehicle to leave the track. In contrast, model trained using the proposed masking scheme enables reliable racing performance in the presence of sensor failures.
Input Signal
Pretrained Baseline
Our Approach
[Sensor fallure
40-
Feature
-20
Original signal Blased signal
-2
Giroud truth
Ground truch Estimate
Estimmate
20
60
80
Time (5)
Time (s)
Time (s)
Figure 1: Although traditional pretraining with masking produces representations suitable for down- stream tasks, these representations can lack robustness to common sensor failures like signal offsets. In contrast, our proposed pretraining scheme yields representations that are robust to sensor faults.
"Contact: jens brandrandi @ th-koeln.de
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Introduction
Sensor failures pose significant challenge to the reliable deployment of machine learning models in safery-critical domains, such as autonomous driving and robotics [Durlik et al. 2024 Ji et al., [022] Failures can stem from hardware malfunctions, challenging environmental conditions, or ma malicious attacks [Schober et al. 2022 Balaban et al., 2009]. To improve reliability in such circumstances, any model ingesting sensor data needs to be robust distribution shifts [Braiek and Khomh 2024
Freiesleben and Grote 2023
Self-supervised learning (SSL) offers promising solution to this problem, enabling the extraction of meaningful representations from unlabeled data. The learned representations can be used for various downstream tasks, such as classification, regression, or anomaly detection [Zhang et al. 2024| Zerveas et al. 2021]. Masked Image Modeling (MIM) is a major family of SSL methods that aims to train generative models capable of predicting original image from corrupted or masked version [Balestriero et al. 2023]. masked modeling for time series applications, common approaches involve replacing randomly selected segments of the input sequence with constant (c.g., zero) or a lcarnable value, and subsequently training the model to reconstruct the original unmasked input [eervass et al., 2022]] Goswami et al. 2024 Dong et al. 2023 Nie et al.) 2022). This masking strategy encourages the model to learn robust representations that effectively capture the underlying patterns and structures in the data, even when missing significant portions of the inputs. This body of work suggests that masked modeling is suitable for learning representations robust to sensor outages. However, while [Bllaban et al. [2009] and Jesus et al. [2017) outline sensor failure modes that are predominantly characterized by additive noise and constant or time-dependent offsets, these differ markedly from the masks typically used in masked modeling. This suggests that sensor outages remain out-of-distributic even after pretraining, potentially leading to performance degradation.
Motivated by this discrepancy and the need for robust sensing in safety-critical applications, we pro- pose a novel self-supervised masking scheme for robust representation learning that explicitly targets sensor failure robustness by incorporating three common outage scenarios into the reconstruction objective. We further systematically analyze the effectiveness of our method on a dataset from the vehicle dynamics domain and demonstrate it in real-world application involving autonomous racing.
2 Related work
Early foundational work by Vincent et al. [2010) introduces stacked denoising autoencoders, demon- strating how models pretrained to reconstruct inputs subject to noise using a local denoising criterion can learn robust and invariant features. Building upon this concept, more recent work the vision domain by He et al.
[022) introduces Masked Autoencoder (MAE), which is trained to reconstruct original inputs from partially masked data, encouraging the model to learn robust representations.
Zerveas et al. 2022 proposes a transformer-based self-supervised framework specifically designed for multivariate time series. Their approach demonstrates improved performance in downstream classification and regression tasks by employing a masked input modeling strategy, where portions of the input are masked (set to zero). Yuan et al. 2024 explores multi-task SSL using large unlabeled accelerometer datasets from the UK Biobank. Their methodology incorporates multiple common time-series augmentation techniques, such as reversing, permuting. and time- taae-warping. alongside a multi-head architecture for pretraining. Their findings highlight the strong generalization capabilities of pretrained deep convolutional neural networks when finetuned on diverse activity recognition benchmarks. Narayanswamy et al. [2024) introduces LSM, multi-modal representation learning model for wearable sensor data. This study of the LSM explores scaling laws and demonstrates superior few-shot learning performance compared to established benchmarks by combining a vision transformer architecture with various masking strategies.
To further evaluate the reliability of deep learning methods beyond standard benchmarks, Hendrycks and Dietterich [2019) provides a dataset for common image corruptions. Building on this, Hendrycks et al. [2019a b) show that an additional self-supervised loss significantly enhances model resilience against common input corruptions, adversarial attacks, and label corruptions. They furthermore show improved robustness for the latter two cases using adversarial pretraining following Madry et al. [2018), with a model pretrained on ImageNet using adversarial training and then fine-tuned on CIFAR- 10 and CIFAR-100.
2

---

2a
2b
3
6a
6b
7a
7b
Figure 2: Sensor failure modes. The faulty sensor output I (red) and underlying signal x (gray).
Alternative approaches include data augmentation and domain randomization, which both expose models varied input conditions during training. Adversarial training represents specialized form of data augmentation that enhances resilience by training on adversarially generated examples designed
to maximize prediction errors [Shorten and Khoshgoftaar 2019 Tobin et al., [017]. Imputation strategies address missing or corrupted data by reconstructing absent values to support robust inference [Wang et al. 2024). While these approaches advance robustness across various domains, real-world sensor failures often exhibit structured patterns, such as persistent biases, drifts, or hard faults, that differ from the random adversarial corruptions typically considered.
These findings motivate the design of pretraining scheme that explicitly prepares models for real- world sensor failures. To date, the impact of pretraining common input corruptions, especially in the domain of multivariate time series data, remains underexplored. In this setting, we aim to train models for accurate predictions under realistic sensor failures and input disturbances (see Figure
3 Sensor failure modes and robustness via reconstruction
3.1 Sensor failure modes
Sensor faults manifest unexpected deviations in measurements [Balaban et al. 2009]. These deviations reflect distinct failure modes independent of their physical root causes or application domain. systematic classification of these modes is essential for fault diagnosis and mitigation. Following the taxonomy of Balaban et al. [2009] (1,2.7) and JJsus et al. [217]] (3-6), we define the principal failure modes relative to nominal sensor model representing normal operation: x Fnom + Cnoms where Trom is the true signal and 6nom is sensor noise. We define various fault functions D(-) that act on produce the faulty sensor signal, Xt, under different failures (see Figure 2
1. Bias. An offset from the true measurement: D(x) x + B, where is a constant offset.
2. Drift. An offset from the true measurement: D(x) + 3(t) where the time-varying offset ((()
is either (a) linear or (b) nonlinear
3. Hard fault: The sensor measurement is stuck constant value C: D(x)
4. Noise, Additive zero-mean noise: ((x) + e, with Var(e) Var(Cuom).
5. Outliers. Occasional deviations from the true measurement random times: D(x) x (+))t)) where (t) E {0, 1} indicates outlier presence and the 6(t) is sampled uniformly over [t, u].
Trimming. Signal values are restricted [1 u] and, for values outside these bounds, are either (a) fixed at the boundary or (b) vary proportionally with their deviation from the boundary:
+ a(x -
if <1
((x)
u a(x u) u otherwise
We consider (a) hard clamping with 0, and (b) controlled dampening strength with 0 < 1. 7. Scaling, A multiplicative error affecting signal amplitude: D(x) aal where (a) the gain is constant, or (b) (() a function of time
These sensor failure modes can occur due to manufacturing errors, wear and tear with long-term usage, incorrect calibration, mishandling [Balaban et al. 2009), and they can have detrimental or even catastrophic effects on the performance machine learning models, especially when deployed in real-world robots and autonomous vehicles.

---

3.2 Robustness via reconstruction
Reconstruction--asee SSL involves training models to reconstruct original input data that has been partially masked or corrupted. For language and image data, common tasks include predicting missing words or pixels, image inpainting, and colorization (Balestriero et al., 2023] In multivariate time series, this typically involves masking intervals of the imput with zeros [Zerveas et al., (021] or learnable masks [Goswami et al. 2024]. Early SSL methods, such as denoising autoencoders
[Vineent et al. (210)] corrupt inputs by masking features and calculate the loss based on the full reconstructed imput. Grounded in the manifold hypothesis [Chapelle et al., 2006]-thne natural high-dimensional data reside near a non-linear low-dimensional manifold-thi corruption process pushes data away from the manifold, forcing the neural network project corrupted inputs back onto it by capturing essential input features (Vincent et al. 2010]. Modern SSL methods modify the reconstruction objective. Approaches like BERT [Devlin et al. 2019) and MAEs (He et al., 2022] compute the reconstruction loss solely on masked input regions. This design choice is deliberate: by not requiring the model to reproduce the already visible parts, it avoids wasting capacity on trivial copying, encouraging the model to focus on inferring the missing content from the context provided by the unmasked parts, emphasizing the learning of global feature dependencies.
We hypothesize that, although calculating the loss only on masked regions relaxes the requirement of exact reconstruction, models still implicitly project inputs onto the data manifold. This occurs because repeatedly training on diverse mask configurations fosters a universal representation that allows reconstructions coherent with the data's true structure. Following the argumentation of Vincent et al.) [2010), since the learned representation is robust to input masking by design, utilizing this training paradigm should enhance robustness to similar input perturbations in downstream tasks. Models trained with partial context learn stable patterns and dependencies, becoming resilient to variations or corruptions. Thus, they retain coherent internal representations under noisy or incomplete inputs, potentially benefiting real-world applications. However, the perturbations encountered in real-world scenarios, such as the diverse sensor failure modes described in Section [3.]] can differ substantially from the masks typically employed during pretraining, potentially limiting direct transferability.
4 Method
4.1 A self-supervised pretraining method to learn robust representations
We propose a pretraining scheme that simulates real-world sensor failures by introducing artificial faults to the input data and training the model reconstruct the original signal (see Figure 3 Following the sensor failure definitions in Section [3.1] we formulate the pretraining objective as a multi-task learning problem with n distinct masking functions {D. Formally, consider an input tensor X E RR SxF where B is the batch size, S is the sequence length, and F is the number of channels. We partition each batch X into n sub-batches { where each X; E IR BaaxXxx XSx has size Banh- B/ n and corresponds one of the n distinct masking functions D..
For each sample in sub-batch X;, we generate a binary mask tensor M, E (0, 1) Bx indicating the indices to be masked. The masking process follows the geometric distribution sampling strategy from Zerveas et al. [202]]. controlled by two parameters: the overall masking ratio mr (proportion of total elements to mask per sample) and the average mask block length Im. This strategy generates contiguous masked spans, making reconstruction challenging via pure interpolation and promoting the learning of feature inter-dependencies. An element Mo 0 indicates that the b-th sample within sub-batch i is masked at timestep s and channel f. Once the mask tensor M, is generated, the corresponding sensor muilure-inspire masking strategy is applied directly to this mask. X; denotes the respective masked sub-batch, where the masked regions are corrupted according to the chosen strategy. The implemented masking strategies, which are applied after feature-wise z-score normalization of the input data, are detailed below.
Mean masking: Sets the values at the masked indices to the respective channel mean Co,5 2y resulting in a tensor C, E Bx (see Section 3.1 case 3). Since the input data X, undergoes whan 7-score normalization before masking, the mean of each channel is zero. Consequently, replacing masked values with μ is equivalent to setting them to zero. This simplification allows us to implement mean masking by directly multiplying the normalized input with the binary mask M,

---

Sample Mask Apply Sensor Fallures
Reconstruct Inpur
Split Batch
(X())
X
Bins
4(8.x)
bone
MAAA
Figure 3: Pretraining scheme: The input X is split into n sub-batches X, each associated with sensor failure mode Dj. For each sub-batch, binary mask identifies positions be corrupted according Dj. The encoder E maps the corrupted input X; latent representation, which is passed through the pretraining head P produce a reconstruction X,. The model trained to reconstruct the original input by minimizing the sum of reconstruction losses.
(where Mo...f O indicates masked position). Formally, the corrupted signal is given by
X, @ M,,
(1)
where @ denotes dlement-wise multiplication. This approach is commonly used in masked modeling for time series data and reflects a standard fallback in technical systems.
Bias masking: Introduces a constant offset to the masked regions (see Section 3.1 case 1). Specifi- cally, for each b-th sample and f-th feature, a bias value Co, f is drawn independently from uniform distribution Co,, ~ [[( ()) where is the lower bound of the uniform distribution and is the upper bound multiplied with the channel-wist standard deviation [f]- This sampled bias Co.
then replicated across S time steps to form the bias tensor C, E BB XSxP. Normalization eliminates the need for separate bounds in each channel. We let
X = = C, 0
where 1 is a tensor of ones matching M,'s dimensions, and the element- wise subtraction inverts M,. Each channel has single sampled offset, and M, ensures that these offsets are applied only to the masked positions. We assume that for short observed sequences, the bias error serves an effective proxy for drift, outliers, trimming, and scaling behaviors. For domains with lower sampling rates or longer sequences, including additional fault types in pretraining could be beneficial.
Noise masking: Adds random Gaussian noise the masked regions (see Section 3.1 4). Specifically, a zero-mean noise tensor C; E IR X is drawn independently for each element Co.e. (0, rot)) where is scaling factor for the channel-wise variance. The noise is applied as in Equation (
Loss calculation:
Once the sub-batches (X, ) with their respective masking corruptions prepared, they are sequentially processed by the model. The reconstruction loss (Mean Squared Error over masked elements) is defined
L
Σ
(Ixilb.as- (Xilb..r)
Mi
(b,s.f)EM,
where M, ((b,,, f) [Mr]..., 0} is the set of masked indices in M,, [Mal denotes its cardinality, and [X f is the reconstruction of the original value [X=]o.s,f the index (b, s, f).
given the masked input [X,]o. The implementation follows the masked reconstruction paradigm from
time series, vision, and language modeling [eerveas et al. 202 l He et al. 2022 Devlin et al.) 2019
Architecture: Our encoder-backbone is based on the transformer architecture described by Zerveas
al. 20021. The encoder E processes the corrupted input X,:
=EXX, 0z),
(4)
5

---

where z; E R Pus Xadddaaa is the latent representation that captures temporal dependencies and contextual information from the input, and F are the learnable parameters. dmedal is the dimension of the initial input projection. The sequence dimension of the latent representation is concatenated:
cat Co.cat(z.) Buss x (S-dnra))
(5)
This concatenated representation is then processed by the pretraining head P(), a multi-layer percep- tron with parameters Op, which performs the reconstruction by mapping the latent representation back into the input space
X; = P(z .p)).
(6)
4.2 Finetuning procedure for downstream use
For finetuning on a supervised task, we replace the pretraining head P with a finetuning head F (again a multi-layer-percepirom)) The encoder is then frozen; its parameters, 0g., are not updated during the finetuning procedure, Only the parameters Op of the finetuning head are trained. For a finetuning batch X E x where BF is the finetuning batch size, and associated labels the frozen encoder computes the latent representation
z = E 0g)
which is concatenated as in Equation ( and mapped to the target space
F(z 0a5 ,pp).
(8)
The finetuning objective is then
(1)))))))) w):
PFT
(9)
where ((.-) denotes a mean squared error loss function. This procedure leverages the robust representations learned during pretraining while adapting the finetuning head to the downstream task.
5 Sensor failure robustness results
We evaluate method on the problem of vehicle sideslip angle (VSA) estimation, as accurate VSA estimation is necessary for vehicle stability and control [Liu et al. 2020). Modern advanced driver-assistance systems, such as Electronic Stability Control, utilize the VSA to adjust brake forces at each wheel, thereby mitigating oversteer and understeer and enhancing overall performance and safety [Liu al. 2020 Chindamo al.) 2018 Bonfitto al., 2020). However, systems that directly measure the VSA can be prohibitively expensive for consumer vehicles, necessitating alternative estimation methods. Virtual sensing, which leverages machine learning models to estimate the VSA from other sensors, such as wheel speed sensors in the car, offers a cost-effective solution. Yet virtual sensing is prone sensor failures, motivating the development of robust models. VSA estimation involves heterogeneous sensor inputs (vehicle control signals, wheel speeds, and accelcrations) with diverse statistical properties. Since similar modalities are used in robotics and aviation, the study of VSA estimation may have broader implications for robust estimation beyond the domain of driving.
The VSA is the angle between the vehicle's 's direction of travel and the heading angle of the vehicle Liu et al. 2020), expressed as a function of the vehicle's 's longitudinal (uz) and lateral (vyy) velocities:
Uty
arctan
(10)
Ux
In practice, the VSA can be estimated directly or computed from the estimated lateral velocities in combination with the typically more accessible longitudinal velocities. Both approaches are common in virtual sensing, depending on the available sensor data and model design [Liu et al. 2020]. In the following, we first assess the effectiveness our approach on a public dataset augmented with simulated sensor failures and then validate it in real-world autonomous racing scenario.

---

5.1 Simulation of sensor failures
The REVS Program Vehicle Database Kegelman et al. 2016a b comprises measurements collected from expert drivers operating under real-world racing conditions representative of extreme vehicle dynamics and operating scenarios. We evaluate the proposed method under several simulated sensor outages an unseen track) and compare its performance to classical masking scheme and non-pretrained baseline, respectively. Details on the dataset, features, and splitting given in Section
tracks, providing data
test (one from the track and another from
yawRate
ayCG
handwheelAngle
engineSpeed
throttle
oxCG
c
chassisAcceIFR
chassisAcceIRL
chassisAcceIRR
no_ failure
Same Track Bias
Same Track Mean
Same Track Noise
Same Track
Scaling
yawRate
ayCG
handwheelAngle
engineSpeed
throttle
brake
axCG
chassisAcce/
chassisAccolFR
chassisAcceIRL
chassisAcceIRRR
no failure
New Track
Blas
New Track Mean
New Track Noise
New Track Scaling
0.0
0.6
0.4
0.0
0.2
0.0
0.6
MSk
MSL
NSE
MSE
Baseline (no pretraining)
Baseline (mean pretraining)
Ours (mean, bias, noise pretraining)
Figure 4: Lateral velocity MSE under individual sensor failures on seen and unseen tracks for the non-pretrained baseline (purple), mean-only pretrained baseline (red), and multi-mask approach (blue). Bootstrapped means (200 resamples) with 95% confidence intervals on the test set.
We pretrain the model using 11 sensor signals with fixed sequence length of 20 and 20 Hz sampling rate. To align the pretraining task close to actual sensor failures, we set the average mask length Im to 20 ma-mmiii sufficient information in the remaining channels for reconstruction. We set the parameters of the noise mask μy and 1.0 and the parameters of the bias mask to -3 and Section T. Further implementation details can be found in Section B As downstream task, we fine-tune an estimator that learns estimate the lateral velocity.
entire channel average and masking ratio of 0.1 to retain
(see
We evaluate the robustness using the three pretraining failure modes (bias: one-standard-deviatte offsst; noise: zero-mean, standard deviation of the respective feature times 0.4; hard faults: channel mean replacement), an unseen scaling failure (a = 2) to test generalization to novel sensor faults, and nominal conditions (see Section 3.1). Each error is applied separately to every channel. We report the lateral velocity MSE (Figure 4) with: model pretrained on all three corruptions (Mean, Bias, and Noise); non-pretrained baseline; and mean-only pretrained baseline (traditional zero masking).
Nominal conditions: Without sensor corruptions, our method improves downstream performance on the seen track and matches the performance of the non- pontrrrtda baseline the unseen track.

---

Seen failures: Under noise corruptions, pretraining yields subtle performance gains. For sensor outages simulated by mean-value replacement, the model pretrained solely on mean masking exhibits the best robustness. However, its robustness against bias errors on an unseen track degrades below the non-pretrained baseline. Our approach achieves the strongest robustness to bias failures as evidenced by reduced error magnitude and variance. On an unseen track, the performance of our approach is largely consistent with those on the seen track.
Unseen failure: The strong estimation performance under the unseen scaling error suggests that the model is robust to sensor faults that were not used during pretraining, indicating that the learned representations generalize to out-of- -f--of---itriutiinn failure modes. Evaluations on all remaining failure modes presented in Section [..] and detailed in Section I show similar results, with our approach consistently improving robustness.
Ablations: Varying the masking combinations confirms that incorporating bias masking is critical for
robustness to both bias and scaling errors, with our approach achieving a good trade-off (Section C).
When subjecting models to increasingly severe sensor failures, our model remains robust across wide range of bias, noise, and scaling strengths, consistently outperforming the baselines (Section D) We also note improved performance over baselines with simultaneous sensor failures, as detailed in
Section [ Repeating training across ten random seeds demonstrates that our scheme is consistent, whereas the performance of a mean-only baseline varies markedly on unseen failure types (Section Finally, we observe that larger models tend to enhance robustness across failure modes (Section Additional experiments indicate that, compared a state-of-the-as baseline (SimMTM), our approach provides enhanced robustness under sensor failure conditions (Section[K) K). Tested in a different domain, our method also improves reliability in environmental sensing tasks (Section M).
5.2 Enhanced robustness for closed-loop control
To show the benefits of the proposed pretraining pro- cedure, we evaluate it in a closed-loop autonomous racing scenario using the controller described in [Lw al. 2024]. This experiment shows how estimation eπrors propagate and impact the coupled controller- vehicle system in real-world application. A modi- fied 2019 Lexus LC 500 is driven on a skidpad, with our model estimating the VSA from other available sensors (for an overview of the available sensors in this experiment and implementation details, see Sec- tion[N] Nj. The estimated VSA is then used for feedback control. While racing autonomously along the planned trajectory shown in Figure 5 we inject
Figure 5: Nominal trajectory of the car (green) and virtual track limits (white).
artificial sensor outages and record the system response. The experiment is repeated with bascline model directly trained on the VSA estimation task without initial pretraining.
Open-loop tests indicate that VSA estimation is most sensitive to potential failures in the wheel speed, steering angle, yaw rate, and longitudinal velocity sensors. Thus, we compare our approach against the baseline by corrupting one wheel-speed sensor at time by applying heavy noise, bias, and hard faults (each channel set to its mean). Additional closed-loop experiments and details are Section
The first attempt with the baseline model fails, and the safety driver must intervene after a few seconds of autonomous driving. The second attempt is shown in Figure σ The pretrained model maintains a good VSA estimate throughout, resulting in a stable trajectory (see top-right subplot) even under challenging conditions. The bascline's VSA estimates under noise show occasional deviations with increased variance. Bias failure leads to significant offsse errors (see bottom-left subplot), whereas mean input errors have minimal effect, suggesting robustness such disturbances. Notably, the baseline's estimation leads to unstable trajectories (see top-left subplot) that cause violation of the virtual track boundaries. Additional experiments with outages the other sensors show similar behavior for the pretrained model (see Section O). One notable exception arises under ongitudinal-velocityy sensor outages: here the pretrained model also produces unstable trajectories, but the baseline's 's parallel open-loop estimations still degrade to greater extent. Our approach demonstrates significant improvements in robustness across a vast majority of tested failure cases, highlighting its potential for reliable deployment in real-world applications.
8

---

Baseline (no pretraining)
Ours (mean, bias and noise pretraining)
sideslip (rad) wheelspeeds (rad/s)
80
(FL)
60
(RR)
(RR)
(AI)
(AI)
10
20
50
20
30 40
50
0.2
0.0
-0.2
-0.4
time (s)
time (s)
Figure 6: Robust virtual sensing in closed-loop control. Top: full driven trajectory (car image illus- trative, indicating driving direction). Middle: perturbed (Ø) vs. true (I) wheelspeed measurements.
Bottom: sideslip estimate (i) under corruptions true VSA (z).
Limitations and future work
The results demonstrate that the proposed masked pretraining significantly improves robustness against common sensor failures for driving applications. However, while the model extrapolates beyond the masks used during pretraining (see Section (.1). studying more complex mode config- urations (e.g., coupled errors) and assessing its performance in additional domains would broaden the applicability of the proposed method. The experiments show that certain sensor failures, such as errors in the longitudinal velocity, are more detrimental to closed-loop performance than others. limitation this work is that all failure modes across all channels are weighted equally, but we speculate that this can be addressed by re-weighting of the loss. Finally, as the method employs multiple mask- based tasks, interference between the gradient updates could lead to poor convergence during training. Leveraging advanced multi-task learning techniques, such Yu et al.) 2020), may mitigate this and improve performance, and will be explored in future work.
Conclusion
In this work, we introduce novel self-supervised masking scheme designed strengthen model resilience by directly incorporating common sensor failure modes into the learning objective. By training models reconstruct signals corrupted by realistic fault simulations, we foster representations significantly more robust to sensor outages than those learned via traditional masking. We demonstrate the effectiveness this approach through extensive validation on a vehicle dynamics dataset and, through successful deployment as virtual sensor in closed-loop control of a real autonomous vehicle operating at limits. The results demonstrate significantly improved robustness against various sensor failures, including generalization to a fault type unseen during pretraining, We believe that by directly addressing this critical vulnerability, our work contributes the development of resilient deep learning systems, which are essential for robust deployment in safety-critical applications.

---

Acknowledgements
This work is funded by the European Commission Key Digital Technologies Joint Undertaking Research and Innovation (HORIZOOKDFJJL2203--RRI))) the ShapeFuture project "Shaping the Future of EU Electronic Components and Systems for Automotive Applications".
under grant agreement No 101 139996,
References
lenkaran Arasaratnam and Simon Haykin. Cubature kalman filters. IEEE Transactions on automatic control, 54(6): 2009.
Edward Balaban, Abhinav Saxena, Prasun Bansal, Kai F. Goebel, and Simon Curran. Modeling, Detection, and Disambiguation of Sensor Faults for Acrospace Applications, IEEE Sensors Journal, 912)::19771111,, December 2009. ISSN 1558- 1748. doi: 10. 10.110//SENN200 2009 2030284. 200.0200844 URL htttps:/ leee.
Conference Name: IEEE
Sensors Journal.
Randall Balestriero, Mark Ibrahim, Vlad Sobal, Ari Morcos, Shashank Shekhar, Tom Goldstein, Florian Bordes, Adrien Bardes, Gregoire Mialon, Yuandong Tian, Avi Schwarzschild, Andrew Gor- don Wilson, Jonas Geiping, Quentin Garrido, Pierre Fernandez, Amir Bar, Hamed Pirsiavash, Yann LeCun, and Micah Goldblum. A Cookbook of Self-Supervised Learning, June 2023. URL www./ 12210 arXiv:2304. 12210 [cs].
Karl Berntorp, Marcus Greiff, and Stefano Cairano. Bayesian sensor fusion of gnss and cam- with outlier adaptation for vehicle positioning. In 2022 25th International Conference Information Fusion (FUSION), pages 1-8. IEEE, 2022.
Angelo Bonfitto, Stefano Feraco, Andrea Tonoli, and Nicola Amati. Combined regression and classification artificial neural networks for sideslip angle estimation and road condition identífica- tion. Vehicle System Dynamics, 58( 1):1166-17877 November Novembee 2020. ISSN 0042-3114. doi: 10. 1ttp://0000000000011113110110111 2019. 1645860
1080/004231
URL
Publisher: Taylor & Francis _eprint:
Houssem Ben Braick and Foutse Khomh. Machine Learning Robustness: A Primer, May 2024, URL httt://arxiv.. org hang/abs/2000 /b/4400002 arXiv:2404.00897 [es].
Olivier Chapelle, Alexander Zien, and Bernhard Schölkopf, editors. Semi-supervised learning. Adaptive computation and machine learning series. MIT Press, Cambridge, Massachusetts, 2006. ISBN 978-0-262-03358-9 118-00262--55899
Daniel Chindamo, Basilio Lenzo, and Marco Gadola. On the Vehicle Sideslip Angle Estimation: Literature Review of Methods, Models, and Innovations. Applied Sciences, 8(3):355, March 2018.
ISSN 2076-3417. doi: 10.3390/aps8000355. URL https: //w::///www... http:/wwmpppp...m/////// www. 2ttmm/27663411//////
355
Number: 3 Publisher: Multidisciplinary Digital Publishing Institute.
Jacob Devlin, Ming- Wei Chang. Kenton Lee, and Kristina Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. Jill Burstein, Christy Doran, and Thamar Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pages 4171-41886 Minneapolis, Minnesota, June 2019. Association for org/
Computational Linguistics. doi: 18.18653/v 1/113//1//1111/// 1423. URL TRLLtttps///aclattooooo #19-14233
Jiaxiang Dong. Haixu Wu, Haoran Zhang. Zhang, Jianmin Wang, and Mingsheng Long. SimMTM: A Simple Pre- Training Framework for Masked Time-Series Modeling, November 2023, URL https://openreviek
Irmina Durlik, Tymoteusz Miller, Ewelina Kostecka, Zenon Zwierzewicz, and Adrianna Łobodzińska. Cybersecurity in Autonomous Vehicles-AA Are We Ready for the Challenge? Electronics, 13 (13):2654, January 2024. ISSN 2079-9292. doi: 10.3390/electronies133 326254.. URL https:
10

---

Dumitru Erhan, Yoshua Bengio, Aaron Courville, Samy Bengio. Why Does Unsupervised Pre-training Help Deep Learning? Journal of Machine Learning Research, 11(19):625-660) 2010. ISSN 1533-7928. URL. URI.htpp: //jmlr org/ papers/
Manzagol, Pascal Vincent, and
v11/erhan10a.. .html
Timo Freiesleben and Thomas Grote. Beyond generalization: theory of robustness in machine learn ing. Synthese, 109, September 2023, ISSN 1573-0964, doi:
URL https://doi....... wtt.:////...00/11222000002111110 021-1222922220304--222 04334-9
Mononito Goswami, Konrad Szafer, Arjun Choudhry, Yifu Cai, Shuo Li, and Artur Dubrawski. MOMENT: A Family of Open Time-series Foundation Models. June 2024.
URL https::/
Kaiming Hc, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollar, and Ross Girshick. Masked Autoencoders Are Scalable Vision Learners. In 2022 IEEE/CVF Conference Computer Vision and Puttern Recognition (CVPR), pages 15979-15988, New Orleans, LA, USA, June 2022. IEEE,
ISBN 978-1-6654- doi: 10.1 N0.10099CVPP62268222222.222
URL/
leee. (eee.org/bbcummat/98773
Dan Hendrycks and Thomas Dietterich. Benchmarking Neural Network Robustness to Com- mon Corruptions and Perturbations, March 2019. http:ttp://arxiii http://axxiv..ggab//////.ss 12261
arXiv:1903.1223 12266 [cs].
Dan Hendrycks, Kimin Lee, and Mantas Mazeika. Using Pre-Training Can Improve Model Ro- bustness and Uncertainty. In Proceedings of the 36th International Conference on Machine Learning, pages 2712-2721, PMLR, May 2019a. URL https:///prcceeddnn/ mlr aalrpress.
v97 h97/hendrycks19a. html ISSN: 2640-3498.
Dan Hendrycks. Mantas Mazeika, Saurav Kadavath, and Dawn Song. Using Self- Supervised Learning Can Improve Model Robustness and Uncertainty. In Ad- vances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019b. URL (tB.3/77CCCc01111172118888888888 Abstract. ctttsract.htmlll
Gonçalo Jesus, António Casimiro, and Anabela Oliveira. A Survey on Data Quality for Dependable Monitoring in Wireless Sensor Networks. Sensors, 17(9):2010, September 2017. ISSN 1424-8220. doi: (03390/s/7092011
BtL/shts::///tt Publisher: Multidisciplinary Digital Publishing Institute.
http://c...c//1440200///////0000 Number: 9
Tianchen Ji, Arun Narenthiran Sivakumar, Girish Chowdhary, and Katherine Driggs-Campbell. Proactive Anomaly Detection for Robot Navigation With Multi-Sensor Fusion. IEEE Robotics and Automation Letters, 7(2):4975 4982, April 2022. ISSN 2377-3766. doi: 10.1109/LRA.2222.. 3153989. URL
ieee. wwwp/orrgggo.t.oooo/oo/o document//7709337
John Kegelman, Lene Harbott, Jackie Liao, and Christian Gerdes. 2013 Monterey Motorsports Reunion, 2016a. URL wtt.ppttps://purl.saaaai stanforrdstrrrrrrrrorrooooo www.:ooor....uueedeeeeee edu/tt1033r6664
John C. Kegelman, Lene K. Harbott, Jackie Liao, and J. Christian Gerdes. 2013 Targa Sixty-Six, 2016b. URL htttp:://purl.staafffffe wtt.:/frrr.eeee//11gg02e2
Thomas Lew, Marcus Greiff, Franck Djeumou, Makoto Suminaka, Michael Thompson, and John Subosits. Risk-Averse Model Predictive Control for Racing in Adverse Conditions, October 2024.
URL http::/atpiii...e/////aaaa org/ abs//a10.... 17183 1rX1v:2410.17183 [cs].
Liam Li, Kevin Jamieson, Afshin Rostamizadeh, Ekaterina Gonina, Jonathan Ben-tzur, Moritz Hardt, Benjamin Recht, and Ameet Talwalkar. A System for Massively Paral- lel Hyperparameter Tuning. Proceedings of Machine Learning and Systems, 2:230-246. org/paper...ppp..oo..ooooo..... wwwgpappr_fiieeeee fupperfflest wwaerffilesppppeeee wttpee///pppre/2////aaae March 2020. URL http:////c..abbba---a-aaat..a... Abstract. html
Richard Liaw, Eric Liang, Robert Nishihara, Philipp Moritz, Joseph E. Gonzalez, and Ion Stoica. Tune: A Research Platform for Distributed Model Selection and Training, July 2018. URL
betp://arxiie org/ htt///bbb//100// 05118
[cs].
11

---

Jizheng Liu, Zhenpo Wang, Lei Zhang, and Paul Walker. Sideslip angle estimation of ground vehicles:
comparative study. IET Control Theory & Applications,, 14(20):3400-3505 2020. ISSN 1751- 8652. doi: 10.
2020.0516. URL https:://onlleellbrrarry wtwp://wwwww..r.....lle.leeeeeee wiley com doi abs/ 10. ww.11049iettcoo cta. 2020 0516 _eprint: http://oiiiiiiiaaaaaaii///////ir
cta. cta.2020.0516.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards Deep Learning Models Resistant to Adversarial Attacks. February 2018. URL https
//openreview wtt.://wwww.ttorrraarrrraadaaaa.
Girish Narayanswamy, Xin Liu, Kumar Ayush, Yuzhe Yang, Xuhai Xu, Shun Liao, Jake Garrison, Shyam A. Tailor, Jacob Sunshine, Yun Liu, Tim Althoff, Shrikanth Narayanan, Pushmeet Kohli, Jiening Zhan, Mark Malhotra, Shwetak Patel, Samy Abdel-Ghaffar, and Daniel McDuff. Scaling Wearable Foundation Models. October 2024. URL https://openreview www.:/fumff/mii yb4QE6b22f
Yuqi Nie, Nam H. Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. Time Series is Worth 64 Words: Long-term Forecasting with Transformers. September 2022. URL https
//openreviev.o wwweerrview.ne//foooeeee
Simo Särkkä and Lennart Svensson. Bayesian filtering and smoothing, volume 17. Cambridge university press, 2023.
Sebastian A. Schober, Yosra Bahri, Cecilia Carbonelli, and Robert Wille. Neural Network Ro- bustness Analysis Using Sensor Simulations for Graphene-Based Semiconductor Gas Sensor. Chemosensors, 1055):12,, April 2022. ISSN 2227-9040. doi: 00r00//ccheeoooesooss00000000000
URL httpp://wwwtt www. mdpi com 2ttm/2277-9900 17277-9040/10/51111
Connor Shorten and Taghi M. Khoshgoftaar. A survey on Image Data Augmentation for Deep Learn- ing. Journal of Big Data, 6(1):60, July 2019. ISSN 2196-1115. doi: 10.1 10.118///4003770119-01997
URL httph:tt//:so :/dp::://do.. org/ 1186/ 1111/1101367711 9100///40777---1111011 0197-0
Chang Wei Tan, Christoph Bergmeir, Francois Petitjean, and Geoffrre I. Webb. Monash University, UEA, UCR Time Series Extrinsic Regression Archive, October 2020. URL http://arxiv org/ abs/2006. 10996 arXiv:2006. 10996 [cs, stat].
Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojeiech Zaremba, and Pieter Abbeel. Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World, March 2017. URL http::ttp///astttt org abs 1703. 06907
arXiv: 114i::1703006900 [cs].
Pascal Vincent, Hugo Larochelle, Isabelle Lajoie. Yoshua Bengio, and Pierre-Antoing Manzagol. Stacked Denoising Autoencoders: Learning Useful Representations in Deep Network with Local Denoising Criterion. Mach. Learn. Res., 11:3371-3408, 2010. ISSN 1532-4435.
Jun Wang, Wenjie Du, Wei Cao, Keli Zhang, Wenjia Wang, Yuxuan Liang, and Qingsong Wen.
Deep Learning for Multivariate Time Series Imputation: A Survey. CORR, January 2024. URL. http::////pperreeieeeeee////rree wtwp:b/eennee//0oooo////////rro. wttp://rrriidoooeneeeeeeeee.e Wenjie_Dul)
Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient Surgery for Multi-Tase Learning. In Advances in Neural Information Processing Systems, volume 33, pages 5824 5836. Curran Associates, Inc., 2020. URL httpp::::ppooeeeeeeeee wttpiippsccpppppr////ee htap://aaaaaaaaaacaaaaaaaaaaaaa0 wbw.444w0wwb-baattt.c...... html
Hang Yuan, Shing Chan, Andrew P. Creagh, Catherine Tong. Aidan Acquah, David A. Clifton, and Aiden Doherty. Self-supervised learning for human activity recognition using 700,000 person-days of wearable data. npj Digital Medicine, 7(1):1-10, April 2024. ISSN 2398-6352. doi: 1038/ 441746-024-01062- URL
nature. http///arrccceeee111--111111 0224747777-0---002------
Publisher: Nature Publishing Group.
George Zerveas, Srideepika Jayaraman, Dhaval Patel, Anuradha Bhamidipaty, and Carsten Eickhoff. Transformer-based Framework for Multivariate Time Series Representation I Learning. In Proceedings of the 27th ACM SIGKDD Conference Knowledge Discovery & Data Mining,
12

---

KDD '21, pages 2114-2124, New York, NY, USA, August 2021. Association for Computing Machinery. ISBN org/doi/10. 1145/3447548 3467401
doi: 10.1145/3444558844474400 httpss//://
Kexin Zhang, Qingsong Wen, Chaoli Zhang, Rongyao Cai, Ming Jin, Yong Liu, James Zhang, Yuxuan Liang, Guansong Pang, Dongjin Song, and Shirui Pan. Self-Suuervised Learning for Time Series Analysis: Taxonomy, Progress, and Prospects, April 2024. 10125 arXiv:2306.1012. [cs].
org/ abs/230/////
Shuyi Zhang, Bin Guo, Anlan Dong, Jing He, Ziping Xu, and Song Xi Chen. Cautionary tales on air-quality improvement in Beijing. Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences, 473(2205):20170445)) 70457, September 2017. doi: 10, 1098//spa.2011 201..a.011211014
URL httppp///ropaaooceeepppo..aaaaip Royal Society.
rapa
Publisher:
Haoyi Zhou, Shanghang Zhang. Jieqi Peng. Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting, March 2021.
URL htt.:/pp//arrr..or///////2222//1 07436 arXiv:2012.07436 [es].
13

---

A Data and preprocessing
The dataset, recorded 2013 and available at https://cars stansssttaarrrrr.o wdu revs- wwwgppoooooiiiileee-e deeeeddymmisssss cs-aitabasee contains various vehicle dynamics measurements. The models were trained on a subset of the 2013 Monterey Motorsports Reunion dataset and tested holdout CSV from the same dataset, well as the 2013 Targa Sixty Six dataset, which is used to test the performance of the models on an unseen track. All measurements were taken on 1963 Corvette Grand Sport. Two CSV files exhibiting anomalous events (engine issue and vehicle spinning) were excluded. We down-sampled the data to 20 Hz.
Table I: CSV files used for training and testing
Train Data
Test Same Track
Test New Track
Excluded
201 30800 01 01
20130811 02 01_001
20130222 01 01 003
20130817 01 01 02
30130810 1000000000000011
20130222_01_022
20130815_01_01_0
201 30130810 1_01_010010__111
20130222 110222002001000
201308 1_006_000000001 00100102
20130223_00
201308 17 17_02_01_02
20130223 200e23301002221 01_02_00
The collected signals include parameters related to vehicle motion, engine performance, control inputs, and positional data. Table lists all channels we used for training our models.
Table 2: Input channels and target for downstream task
Feature
Deseription
vyCG (target)
Lateral velocity (m/s)
engineSpeed
Engine speed (RPM)
Angle
Handwheel angle (deg)
throttle
Throttle position (%)
brake
Brake pedal force (%)
axCG, ayCG yawRate
Longitudinal and lateral accelerations (m/s²)
Yaw rate (deg/s)
chassis/ccel (FL, FR, RL, RR)
Vertical chassis accelerations (m/s²)
All rows where the longitudinal velocity vxCG was below m/s were removed, as data in this range exhibited outliers in the sideslip angle, likely resulting from its computation based longitudinal and lateral velocities. Given Equation (TO), small measurement noise in vyCG can lead to large errors in the sideslip angle when vxCG is low. Input data standardized using 7-score normalization, and 10% of the training set was used for validation during training.
B Implementation details
All models were trained using the same architecture and hyperparameters, except for the regularization parameters of the non-pretrained baseline model. The pretrained models (Mean, Bias, and Noise, and only Mean masking) were pretrained for 600 epochs and then finetuned for 50 epochs. The non- pretrained baseline was trained for 50 epochs. The models were implemented in Py Torch and trained on single NVIDIA H100 GPU, The code is available htttp:://gttaee com/ JBranát97/
FaultaToFeatures
Pretraining
The proposed model is based on transformer architecture composed of backbone and a projection head. The transformer backbone consists of 4 encoder layers, each with an embedding dimension (dlmatet) 512, 16 attention heads, and feedforward network dimension of 2048. GELU activations, batch normalization, and residual connections are employed within each transformer block. Dropout is not applied. The projection head transforms the flattened transformer encoder output (sequence length of 20 time steps 512 dimensions 10,240 dimensions) into an
Model architecture:
14

---

output dimension of 220 (20 time steps X 11 features) through an intermediate hidden layer of size 512. Similar the backbone, dropout is set to O.
Optimization procedure: The model was trained for 600 epochs using an Adam W-based optimizer with custom hyperparameters. The learning rate schedule (cosine) consisted of an initial learning rate of 3.6040- 10 10 increased linearly to a peak learning rate of 6.6039 100 -4 and then decreased to a minimum learning rate of 6.3811 10 Gradient clipping was applied with maximum norm of 1.0. The optimization parameters 31 and 32 were set to 0.96075 and 0.96127, respectively. Weight decay was not used, and a linear warm-up strategy scaled the learning rate from an initial division factor of 20 (total training steps divided by 20) to the peak learning rate. Training was conducted with batch size of 4096.
Finetuning
The pretraining projection head was exchanged with the finetuning regression head. The regression head flattens the latent space and consists of two intermediate layers with 256 and 64 neurons, respectively, and a final output layer with 1 neuron. The activation function is ReLU, and dropout is set to 0.32 and 0.14.
Regression head:
The finetuning head was trained for 50 epochs using an Adam b-sad optimizer. The learning rate schedule (cosine) involved an initial learning rate 8.6013 10-5 linearly increasing to peak learning rate of 4.233 10 and subsequently decreasing to a minimum learning rate of 1.987 10 Gradient clipping was applied with maximum norm of 1.0. The Adam optimizer parameters 31 and $2 were set to 0.81596, and 0.96962, respectively. A weight decay of 0.0584 was used, linear warm-up strategy scaled the learning rate from an initial division factor of 10 (total training steps divided by 10) up to the peak learning rate. The encoder backbone remained frozen during this phase.
Optimization procedure:
Non-pretrained baseline model
This baseline utilizes the same transformer backbone and regression head architecture employed during the fine-tuning stage of the pretrained models. However, it was trained directly for 50 epochs without any pretraining phase, and the encoder backbone baseline model used the same learning rate schedule and Adam optimizer parameters (31, (2) as the finetuning stage of the pretrained models, its regularization parameters sprecfically weight decay, backbone dropout, and gradient clipping norm-wee tuning was necessary as pretraining can act as a form of regularization [Erhan al. [010], which the model lacks. The dropout rate in the backbone was set to 0.04, and the weight decay was set to 0.001. Gradient clipping was applied with a maximum norm of 3.0.
not frozen. While the non-pretrained
tuned independently. This separate
Hyperparameter selection
The hyperparameters for pretraining, finetuning, and the non- con-pretrained baseline model were deter- mined through combination of automated hyperparameter optimization and informed selection based on established practices in related literature. For the automated tuning part, we utilized Ray Tune [Liaw et al.. 2018)] with an Asynchronous Successive Halving Algorithm (ASHA) scheduler [Li et al. 2020). Architectural choices and certain training configurations also drew inspiration from successful transformer-basee time series models such as Informer [Zhou et al. 202 and the work
by /Zervaas et al. 22221)
Experiments compute resources
All pretraining and fine-tuning experiments were conducted on a single NVIDIA H100 GPU, For the specific experiments presented in Section [.I] each complete training run (pretraining followed by finetuning) required approximately 3 hours. Model evaluations, including the simulation of sensor failures and performance assessment for both baseline models and our approach across all test sets, were performed on an Apple MacBook Pro equipped with an M3 Max processor and 128 GB of unified memory. A full evaluation cycle covering all sensor failure modes took approximately 20 minutes (Figure 4 The total computational effort for the entire project was substantially greater, including extensive hyperparameter tuning and architectural refinements conducted over several months, as individual model training runs never utilized more than single H100 GPU.
15

---

C Different masking combinations
To further understand the contribution of each mask the overall robustness and downstream perfor- mance, we conducted an ablation study where models were pretrained using different combinations of the mean, bias, and noise masking objectives introduced in Section 4.1 Specifically, we trained sepa- rate models using each masking type (Mean only, Bias only, Noise only). all pairwise combinations (Mean+Bias, Mean+Noise, Noise, and the full combination Mean+BBas+NNoiser presented in the main paper. Each pretrained model was subsequently finetuned on the lateral velocity estimation task. We then evaluated these models following the same protocol as Section [.T] assessing their robustness against the four sensor failure modes (mean, bias, noise, scaling) across all input channels on both the seen and unseen test tracks, as well as their baseline performance without induced failures. The results of this comparative analysis are presented in Figure 7
Each data point (color/shape) represents the model's s MSE under a specific sensor failure mode, illustrating robustmess and performance variability across failures; the black line denotes performance without failures. Models lacking bias masking exhibit increased variabilityy emors, while those with bias masking demonstrate enhanced robustness. The three-masking approach achieves second-best downstream performance without errors and shows decreased variability and good robustness for the seen and unseen tracks.
bias and scaling
Pretrain Mask Comparison Individual Sensors (Same Track)
0.45
Applied Error
0.40
No Failure
Mean
0.35
Blas
MSE per Sensor
Noise
0.30
Scaliin
0.25
0.20
0.15
0.10
0.05
MN
MBN
BN
BASE
Pretrain Mask Comparison Individual Sensors (New Track)
0.70
Applied Error
No Fallure
0.60
Mean
MSE per Sensor
Noise
Scaling
0.20
0.10
0.00
BASE
BN
Pretraining Tasks
Figure 7: Robustness evaluation for all different masking combinations.
16

---

D Robustness to increasing sensor failure strength
This section explores the impact of varying sensor failure strengths lateral velocity estimation. We tested the models with different severity for bias (01 E [0.5, 2.5),, noise (Vr E [0.2, 1.0), and scaling (a E [1.5, 3.5) failures. The results show a clear distinction: baseline performance degrades notably with stronger failures, while our pretrained model remains robust. The results also highlight significant variation in impact across input features. For the non-pretrained baseline, the lateral velocity estimation shows high sensitivity failures in some input features, while failures in others have minimal impact, implying the latter may contribute less to the overall estimation.
Same Track Error Type: Bias
engineSpeed
handwheelAngle
throttle
brake
axCG
ayCG
MSE
9.9
20
yawRate
chassisAcceiFL
chassisAccelFR
chassisAcceiRL
chassisAcceIRR
Strength
MSE
20
25
25
25
Strength
Strength
Strength
Blas Strength
Strength
Same Track Error Type: Noise
engineSpeed
handwheelAngle
throttle
brake
axCG
ayCG
0.3
MSE
0.2
0.1
0.6
8.5
0.3
yawRate
chassisAcceIFL
chassisAcceIFR
chassisAcceIRL
chassisAcceIRR
Noise Strength
0.2
0.1
A
1
2
0
8
8
10
0
8
8
8
8
10
0
8
Strength
Strength
Strength
Strength
Strength
Same Track Eror Type: Scaling
engineSpeed
handwheelAngle
throttle
brake
axCG
ayCG
MSE
2
2
yawRate
chassisAcceiFL
chassisAccelFR
chassisAcceIRL
chassisAcceIRR
Scaling Strength
NSE
1.5
29
2.5
39
35
55
28
25
30
55
5
29
23
3
35
15
2
25
300
35
55
100
25
35
Scaling Strength
Scaling Strength
Scaling Strength
Scaling Strength
Scaling Strength
Baseline (no pretraining)
Baseline (mean pretraining)
- Ours (mean, bias, noise pretraining)
Figure MSE on lateral velocity estimation under sensor failures with varying strengths. This experiment was done on data from track included in the training data.
17

---

New Track Error Type: Bias
engineSpeed
handwheelAngle
throttle
brake
axCG
ayCG
MSE
0.
1
3
20
23
yawRate
chassisAccelFL
chassisAcceIFR
chassisAcceIRL
chassisAcceIRR
Blas Strength
0
0.5
2
29
19
20
25
05
20
25
0.
20
25
69
10
13
20
25
Strangth
Strength
Strength
Strength
Strength
New Track Error Type: Noise
engineSpeed
handwheel/ngle
throttle
brake
axCG
ayCG
0.2
0.1
0.4
0.0
yawRate
chassisAcceIFL
chassisAcceIFR
chassisAcceIRL
chassisAcceIRR
Noise Strength
0.2
88
10
2
0
8
8
02
0
6
68
10
2
0
6.
6
2.
0
0
8
Noise Strength
Noise Strength
Noise Strength
Noise Strength
Noise Strength
Track Error Type: Scaling
engineSpeed
handwheelAngle
throttle
brake
axCG
ayCG
yawRate
chassisAccelFL
chassisAcceiFR
chassisAcceIRL
chassisAcceIRR
Scaling Strength
M5E
200
25
3.0
5
200
2.5
30
35
19
2.0
3
Scaling Strength
Scoling Strength
Scaling Strength
Scaling Strength
Scaling Strength
Baseline (no pretraining)
Beseline (mean pretraining)
Ours (mean, bias, noise pretraining)
Figure 9: MSE on lateral velocity estimation under sensor failures with varying strengths. This experiment was done on data from track seen during training.
E Pretraining and finetuning over multiple seeds
To assess the influence of random initialization on model performance and robustness, we repeated the pretraining and finetuning process for the non-pretrained baseline model, the model pretrained solely with mean masking, and our proposed model (pretrained with mean, bias, and noise masking) using 10 random seeds, Each resulting model was evaluated following the same robustness procedure described in Section 5.1 The results are shown in Figure I0, where columns distinguish between evaluations on the test set from the same track used during training (left) and an unseen track (right). The rows correspond to the four different sensor failure modes applied during evaluation (mean, bias, noise, scaling). Within each subplot, individual points represent the MSE for single model trained with specific seed, color-coded according to the training scheme.
18

---

Same Track
New Track
1.2
MSE (Individual Seeds)
0.8
Bias
0.6
0.4
0.2
86
Rap
M45E (individual Seeds)
1.0
0.8
Mean
0.6
0.4
0.0
1.2
1.0
MSE (Individual Seeds)
Noise
0.6
0.4
0.2
0.0
022
1oz
t.z
R.z
8r.
40
1.2
MSE (individual Seeds]
0.8
Scaling
0.6
0.4
0.2
no Seilure
charslokccaRt
chassiokcoaft
chantolcossii
chessistcoelL
BRCG
brate
Draitle
SnginSppaed
handwl Randelsellingle
DyCc
yewRttte
charsssccaaii
chanstokcoaR
ensctbAcosFFi
chasstskccelL.
DXCG
thrntti
oyOG
handehhelllgge
Baseline pretraining)
Baseline protraining)
Ours
pratraining)
Figure 10: Robustness evaluation across 10 random seeds for the con-pretrained baseline, mean-only pretrained baseline, and our approach. Columns distinguish between test sets (left: same track as training. right: unseen track). Rows represent the four sensor failure modes applied during evaluation (mean, bias, noise, scaling). Each point shows MSE for single seed, color-coded by model type.
The results indicate that robustness against noise failures is largely unaffected by the random seed for all three model types. Our proposed approach (Pretrained Mean+Bias+NNiss demonstrates consistently strong robustness across all failure modes and seeds, exhibiting minimal performance
19

---

variation. While the model pretrained only on mean masking also shows consistent robustness for mean and noise failures, its performance under bias and scaling failures varies significantly depending on the seed. This suggests that achieving robustness against failure modes not explicitly seen during pretraining (like bias and scaling for the Mean-only model) can highly sensitive initialization; specific seed might yield model robust to a particular unseen failure, while another might not. In contrast, our multi-n approach improves generalization and is more robust to the unseen scaling failure across different seeds.
Effect of model size on robustness
To investigate the influence of model capacity robustness, we pre-trained models with varying sizes. Specifically, we adjusted the transformer encoder's embedding dimension dmodel through the sequence [32, 64, 128, 256, 512], setting the feedforward dimension to 4 X dmodel accordingly.
Model Size Comparison Individual Sensors (Same Track)
0.50
Applied Error
Failure
0.40
Mean
MSE per Sensor
Bias
Noise
0.30
Scaling
0.20
0.10
MBN
MBN 256 1024
MBN-128-512
MBN 64 256
MBN-32-128
BASE
Model Size Comparison Individual Sensors (New Track)
Applied Error
Failure Mean
0.70
MSE per Sensor
0.60
Dias
Noise
0.50
Scaliin
0.20
0.10
0.00
BASE
MBN 128 512
MBN
MBN 32 128
MBN-64-256
MBN 256-2566-1001 NMN-256-1024 1024
Pretraining Tasks
Figure 11: Robustness evaluation for models pretrained with different embedding dimensions (dmodet). Each point represents the model's MSE under specific sensor failure mode (color/shape) for a given model size. The upper and lower plots show results on the training and holdout tracks, respectively.
The model (MBN) with dmodel = 512 corresponds to the architecture used in the main paper. Each pretrained model was finetuned and evaluated using the same robustness protocol described in Section 5.1 assessing performance under mean, bias, noise, and scaling failures across all input channels for both seen and unseen tracks. The results, presented in Figure [T] indicate a general trend where larger models exhibit enhanced robustness and reduced performance variability across
20

---

different sensor failure modes and input channels. This suggests that increased model capacity allows for learning more comprehensive representations that better handle diverse imput corruptions.
G Effect of pretraining bias bounds
This experiment explores how the range of bias values encountered during pretraining affects robust- ness to bias and scaling failures of varying strengths during evaluation. We pretrained separate models using the full Boinn++Bas+Noiii masking strategy, where the bias masking offset (Section 4.1) was sampled uniformly using bounds / = k and u = h, with & E {1, The model with corresponds to the configuration used in the main paper. These models were then evaluated simi- larly to the experiments in Section D specifically focusing on their performance degradation as the severity of applied bias and scaling failures increased. As shown in Figure 12 and Figure [3] models pretrained with narrower bias bounds (e.g.. & = 1) exhibit noticeable decline in robustness when the applied bias or scaling errors exceed the magnitude seen during pretraining. This suggests that while pretraining enhances robustness within the experienced distribution of failures, generalization significantly stronger, ant-of-distribution failures remains challenging. highlighting the importance of selecting appropriate pretraining parameters based on expected real-world failure characteristics.
Same Track Error Type: Bias
engineSpeed
handwheelAngle
throttle
brake
ayCG
10a
10-1
yawRate
chassisAccelFL
chassisAcceIFR
chassisAccelRL
chassisAcceIRR
Bias Strength
10-1
0.5
9.9
10
23
5.
10
19
23
0.
10
1.
29
25
9.
3.
19
13
3
Blas Strength
Blas Strength
Blas Strength
Blas Strength
Blas Strength
Same Track Error Type: Scaling
engineSpeed
handwheelAngle
throttle
brake
axCG
ayCG
10
1
15
2
2
30
yawRate
chassisAcceIFL
chassisAcceIFR
chassisAcceIRL
chassisAcceIRR
Scaling Strength
10a
10-1
1.
20
2
3
33
29
35
29
2
30
30
29
20
2
30
3
1
29
2
3
Scaling Strength
Strength
Scaling Strength
Scaling Strength
Scaling Strength
Baseline pretraining)
(Bias
(Bias
Figure 12: MSE on lateral velocity estimation under bias and scaling failures with varying strengths, evaluated on the same track as training data. Lines represent models pretrained with different bias bounds (k {1, 21,2,3, 2,3.4) 4)). Log-scaled y-axis.
21

---

New Track - Error Type: Bias
engineSpeed
handwheelAngle
throttle
axCG
ayCG
10"
10-1
yawRate
chassisAccelFL
chassisAcceIFR
chassisAcceIRL
chassisAcceIRR
Bias Strength
10-1
0.7
5.0
2.9
20
23
0.9
10
1.
2
25
0.5
1.
29
23
8
4.9
1
23
0.
10
3.9
2
2
Bias Strength
Blas Strength
Blas Strength
Blas Strength
Blas Strength
New Track Error Type: Scaling
SpeideSpeed
hangwheelAngle
throttle
brake
axCG
ayCG
10
1
2
2
RawRate
chassisAccelFL
chassisAccelFR
chassisAcceIRL
chassisAcceIRR
Scaling Strength
100
10
19
30
3
19
2
2
30
39
13
2
2
30
39
13
20
3.9
90
33
5
29
13
30
Scaling Strength
Strength
Scaling Strength
Scaling Strength
Scaling Strength
pretraining)
Figure 13: MSE on lateral velocity estimation under bias and scaling failures with varying strengths, evaluated on an unseen track. Lines represent models pretrained with different bias bounds (k 2,3, 4)). Log-scaled y-axis.
H Effect of pretraining noise ratio
Complementary to the bias bounds experiment, we investigated the impact of the noise level used during pretraining on robustness against noise failures of varying strengths. Models were pretrained using different noise ratios /r E {0.5, 1.0, 1.5, 2.0}. The model with /F = 1.0 corresponds to our main configuration. We evaluated these models against noise failures with increasing severity (variance scaling /r E [0.2, 1.0)), following the protocol in Section D The results, depicted in Figure 14] indicate that the magnitude of noise applied during pretraining generally has limited influence on robustness against noise failures at inference. An exception is the model pretrained with the smallest noise ratio; this model exhibits reduced robustness, particularly when confronted with stronger noise failures during evaluation. This observation aligns with the findings from the bias bounds experiment, where robustness was most pronounced when evaluation conditions closely matched pretraining conditions.
22

---

Same Track Error Type: Noise
engineSpeed
Aangw
throttle
brake
axCG
ayCG
MSE
Rate
chassisAccelFL
chassisAccelFR
chassisAcceIRL
chassisAcceIRR
Noise Strength
MSE
10-1
0.3
0.*
8
0.3
10
0.2
0
0
10
02
0
0.9
8
10
0
6
0.
0.2
0
8
Noise Strength
Noise Strength
Noise Strength
Noise Strength
Noise Strength
New Track Error Type: Noise
engineSpeed
handwheelAAggl
throttle
brake
axCG
ayCG
MSE
10
oa 1 Noise Strength
0
6
yawRate
chassisAccelFL
chassisAccelFR
chassisAcceIRL
chassisAcceIRR
MSB
6 2
8 Strength
6 10
0.
0
8 Strength
6
10
0.
0
6
8
6
0 8
0
6.2
Strength
Strength
Baseline pretraining)
(Noise 0.5)
Figure 14: MSE lateral velocity estimation under noise failures with varying strengths, evaluated on both the same track and an unseen track. Lines represent models pretrained with different noise ratios (vF {0.5, 1.0, 1.5, 2.0)). Log-scaled y-axis.
I Effect of coupled sensor errors
To evaluate robustness under multi-sensor failure scenarios, we systematically vary the number of failing sensors from 1 to 10 out of 11 total signals, where all failing sensors exhibit the same failure mode: bias shifts (multiplicative factor 1.0), additive noise factor 2.0), complete sensor dropout (mean imputation). For each combination of failure count and error type, we sample random sensor combinations and report average performance. We compare three model configurations: (1) pretraraadded baseline, (2) mean-only pretrained baseline, and (3) our multi-fault pretraining approach (mean, bias, noise). Performance is assessed on both the same track and the new track test data using mean squared eтror (MSE).
0.4). scaling errors (multiplicative
Our approach consistently outperforms the non-pretrained baseline until approximately 5 sensors fail, and surpasses the mean-only pretrained baseline across all tested fault types except complete sensor dropout (mean imputation). expected, we observe performance degradation for all models as the number of failing sensors increases, with our approach demonstrating improved stability across multiple simultaneous failures. Notably, pretraining exclusively on mean failures can actually worsen robustness to other failure modes, such as scaling, compared to the non-pretrained baseline, highlighting the importance of diverse fault exposure during pretraining. Similar performance patterns observed in the evaluation unseen track.
23

---

Same Track Mean Failures
Same Track Bias Failures
-0-
Saselin (ne pretrannig))
Baseline pretraining)
0.5
(inca,, biss, noise pretzalning)
0.4
0.3
MSE
0.2
0.2
0
10
10
Same Track Noise Failures
Same Track Scaling Failures
0.12
0.6
0.4
0.3
0.06
0.
Figure 15: Performance degradation under multiple simultaneous sensor failures test data on the same track. The x-axis denotes the number of simultancously failing sensors.
New Track Mean Failures
New Track Bias Failures
0.7
0.18
0.6
0.16
0.14
0.12
0.10
0.08
0.00
Easeine pretraining)
0.04
0.02
Ours Imean, blos. rose pretraining)
New Track Noise Fallures
New Track Scaling Fallures
0.18
0.5
0.16
0.14
0.3
M55
MSE
0.06
0.1
0.68
0.2
0.0
10
10
Number of Failed Sensors
Number of Failed Sensors
Figure 16: Performance degradation under multiple simultaneous sensor failures on test data on new track.

---

Robustness on further sensor failure modes
To comprehensively evaluate approach's 's generalization the complete taxonomy of sensor failures (see Section 3.1). we conducted additional experiments assessing robustness against drift, outliers, trimming, and varying scaling failures the lateral velocity estimation task. Following the same experimental protocol and using the same dataset as Section 5.1 we applied these failure modes individually each input channel and compared the performance of our pretrained model, the mean-onl, baseline, and the non non-pretrained baseline. The linear and nonlinear drift masks are parameterized such that the maximum drift the final timestep equals twice the feature standard deviation. Outlier errors are injected with probability of 0.05 per timestep, with deviations sampled uniformly from [-30, 3o]. Trimming errors restrict values Iu - o, + o], with constant trimming clamping at the boundary and dampened trimming using a factor = 0.6. Time-varying scaling errors linearly interpolate the scaling factor from 1.0 to 2.0 over the sequence length. The results demonstrate that our multi-fault pretraining approach maintains improved robustness across these unseen failure modes.
yawRate
ayCG
handwheelAnglle
engineSpeed
throttle
brake
axCG
chassisAcce
chacsisAcceIFR
chassisAcceiRL RL
chassisAcceIRR
no, failure
Linear, drib
Nariinear drib
Cutlera
Tiiming constant
Bimming varyin
Time, varying scaling
yawRate
ayCG
handwheelAngle
engineSpeed
throttle
brake
axCG
chassisAcceiFL
chassisArce/FR
chassisAcce/RL
chassisAcceIRR
failure
Uiner dnh
Nontheear drft
Oustero
Vimmiig constart
Bimming
Time wwryin.arrnnn...........
0.0
0.2
0.5
0.0
0.2
0.5
0.0
0.2
0.5
0.0
0.2
0.5
0.0
0.2
0.5 0.0
0.2
0.5
MSE
Mikk
Baseline (no pretraining)
Baseline (mean pretraining)
Ours (mean, blas, noise pretraining)
Figure 17: Robustness evaluation of our pretraining approach across additional sensor failure modes on lateral velocity estimation.
K Further Baseline Comparison
Next, compare approach against SimMTM (Dong et al. 2023 recent pretraining method for time series. This comparison demonstrates the specific benefits of robustness-focun pretraining compared general-purpose time-series representation learning methods.
We adapt SimMTM from its original classification and forecasting setting to our state estimation task, using the recommended hyperparameters from the original paper. Both methods are evaluated on
25

---

the REVS dataset following the same experimental protocol described in Section 5.1 applying each failure type individually all sensor channels and measuring MSE on lateral velocity estimation.
The comparison demonstrates the same performance patterns observed our baseline comparisons. Our robustness-aware pretraining consistently outperforms SimMTMM across all sensor failure scenar- ios, with particularly pronounced improvements for bias failures. While SimMTM performs very well under nominal conditions (for which it is designed), the performance gap widens substantially when sensors degrade. These results confirm that seneral-purposs pretraining methods, despite achieving strong nominal performance, fail to learn representations that are robust to common sensor failures.
yawRate
ayCG
handwheelAngle
engineSpeed
throttle
brake
axCG
chassisAcce/FL.
ChassisAccelFR
chassisAcceIRL
chassisAcceIRR
no_ failure
Same Track Bias
Same Track Mean
Same Track Noise
Same Track Scaling
0.0
0.2
0.4
0.0
0.2
0.4
0.0
0.2
0.4
0.0
0.2
0.4
MSE
MSE
SimMTM
Baseline (mean pretraining)
Ours
Figure 18: Robustness evaluation of our pretraining approach and SimMTM across different sensor failure types.
L Comparison with Kalman Filtering for VSA estimation
Next, we compare the proposed method with nonlincar Kalman filter (KF) (see, c.g., Särkkä and Svensson (2023]) The KF estimates a latent state {øx k = 1,.., K} given set of measurements {y)- k O,... K}. is natural to consider the KF for VSA estimation, as this signal can be modeled in the latent state [Berntorp al. 2022]. The filter assumes known estimation model
3o p(zn)
N( Ma, Po),
(11a)
26 p(xx
N(wx; f( (0 k 1), Q),
(1
p(y )x))
N(w;; (xxx), R),
(11c)
where the prior (mo, Pu]. the noise covariance R]. and the nonlinear functions f. h are known, and N denotes multivariate Gaussian PDF. Following (Berntorp et al.) 2022]. we implement single-track bike model, here defined with the full nonlinear Fiala tire model parametrized in [Lew
et al. 2024 We define state space x (r, w) comprising yaw rate (rad/s), longitudinal velocity v (m/s), side-slip angle 3 (rad), and average rear wheel speed (rad/s). addition, we define control signals
(5, r), with steering angle (rad) and engine torque (Nm). We describe the time-evolution the state by a differential equation (+t) = (((()), ut)) as in [Lew et 2024 use the shorthand 3g x (.hkk), and discretize it by forward Euler at time-step of h (8), yielding
aF #:-1 hJ ( 1 ( 1),
(12)
For the measurement model, we simply include subset of the measurements used as input to the pretrained transformer model, in particular, the yaw-rate, side-slip angle, and rear-wheel speed, as h(æz)) = (rk, Uk, w.) We then implement a nonlinear Cubature KF [Arasaratnam and Haykin 2009 to compute the marginal filtering posterior p( ( and tune the noise levels (Q, Rj manually as hyperparameters to minimize the root mean square error (RMSE). We refer to [Berntorp et al.) 2022] for algorithm details and the implementation for parameter choices (see Section B). In
this setting, the nonlincar KF has access to more information than the pretrained transformer, as
26

---

it assumes parametrized physics physics-based based model along with the control signal trajectory u(t). This makes a direct and truly fair comparison challenging. Nonetheless, we report the MAP estimate when using the same disturbed wheel speeds as realized in the experiment in Sec. 5.2 (see Fig. 6 and compare the side-slip angle reconstruction of the resulting KF to that of the baseline and pre-trained transformer models, with the results reported in Tab. and shown qualitatively in Fig- 19 We note that the nonlinear KF significantly improves over the baseline, despite the failure modes violating the assumptions encoded in the estimation model. However, the model utilizing the proposed pretraining method yields better VSA results, despite not having access to the physics-based vehicle model.
Table 3: Root mean square error (RMSE) and maximum absolute error (MAE) of VSA estimate.
Method
RMSE (rad)
MAE (rad)
Baseline
0.047
0.462
KF
0.018
0.170
Ours (PT)
0.006
0.030
VSA estimation with the nonlinear Kalman filter
wheel speed (rad/s)
25
x (true)
x (disturbed)
20
15
5
0
10
20
30
40
50
y (true)
sideslip (rad)
0.1
y (est)
0.0
-0.1
0
10
20
30
40
50
time (s)
Figure 19: Disturbed average rear wheel speed signals and VSA estimate with the KF (c.f., Fig. 6
M Robust estimation in environmental sensing
To test the generalization of our approach beyond automotive applications, we evaluate our method on the Beijing PM10 and PM2.5 air quality prediction tasks from the Time Series Extrinsic Regression (TSER) benchmark [Tan et al. 2020) Zhang et al.) 2017). where the objective is to predict PM10 and PM2.5 concentrations from nine environmental sensors (SO2, NO2, CO, O3, temperature, pressure, dew point, rainfall, wind speed). We simulate individual sensor failures using the same four error types as in our automotive experiments: bias shifts (two-standard-deviatoo offset), additive noise (a 0.8), scaling eтrоrs (multiplicative factor 2.0), and complete sensor dropout. Each sensor failure is evaluated independently assess the impact of losing specific environmental measurements on PM10 and PM2.5 prediction accuracy. We compare three model configurations: baseline without pretraining, a baseline with mean-onle pretraining, and our multi-fault pretraining approach (mean, bias, noise). Our approach again improves reliability across failure modes and failing sensors, with particularly pronounced benefits observed for scaling enrors in pressure measurements, which cause substantial performance degradation in the baseline models. While the overall performance differences are less pronounced than in the automotive domain, the results demonstrate the cross-domain applicability of pretraining strategy.
27

---

502
NO2
CO
03
TEMP
PRES
DEWP
no_failure
Test - Bias
Test Mean
Test Noise
Test Scaling
10'
10a
10"
10
10'
10'
10
10
RMSE
(log
RMSE
RMSF
Baseline (no pretraining)
Baseline (mean pretra ning)
Ours (mean, blas, noise pretraining)
Figure 20: Robustness on Beijing PM10 air quality prediction with single sensor failures.
no
Test Blas
Test Mean
Test Noise
Test
Sca ling
102
10a
10a
10
10
10a
10
10
scale)
(log scalel
(log
Flog
Baseline (ni pretraining)
Baseline (mean pretraining)
Ours (mean, bias, noise pretraining)
Figure 21: Robustness on Beijing PM2.5 air quality prediction with single sensor failures.
N
Closed-loop experiments
The closed-loop experiments conducted on the Lexus LC 500 were implemented utilizing ROS2 Humble. The experimental setup was adapted mirror the conditions under which the models were originally pretrained and subsequently finetuned. To ensure congruity with the initial training regimen, which was based on the REVS dataset, several modifications were made concerning feature extraction methodologies, input signal frequencies, and overall system integration.
Real-time sensor data from the vehicle served the principal interface for the estimator. While the REVS dataset was sampled down to frequency of 20 Hz, the on-vehicle system operated at 62.5 Hz. To preserve temporal alignment with the REVS dataset, the input data stream down-sampled by selecting every third sample, yielding an effective input frequency of approximately 20.8 Hz. Consequently, the estimator maintained sequence length of 20, thereby encompassing approximately one second of driving history, a configuration consistent with the previous experiments.
The feature set tailored to the available onboard signals. The extracted features incorporated yaw rate, longitudinal and lateral accelerations, vertical chassis accelerations at all four corners, individual wheel speeds, steering angle, wheel torques, brake torques, and longitudinal velocity. Table provides an overview of the channels employed for the LC 500 experiments.
We trained a new model using historical data acquired from previous tests with the Lexus LC 500, and the architecture underwent further tuning. contrast to the REVS dataset approach, the non- pretrained bascline (hereafter referred to as baseline) and pretrained architectures for the Lexus
28

---

Table Input channels and target for the LC 500 deployment
Feature
Unit
Sideslip Angle (target)
rad rad s m ms=
Yaw Rate
Longitudinal Acceleration
Lateral Acceleration
m 2
Vertical Acceleration
Wheel Speed FL
m s
Wheel Speed FR
m
Wheel Speed RL
Wheel Speed RR
S
Steer Angle FL
rad
Wheel Torque RL
N m
Brake Torque FL
m N m
Brake Torque FR
Brake Torque RL
m N m m s
Brake Torque RR
Longitudinal Velocity
platform were tuned independently to ensure optimal performance for the closed-looy tests under nominal conditions. The primary distinctions between the Lexus Pretsrineeddd architecture (Lexus PreT)), Lexus Baseline architecture (Lexus Base), and the REVS architecture are delineated below:
Table 5: Comparison of encoder architecture parameters
Parameter
Lexus Base
Lexus PreT
REVS
Feature Dimensionality
15
Emhedding Dimension
128
256
512
Encoder Layers
Attention Heads
4
16
16
Feedforward Network Dimension
128
512
2048
Backbone Output Dimension
300
300
220
Table 6: Comparison of regression head parameters
Parameter
Lexus Base
Lexus PreT
REVS
Input Size
2560
5120
10240
First Estimation Head Dimension
128
256
256
Estimation Head Dropout Rates
(0.4, 0.34)
(0.32, 0.14)
(0.32, 0.14)
The variations between the Lexus model and the REVS model are primarily attributable to the requirement for real-time execution on Intel Xeon E2278GE CPU @ 3.30 GHz. The difference between the baseline and pretrained models stems from the comparatively simpler nature of the classical end-to-end learning task for the baseline.
N.1
Setup
The architecture of the experimental setup is most effectively elucidated by the diagram presented in Figure 22] The vehicle publishes its state, comprising the features detailed in Table A dedicated "disruptor node" subscribes to both the vehicle state and predefined disruption schedule. This schedule specifies the timestamp, the sensor to targeted, the type error to injected, and the magnitude of the error (applicable for noise and bias injections). The disruptor node then publishes a "disrupted vehicle state". a replica the original vehicle state, except for the designated sensors, which are perturbed according the disruption schedule. The sideslip estimator (which, depending
29

---

on the specific experiment, can be the baseline model, the pretrained model, both) subscribes to this "disrupted vehicle state". It subsequently publishes a "Sideslip Angle Estimation". This estimation, in conjunction with the "disrupted vehicle state", constitutes the "modified vehicle state". This modified vehicle state serves as the control input for the controller, which, in turn, generates the control output directed to the vehicle, thereby closing the loop.
Disruption Schedule
Disruptor Node
Vehicle
Vehicle State
Disrupted Vehicle State
Control Output
Modified Vehicle State
Controller
Sideslip Angle Estimation
Sideslip Estimator
Figure 22: Experimental setup illustrating the closed-loop data flow. The vehicle state is subjected to scheduled disruptions, processed by the sideslip estimator, and the resulting estimation is integrated into the modified vehicle state used by the controller.
To assess the estimator's robustness under authentic closed-loop conditions, controlled fault injection experiments were performed during autonomous driving operations with the LC 500. These experi- ments were meticulously designed to replicate the perturbation regimes employed with the REVS dataset, while simultaneously demonstrating real-world applicability by directly incorporating the estimated sideslip angle into the vehicle's 's controller.
Fault schedules were structured sequentially, perturbing single sensor at a time for five seconds. This perturbation phase was followed by a recovery interval, during which no faults were active. This experimental design facilitated the unambiguous attribution of observed vehicle behavior to specific sensor faults and ensured adequate recovery time between successive fault injections. The injected sensor faults encompassed several distinct perturbation types. Bias eπrоrs were introduced by adding one or two standard deviations of the respective sensor signal. Hard faults were simulated by substituting sensor readings with their corresponding channel means. Additionally, Gaussian noise perturbations were applied, characterized by standard deviations of 0.4 and 1.0 times the feature-specific standard deviation.
additionally tested our method on a race track to evaluate performance across different driving contexts: the skidpad setup being more reflective of regular driving conditions, while the race track introduces racing-like scenarios with increased complexity and variability. The initial results showed comparable robustness across both environments. Ultimately, we decided to focus the skidpad experiments for the paper due to their superior reproducibility through structured, repeatable maneuvers and clearer communicability of results.
Sensor failure experiments
For the comparative analysis of the baseline and pretrained models, two distinct experimental scenarios were investigated: Parallel Loop and Separate Loop.
In the Separate Loop scenario, both models functioned as the sideslip estimator in discrete test sessions. This approach permits robust comparison of the models and an examination of how their respective estimation errors propagate. However, it does not guarantee identical experimental
30

---

conditions across tests, primarily due to the controller's 's reaction to estimation inaccuracies. For instance, if an estimation eπror causes the vehicle decelerate, a time-scheduled sensor disruption will occur during a different driving maneuver than if the vehicle had maintained its original speed. This methodology was employed for the Wheel Speeds sensor experiments, the results of which are documented in the main paper (Section 5.2
In the Parallel Loop scenario, the pretrained model served as the sideslip estimator within the closed- loop system, as previously described. Concurrently, the baseline model operated in an open-loop configuration, processing the same input data but without its output influencing the vehicle controller. The principal advantage of this test scenario lies in the direct comparability of both models using identical input data. Conversely, its limitation is that only the closed- -lop model's estimation error can propagate through the system, thereby revealing the real-world consequences for only one of the two models. By employing both testing scenarios, a comprehensive understanding of model performance under sensor failure conditions is achieved. This experimental paradigm was implemented for the wheel speed sensors, the yaw rate sensor, the steering wheel sensor, and the longitudinal velocity sensor.
0.1 Parallel loop skidpad: wheel speed sensor perturbations
The experiments involving the disruption of wheel speed sensors were conducted using a structured perturbation schedule. This schedule consisted of the following sequence for each targeted sensor:
1. seconds of Gaussian noise, with a standard deviation (a) equal the standard deviation of the original sensor signal.
2. second recovery interval (no perturbation).
3. seconds of bias error, where value equivalent to 2a of the original signal was added to the sensor readings.
4. second recovery interval.
5. seconds where the sensor signal was replaced by its mean value.
6. second recovery interval.
This perturbation sequence was applied sequentially to each wheel speed sensor: front-left (FL), front-right (FR), rear-left (RL), and reah- (RR). The 1-second recovery intervals ensured that each 1-second input sequence to the model contained at most one type of disruption, enabling clear analysis of each perturbation's 's isolated effect. The resulting disruption patterns are shown in Figure 23
Our pretrained model demonstrated no significant degradation in performance in response to any of the applied disruptions on any wheel. This outcome suggests the successful impartation of robustness through the pretraining process. Conversely, the baseline model experienced dramatic decreases in performance, with the most severe instance observed during the bias error injection on the front-right wheel speed sensor. This event led to estimation error exceeding 0.30 rad.
is also noteworthy that bias error consistently induced a constant-like error pattern in the baseline model's estimation. In contrast, noise eтror (most observable with the rear-right wheel speed sensor perturbation) resulted in a noisy error pattern the estimation. The mean value substitution, however, produced varied error patterns: sometimes spikey error pattern (e.g., front-left wheel speed), sometimes no discernible real error pattern (e.g.. front-right wheel speed), or a pattern similar to that of the bias error (e.g., rear-right wheel speed). These observations confirm that, despite the nearly identical information content across all wheel speed sensors, the baseline model remains susceptible to eπrors when any single one of them is compromised.
Additionally, a separate experiment was conducted in which constant bias of 2o (relative to the standard deviation of the original signal) was applied to the rear-right (RR) wheel speed sensor shown in Figure 24 This sustained perturbation led to only minor estimation errors our pretrained model (occurring around the 35-second mark in the experiment). In contrast, the baseline model exhibited severe errors. Interestingly, the error magnitude of the baseline model decreased as the absolute value of the wheel speed increased (indicative of the vehicle traveling on a straight path, where the side-slip angle is typically small or negligible). Conversely, as the wheel speed decreased, the estimation error of the baseline model increased.
31

---

Wheel FL (Original Disrupted
Drignna
25.00
0.00
Wheel Speed HK (Original vs Disrupted X)
v(rades]
75.00
Driginal (c)
Darupbec (9)
0.00
Wheel Speed RL (Original Disnupted R)
75.00
(8)
$5.00
25.00
Speed (Onginal Disrupted
Original
Disrupter (R)
50.00
0.00
Sideslip Angle Comparison ly vs
Ground Iith
Sidesiip Angle (red)
0.20
Mode (R)
Base (5)
6.30
4.40
Emor [re]]
0.20
0.00
Bused in Model Erar B-
0.20
Time
Figure 23: Disruption patterns applied sequentially to individual wheel speed sensors (FL, FR. RL,
RR) and the corresponding sideslip angle estimation performance of our model versus the baseline model. The top four plots show the original versus disrupted wheel speed signals. The subsequent plots show the sideslip angle ground truth, our model's estimation, the baseline model's estimation, and their respective errors.
32

---

Wheel (Original Disrupted
Driinnl
80.00
60.00
20.00
Sideslip Angle Comperison (y
0.10
Ground Ii-h Our Mode Sase (5)
Sidesll Angle (rec)
0.00
20.30
2.20
0.30
2.40
4.50
Our Model Enor B >
Enor
0.25
0.00
Error (rad)
Eointt te Model Enor l y!
0.25
0.00
10
20
30
60
Time [s]
Figure 24: Disruption patterns applied constantly the rear right wheel speed sensor (RR) and the corresponding sideslip angle estimation performance of our model and the baseline model. The plot shows the original versus the disrupted wheel speed signal. The subsequent plots show the sideslip angle ground truth, our model's estimation, the baseline model's estimation, and their respective errors.
O.2 Parallel loop skidpad: yaw rate sensor perturbations
The robustness of the sideslip angle estimation was further evaluated by introducing simulated perturbations to the yaw rate sensor signal. The perturbation schedule was structured distinet phases:
1. Initial phase (0-18 seconds):
6 seconds of additive Gaussian noise with a standard deviation equal to 0.4g of the original signal's standard deviation.
1 second recovery interval.
6 seconds where the sensor signal was replaced by its mean value the recording duration.
second recovery interval.
6 seconds of constant bias error, with an added value equivalent to lo of the original signal's standard deviation.
1 second recovery interval.
2. Incremental noise phase (18 seconds onwards for a defined duration):
The intensity of the additive Gaussian noise was progressively increased second-wise, starting from Oo and ramping up to level exceeding l of the original signal's 'ss standard deviation.
1 second recovery interval.
3. Incremental bias phase (Following incremental noise phase for a defined duration):
The magnitude of the constant bias error was incrementally increased, starting from 0.5g and escalating to 2a of the original signal's standard deviation.
33

---

The impact of these perturbations on the yaw rate signal and the corresponding sideslip angle estimations from both our pretrained model and the baseline model are depicted in Figure 25
yewhtate (Original vs Disnupted
Original (el
Uisruptes (6)
135
1.00
0.0
Sidezlip
0.10
(5)
0.00
0.15
Eroor [[ad]
Our Model broor l
Mo
Enor [od]
Bacol te Model Enor
20 Time
Figure 25: Perturbation effects on yaw rate and sideslip angle estimation during the Parallel Loop Skidpad maneuver. The top panel shows the original versus the disrupted yaw rate signal. The middle panel presents the ground truth sideslip angle alongside the estimations from our model and the baseline model. The two bottom panels display the estimation error for both models.
The yaw rate is a critical input for sideslip estimation, directly reflecting the vehicle's 's rotational dynamics. As illustrated Figure 255 our pretrained model consistently maintained low estimation error across all phases of the perturbation schedule. During the initial phase, the introduction of low- magnitude noise (0.4o) and mean substitution resulted in negligible deviations our model's output from the ground truth. Even during the subsequent incremental noise phase, where the perturbation intensity became significantly more pronounced, our model demonstrated remarkable resilience, with estimation errors remaining minimal. Notable deviations in the estimation for our model occurred during the bias perturbations, both the initial la bias and subsequently during the incremental bias phase.
In contrast, the baseline model exhibited significant sensitivity to the yaw rate perturbations. The la bias applied around 15-21 seconds induced a noticeable and sustained error in its sideslip angle estimation. As the noise intensity was incrementally increased (e.g., observe behavior post 25 seconds). the baseline model's 's error correspondingly escalated, displaying significant noise in its output. The incremental bias phase (e.g.. observe behavior towards the latter part of the experiment) led to diverging estimation eтror for the baseline model. For instance, at a bias of 2o, the baseline model's s estimation error reached approximately 0.25 rad. This underscores apparent over-reliance on the yaw rate signal and its limited capacity to mitigate the effects of significant sensor corruption, highlighting stark contrast to the robustness achieved by our pretrained approach.
baseline model's
34

---

0.3 Parallel loop skidpad: steering angle sensor perturbations
To assess the impact of corrupted steering input on sideslip angle estimation, a series simulated perturbations applied to the steering angle sensor signal. The experiment employed the same multi-stage perturbation schedule as applied for the yaw rate sensor evaluation. Figure 26 illustrates the original versus perturbed steering angle signal alongside the sideslip angle estimations from our pretrained model and the baseline model, as well as their respective estimation errors.
Steer (Original Disrupted
Deginal
0.42
Angle
0.20
-0.00
Sideslip Angle Comparison (y vs
Ground Iuth
Our Mode 16)
0.00
Ravellin Mode (5)
0.04
Sidedip
0.04
Entr [rod]
Our Model Enor 1
0.00
Emar [rad]
Exvan Model Froor $
0.00
Time [s]
Figure 26: Perturbation effects steering angle and sideslip angle estimation during the Parallel Loop Skidpad maneuver. The top panel shows the original versus the disrupted steering angle signal. The middle panel presents the ground truth sideslip angle alongside the estimations from model and the baseline model. The two bottom panels display the estimation eтror for both models.
The results presented in Figure 26 highlight the differential response of the two models to steering sensor corruption. Our pretrained model demonstrated robust performance throughout experiment. The initial perturbations (low noise, mean substitution, and Ta bias) had a minimal effect its estimation accuracy. More critically, during the incremental noise and incremental bias phases, our model's estimations remained closely aligned with the ground truth, with only minor fluctuations in error.
The baseline model's performance under steering angle perturbations exhibited behavior largely similar to our model during the initial low-noise and bias phases. However, critical distinction emerged with the introduction of the mean value substitute (around 9-15 seconds), which resulted in a clear and sustained deviation the baseline model's 's sideslip estimation. This deviation highlights specific sensitivity of the baseline model to this type of sensor corruption.
0.4 Parallel loop skidpad: longitudinal velocity sensor perturbations
The final single-sensor perturbation experiment focused on evaluating the impact of corruption in the longitudinal velocity signal. The same structured perturbation schedule utilized for the yaw rate and steering angle sensor experiments applied. Figure 27 displays the original versus perturbed
35

---

longitudinal velocity signal, the ground truth sideslip angle, and the estimations and errors from both our pretrained model and the baseline model.
Longitudinal Velocity (Original Disrupted XI
Driginal (4) Danupbes (8)
40.00
Longitudiral Wilectt
20.00
10.00
-10.00
Sidesiip
(5)
Sidesiic (rad)
(5)
4.50
-1.00
Error (red)
Model Froor y
0.50
trror (red]
Exreel Model broo f
2.50
Figure 27: Perturbation effects on longitudinal velocity and sideslip angle estimation during the Parallel Loop Skidpad maneuver. The top panel shows the original versus the disrupted longitudinal velocity signal. The middle panel presents the ground truth sideslip angle alongside the estimations from model and the baseline model. The two bottom panels display the estimation error for both models.
Under these perturbation conditions, both models experienced significant degradation in performance. The baseline model exhibited near-complete failure in estimating the sideslip angle from the beginning of the first phase, rapidly reaching errors exceeding 0.5 rad. Our model estimate degraded particularly during the lo bias perturbation, which led to errors close to 0.3 rad. The safety driver had to intervene to maintain control the vehicle, necessitating the premature termination the experiment as soon the incremental bias phase was initiated, preventing the completion of the full perturbation schedule. Nevertheless, despite the challenging conditions, our model consistently outperformed the baseline model across all applied perturbations before the experiment was stopped. More about this in Section
36

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: See Section 4. for proposed pretraining scheme. See Section 5.1 and Section 5.2 for results robustness and closed-loop control.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contribu- tions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
See Section 6 for discussion of limitations.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point any strong assumptions and how robust the results to vi- olations of these assumptions (e.g.. independence assumptions, noiseless settings, model sell-specification asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by review- ers as grounds for rejection, a worse outcome might that reviewers discover limitations that aren't it acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each thcoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
37

---

Justification:
The paper does not include theoretical results. Results are based on empirical evaluations on public datasets and real-world experiments.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by
formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experi- mental results of the paper the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
The pretraining scheme is described in detail in Section 4. The experimental setup is described in Section 5. Section B Section 5.2 and Section N The data split and preparation is detailed in Section A The code and data for the evaluation in Section [_1 will be made available. While we can't 't publish the data collected during the closed-loop control experiments, we provide a detailed description of the experimental setup and results in Section 5.2
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or the contribution is specific model and empirical evaluation, may be necessary either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access to hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate the research performed.
While NeurIPS does not require releasing code, the conference does require all submissions provide some reasonable avenue for reproducibility, which may depend the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-soure dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g.. to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
38

---

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: The code for the evaluation in Section 5.1 and the appendix will be made available. While we can't 't publish the data collected during the closed-loop control experiments, we provide a
detailed description of the experimental setup and results in Section 5.2 and Section N
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines
guides https///oooouummmssssooooooooooo
for more details.
While we encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed run to reproduce the results. See the NeurIPS code and data submission guidelines (https //nips. //wpps.cc/ublll... wwtppaalcc...uies/e//ooeoocapa Setggdde//ooeeuumiieee.... wwwiissssooppyiiii for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification: See Section 5.1 Section
B
Section
5.2 Section and Section A
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: The experiment in Section 5.1
reports the bootstrapped estimates (200 resamples) showing mse means and their 95% confidence intervals on the test set. Furthermore, in Section D show the results for different severities of the sensor failures. In Section E we show the evaluation done in Section 5.1 for 10 models trained with different random seeds.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, statistical significance tests, at least for the experiments that support the main
claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
39

---

The method for calculating the error bars should be explained (closed form formula, call to library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors). It should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). If error bars are reported in tables or plots, The authors should explain in the text how they were caleulated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type compute workers, memory, time of execution) needed reproduce the experi- ments?
Answer: [Yes]
Justification:
See Section B and Section
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experi- mental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: https::/neerips: cccpublic//eeeeeeeeeeeeeeeeeeeee hatool/cc/chhce@uieeeeeeee
Answer: [Yes]
Justification:
This research focuses improving the safety and robustness of deep learning models in critical applications, directly aligning with ethical considerations for Al deployment. We uphold principles of transparency and reproducibility by providing detailed methodology and releasing code, adhering to the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The primary positive societal impact-eehanciin the safety and reliability of ML systems reliant on potentially faulty sensors- discussed throughout. Potential negative impacts implicitly addressed in Section 6 which discusses scenarios where the method might fail (c.g. unevaluated failure modes/domains), potentially leading to unsafe outcomes if deployed without considering these limitations.
Guidelines:
40

---

The answer NA means that there is no societal impact of the work performed. If the authors answer NA or No, they should explain why their work has societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploy- ment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point out. For example, it legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This work does involve the release of trained model. The research focuses methodology for improving robustness using specific vehicle dynamics time series data, which itself does not pose a high risk for misuse. The primary dataset used is publicly available.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions access the model implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The primary dataset used is publicly available and properly credited. Our codehase utilizes parts of the publicly available code from Zerveas et al. [202T]. which is also credited. Standard open-source libraries used are governed by their respective permissive licenses.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from particular source (e.g., website), the copyright and terms of service of that source should be provided.
41

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. cww /anaaatsssss has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The code implementing the proposed pretraining methodology will be released. Documentation (e.g., README, comments) will be provided alongside the code to facilitate understanding and usage.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their sub- missions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Question:
Answer: [NA]
Justification:
This research does not involve crowdsourcing or experiments with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification:
This research does not involve crowdsourcing experiments with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may required for any human subjects research. If you obtained TRB approval, you should clearly state this in the paper.
42

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness. or originality of the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs as any important, original, or non-standard component.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, or non-standard components.
Please refer our LLM policy hetp:s//meeurisseccc//ffferrrrre should or should not be described.
for what
43


Reviews:
[{"review_id": "qTQLDQJb44", "reviewer": "Reviewer_nzcH", "summary": "This paper introduces a self-supervised pretraining approach for multivariate time series aimed at enhancing robustness against realistic sensor failures. Unlike conventional methods that use simple zero-masking, this work simulates specific failure modes such as bias, noise, and hard faults during pretraining. The goal is to encourage the model to reconstruct the original signal despite these complex corruptions. The approach is evaluated on vehicle dynamics data for sideslip angle estimation, where it demonstrates improved robustness on both seen and unseen test tracks. Furthermore, it is deployed in a real-world autonomous racing scenario, successfully enabling stable closed-loop control even when sensor faults are intentionally injected.", "questions": "1. Could you provide a more systematic analysis or justification for why the pretraining was based on only three of the seven identified fault modes? Is there empirical evidence to show that the chosen masking strategies (bias, noise, hard fault) can effectively represent the other un-tested failure types like drift and outliers?\n\n2. How robust is the method to changes in the pretraining hyperparameters, such as the masking ratio, the bounds set for bias injection, and the levels of noise?\n\n3. How do you see this approach transferring to other time series domains, such as health monitoring or industrial IoT, where sensor outage characteristics might be significantly different?\n\n4. Given that the model is trained to reconstruct signals from specific fault types, could this pretraining strategy also be leveraged to detect or classify which failure mode is present at test time, in addition to improving robust estimation?", "limitations": "yes", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "# strength\n1. Problem Significance: The paper addresses the important and practical problem of safety-critical sensor failures, which is highly relevant for real-world systems like autonomous vehicles.\n\n2. Novel Pretraining Strategy: The proposal to use masking methods that are grounded in realistic sensor faults (e.g., bias, noise) is a clear strength over standard mean-only masking.\n\n3. Strong Empirical Results: The experimental results, on both synthetic data and a real-world application, clearly demonstrate the robustness of the proposed masking approaches (bias, noise, and mean) compared to a standard mean-only masking baseline. The successful deployment in a closed-loop autonomous racing scenario is a particularly strong piece of evidence.\n\n4. Thorough Ablation Studies: The paper provides valuable ablation analyses that examine the effects of different masking strategies, failure intensities, model sizes, and the model's ability to generalize to unseen fault conditions.\n\n# weakness\n1. Incomplete Evaluation of Fault Modes: The primary weakness is that the experiments do not cover all seven fault modes listed in Section 3.1 (e.g., drift, outliers). The mapping between the three tested masking techniques (mean, bias, noise) and the seven real-world failure modes is only described qualitatively. For example, the assumption that bias corruption is sufficient to model sensor drift is not empirically validated, leaving the model's robustness to these other modes unclear.\n\n2. Limited Baselines: The comparison is limited to a non-pretrained baseline and a simple mean-only masking baseline. The evaluation would be significantly stronger if the proposed method were compared against other state-of-the-art time-series pretraining and masking methods (e.g., Ti-MAE, Sim-MTM).\n\n3. Questionable Generalizability: The evaluation focuses on a single application domain (vehicle dynamics). This concentration, while providing a strong case study, limits the evidence for the method's generalization to other multivariate time series settings (e.g., healthcare, industrial IoT) which may have different fault characteristics. Additionally, the real-world tests are confined to controlled racing environments, making it unclear how the method would scale to more diverse and unstructured contexts.", "was_revised": true, "final_justification": "Updated the score to 4. The paper lacks on generality and fault models.", "rebuttal": "We thank the reviewer for the detailed feedback, and for highlighting the novelty of our pretraining strategy and the strength of our empirical results, as well as for the constructive suggestions for improvement.\n### **Weaknesses**:\n\n**W1 & Q1**: Due to the combinatorial complexity of covering all 10 failure modes and their combinations, we focused on three representative types informed by our experience with VSA in the driving domain - bias, noise, and hard fault. All combinations of these three failure types were systematically evaluated (see Appendix C).\n\nBias masking was chosen to approximate several untested faults: short, sharp offsets during training mimic outliers, while drift, trimming, and scaling resemble varying bias over time. Given our short sequence lengths, we assume minimal temporal dynamics, making a static bias a reasonable proxy. Our evaluation of the holdout scaling fault supports this, showing effective generalization. For domains with lower sampling rates or longer sequences, including additional fault types in pretraining could be beneficial. We hope this also addresses Question 1 sufficiently.\n\nYou raise a very good point, and we will clarify this in the paper.\n\n**W2**: Thank you for the suggestion. We agree that comparing against recent time-series pretraining methods offers valuable insight, especially regarding nominal performance and robustness.\n\nOur primary goal was to isolate the effect of adapting the pretraining task to handle sensor failures, rather than to outperform the best nominal models. To do this, we used the same architecture with and without robustness-focused masking, emphasizing the impact of our approach. Methods, such as Sim-MTM and Ti-MAE, do not explicitly address sensor outage robustness in their pretraining procedure.\n\nNonetheless, we conducted additional experiments with Sim-MTM. Since it was originally evaluated on classification and forecasting, we adapted it for the state estimation setting. While Sim-MTM performs well under nominal conditions, our method shows superior performance under sensor outages, underscoring the value of robustness-aware pretraining. A summary of results is provided below, with full details to be included in the appendix of the paper. The table shows the MSE of the lateral velocity estimation (as in Fig. 3), averaged across all failing sensors for a specific failure mode.\n\n**Comparison of Our Method vs. SimMTM Across Failure Types**\n\n| Failure Type | Ours     | SimMTM  |\n|--------------|----------|---------|\n| Bias         | **0.0484*** | 0.1352  |\n| Mean         | **0.0536*** | 0.1203  |\n| Noise        | **0.0283*** | 0.0423  |\n| Scaling      | **0.0433*** | 0.1009  |\n| No failure   | **0.0259*** | 0.0400  |\n\n\\* Lowest error per row marked in bold.\n\nSim-MTM was evaluated using the original paper’s recommended hyperparameters. While further tuning might boost nominal performance, we don’t expect significant gains in robustness, as the method isn’t designed for such scenarios.\n\nWe appreciate the reviewer’s suggestion and believe these additions strengthen the paper.\n\n**W3 & Q3**: We would like to clarify that Sec. 3.1 describes a set of general sensor outages, building on a body of work from NASA [Balaban et al., 2009] and a work about wireless sensor networks [Jesus et al., 2017], distinct from our driving application. Training with these finite general failures improves sensor outage robustness in driving, and it could bring improvements in other domains, such as GNSS positioning (where multi-path effects are present).\n\nIn response to your comment, we conducted additional experiments on the Beijing PM10 dataset from the Time Series Extrinsic Regression (TSER) benchmark. This dataset involves predicting PM10 air quality levels in Beijing based on various environmental measurements, such as pollutant concentrations, temperature, and pressure. Our results demonstrate that the proposed method drastically improves robustness to sensor failures in this setting. For comparison, we report the XGBoost RMSE from the TSER benchmark website, which represents the strongest performance on the dataset under standard (non-failure) conditions.\n\n**Beijing PM10 Benchmark - Mean RMSE ± Std across Sensor Failures**\n\n| Error Type   | Baseline         | PT Baseline      | Ours              | XGBoost (RSER BM) |\n|--------------|------------------|------------------|-------------------|-------------------|\n| bias         | 117.63 ± 21.06   | 127.22 ± 27.48   | **104.89 ± 6.29*** | -                 |\n| mean         | 107.41 ± 12.36   | 103.29 ± 3.68    | **101.07 ± 5.21*** | -                 |\n| noise        | 98.19 ± 6.92     | 108.20 ± 6.78    | **97.52 ± 2.59***  | -                 |\n| scaling      | 359.13 ± 756.87  | 402.42 ± 843.38  | **117.44 ± 27.10***| -                 |\n| no_failure   | 95.06            | 97.98            | 94.05              | **93.13***        |\n\n\\* Lowest error per row marked in bold.\n\nOur method enhances robustness to sensor failures in the driving domain and clearly improves robustness in estimating air quality in the Beijing PM10 dataset. Its general applicability to other domains remains an open question, which we plan to explore in future work. \n\n1) To test the generalizability to unseen sensor errors, we evaluated it on the previously unseen failure mode of \"scaling,\" demonstrating that the model better generalizes to novel sensor faults not seen during training (Sec. 5.1).\n2) Additionally, in experiments not reported in the paper, we also tested our method on a race track, where the vehicle was driven at higher speeds and encountered more dynamic conditions. While the skidpad setup is more reflective of regular driving, the race track introduces racing-like scenarios with increased complexity and variability. Our method showed comparable robustness in both settings, suggesting its ability to scale beyond structured environments. We focused on the skidpad in the paper due to its higher reproducibility and clearer communicability, but the track results provide additional support for real-world applicability.\n\nIn response to your concern, we will add the additional benchmark and a discussion of point 2) to the appendix, and will clarify point 1) in the main text. We hope this also addresses Question 3 sufficiently.\n### **Questions**:\n\n**Q2**: Early hyperparameter studies showed that a 10% masking ratio, paired with an average mask length matching the input sequence length, offers the best trade-off between robustness and downstream performance. Nearby values (e.g. 15%) showed only minor degradation. We speculate that this choice prevents simple interpolation and encourages the model to learn cross-channel dependencies. It may also help simulate outlier scenarios by occasionally masking parts of other channels. For intuition, with an input of 220 scalars (sequence length × features), a 10% masking ratio results in 22 masked data points, with one channel being completely masked on average.\n\nWe show that too-small bias bounds during pretraining hurt performance under strong offset and scaling errors, likely due to the inputs deviating too far from the training distribution. Using bias levels up to three standard deviations improves robustness (see Appendix G).\n\nFinally, we find that while noise level effects are less pronounced, too little noise during pretraining still leads to degraded robustness. A moderate amount of noise helps the model generalize to noisier test conditions (see Appendix H).\n\n**Q4**: This is a very good suggestion. Early experiments using the pretraining reconstruction head as an anomaly detector showed promise but were ultimately left out of the paper. Extending this to classify outage types is a strong direction for future work, as it may provide additional actionable information for downstream tasks and diagnostics."}, {"review_id": "BwBrUuP7YS", "reviewer": "Reviewer_J9ZX", "summary": "This paper proposes a self-supervised masking method to learn how to recover timeseries signals from common sensor faults. Specifically, the authors propose an encoder-based model that is pretrained on masked data that incorporates three common corruptions: mean masking, constant biasing, and Gaussian noising. Then, the encoder is fixed and the decoder is finetuned for the downstream task at hand. The proposed method is evaluated in simulation using a vehicle dataset, and it is shown that the proposed method achieves lower reconstruction error than the baselines (either with no pretraining, or with pretraining using only mean masking), indicating a higher degree of robustness against the sensor faults at hand. In the simulation, it is also shown that the proposed method achieves better reconstruction than the baselines in the case of unseen sensor faults (that is, scaling faults, which were not incorporated in the pretraining). The model is also demonstrated on an actual autonomous vehicle experiment, where it is shown to yield reconstruction estimates good enough to result in stable closed-loop vehicle control, whereas the baseline method either led to the vehicle leaving the track or needing human interference.", "questions": "Below I have enumerated a few specific questions, suggestions, and comments, most of which are minor:\n\n1. Minor formatting issue: I would suggest placing Figures 1 and 2 after the main body of the paper has begun (i.e., after some of the text in Sections 1 and 2, rather than immediately after the Abstract).\n\n2. Are the assumed sensor fault modes enumerated in Section 3.1 fairly exhaustive of real-world sensor failures? What sort of realistic failures might your assumed failure modes not capture?\n\n3. Line 96: By $\\mathcal{U}(l,u)$, do you mean the uniform distribution on the interval $[l,u]$? If so, it would be best to clarify this choice of notation to the reader.\n\n4. Figure 2 appears long before it is introduced and discussed in the text. It would be best to place it closer to where it is discussed in the text.\n\n5. Minor notation issue: In Line 154, you use the notation $\\mu_{[f]}$ to denote the channel mean, but in Line 157, you use the notation $\\mu_f$. Are these meant to be the same, or different values? The notation should be consistent if they are intended to denote the same value.\n\n6. Overall, Figures 5 and 6 are really compelling and interesting. However, why do the vehicle positions look the exact same between the three overhead images? Why is the vehicle position not aligned with the experimental trajectory (purple line) in Figure 6 (left)?", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Quality: Overall, this is a high-quality and complete piece of work, with technically sound approaches and evaluations. The evaluation of the proposed method is thorough, with plenty of ablation studies provided. No theoretical developments or claims are made, so all of the support provided is empirical. The authors are transparent about the limitations of their work (dedicating the entire Section 6 to describe limitations and possible future work to overcome such limitations).\n\nClarity: The paper is very well-written and easy to follow; I enjoyed reading it. For the most part, the paper is formatted and structured well (see my enumerated \"Questions\" below for specific questions and suggestions on how to improve the clarity and presentation). The literature review is relatively thorough, giving the reader a good idea of recent relevant work. Enough data, code, and experimental descriptions are provided (with much of it in the appendices and supplement) to reproduce the results.\n\nSignificance: The problem considered is an important one; indeed, it is necessary for safety-critical autonomous systems to be resilient against sensor faults and timeseries data corruptions, especially as control systems become more dependent on learning-based methods. Experimentally, this work appears to offer significant advantages over the baseline methods in realistic problems. The paper does not provide any theoretical analysis of the proposed model, which would certainly boost the significance of the work, as it is also important to grant safety-critical systems provable reliability and robustness guarantees. That said, this paper is an empirical one, appears to still offer significant advances to the field, and opens the door to future theoretical studies of the proposed methods.\n\nOriginality: This work offers novel methods that appear to advance the field, as supported by the impressive experimental results. This is achieved by employing a novel combination of existing techniques, with the relationship to past methods clearly articulated with proper references. The work does not necessarily provide new insights, deepen understanding, or highlight important properties of existing methods, as it is primarily focused on evaluating the newly proposed method.", "was_revised": true, "final_justification": "I thank the authors for their responses to my questions/suggestions, and for clarifying/fixing the potentially misleading aspects of the overhead images. I have also read through the other reviews and responses, and I am happy to see the authors have performed a handful of additional comparisons to baselines and ablation studies, all in support of their proposed methods. Overall, I find this to be a very nice paper making strong, applied contributions to the communities at the intersections of reliable ML, sensing, and autonomous driving. Obviously, there is still room for future improvement in this domain, but I don't think that warrants rejecting the paper just because the proposed methods rely on some assumptions about the failure modes and aren't able to fully resolve every possible real-world sensing failure situation. Therefore, I maintain my score of accept (5).", "rebuttal": "We sincerely thank the reviewer for the positive and encouraging feedback, particularly regarding the significance and originality of our work. We also appreciate the constructive questions and suggestions aimed at improving the clarity and presentation of the paper.\n### **Questions**:\n\n**Q1 & Q4**: Thank you for the suggestion. We chose to include Figures 1 and 2 early in the paper to serve as a kind of graphical abstract, providing readers with a quick visual overview of the main ideas and results. However, we recognize the potential disruption this placement may cause to the narrative flow, as you pointed out. We will move their location in the final manuscript to improve the clarity of the presentation.\n\n**Q2**: Sec. 3.1 describes a set of general sensor outages, building on a body of work from NASA [Balaban et al., 2009] and a work about wireless sensor networks [Jesus et al., 2017], distinct from our driving application. We neglected faults that manifest as NaN values, which we view as a practical variant of the hard fault already described. Additionally, we excluded time-series anomalies like flipped seasonality, as they are not directly caused by sensor faults.\n\n**Q3**: Yes, that is correct. We refer to the uniform distribution over the interval [l,u]. We will revise the text to make this notation explicit and avoid any ambiguity.\n\n**Q5**: Thank you for pointing this out. The notations are intended to represent the same value, and we will make the notation consistent throughout the manuscript to improve its clarity.\n\n**Q6**: The overhead images are constructed using positional data and drone footage. We overlaid a car image to indicate driving direction, but since we didn’t record every trial with a drone, the car position is fixed. The trajectory (purple line) is the key data point, but we agree that the static car image may be a bit misleading and will clarify this in the figure caption."}, {"review_id": "cUwWMGSGm7", "reviewer": "Reviewer_hpro", "summary": "This paper proposes a self-supervised pretraining approach to improve the robustness of a model against real-world sensor faults, focusing on an autonomous driving setup. Instead of a generic masking strategy, the authors consider a set of failure modes and devise a fault-aware masking strategy to address these. The approach is evaluated on a real autonomous driving system (Lexus LC 500), considering various failure modes.", "questions": "Please address the comments in the weaknesses section.", "limitations": "yes.", "rating": 2, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 4, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n1. Considering a fault-aware masking strategy appears effective for handling the domain-specific scenarios.\n\n2. The authors consider a set of real-world failure modes that are relevant to the autonomous driving setup.  \n\n3. The approach demonstrates effectiveness in a real autonomous driving system (Lexus LC 500).\n\n4. Evaluations consider known and unseen tracks, ablations, and sensitivity analysis of the approach.\n\n\nWeaknesses:\n\n1. The approach assumes the failure modes are known in advance. This may be appropriate in specific domains, but mostly does not hold in practical scenarios. \n\n2. How about composite failures? Can the pretraining on individual mode handle the situations while multiple sensor failures occur at the same time?\n\n3. How is the proposed approach fundamentally different from domain randomization, which is commonly used to adapt to novel scenarios in robotics? \n\n4. The loss function (equ. 3) imposes an equal weight to all failure modes. However, in real-world scenarios, this might not be the case as failures are evaluated based on their impacts.", "was_revised": true, "final_justification": "Thanks for conducting the additional experiments and providing additional results. I do appreciate the effort.\n\nWhile the numbers show the approach can handle some of the novel failures, I fail to understand what aspect of the approach allows this capability. There is no explicit mechanism to address novel failures.\n\nSimilarly, even though the experiments show that some of the composite failures can be handled, it is not what enables this. As the augmentations are made independently, and the loss function is a linear combination of the individual losses, the ability to capture composition is limited.\n\nI was hoping to see some technical reasoning for these two scenarios. Experimental results alone may not be sufficient, as we don't know how orthogonal or similar the failures are. As these are crucial for real-world applications, I stick to the original score.", "rebuttal": "We thank the reviewer for the valuable feedback, for recognizing the relevance of our approach in the autonomous driving setup, and for raising important questions regarding its assumptions and generalizability.\n### **Weaknesses**:\n\n**W1**: We would like to clarify that Sec. 3.1 describes a set of general sensor outages, building on a body of work from NASA [Balaban et al., 2009] and a work about wireless sensor networks [Jesus et al., 2017], distinct from our driving application. Training with these finite general failures improves sensor outage robustness in driving, and we speculate that it could bring improvements in other domains, such as GNSS positioning (where multi-path effects are present).\n\n1) To test the generalizability of our model, we evaluated it on the previously unseen failure mode of \"scaling,\" demonstrating that the model better generalizes to novel sensor faults not seen during training (Sec. 5.1).\n2) Additionally, in experiments not reported in the paper, we replaced some vehicle signals with data from an external Inertial Measurement Unit (IMU) with different noise statistics and biases to the signals in the training data. Here too, the reconstruction was of good enough quality to permit closed-loop control on the estimated sideslip angle in the presence of input disturbances similar to those in Fig. 6.\n\nIn response to your comment and to strengthen our evaluation, we conducted additional experiments on the Beijing PM10 dataset from the Time Series Extrinsic Regression (TSER) benchmark. This dataset involves predicting PM10 air quality levels in Beijing based on various environmental measurements, such as pollutant concentrations, temperature, and pressure. Our results demonstrate that the proposed method also improves robustness to sensor failures in this setting. For comparison, we report the XGBoost RMSE from the TSER benchmark website, which represents the strongest performance on the dataset under standard (non-failure) conditions.\n\n**Beijing PM10 Benchmark - Mean RMSE ± Std across Sensor Failures**\n\n| Error Type   | Baseline         | PT Baseline      | Ours              | XGBoost (RSER BM) |\n|--------------|------------------|------------------|-------------------|-------------------|\n| bias         | 117.63 ± 21.06   | 127.22 ± 27.48   | **104.89 ± 6.29*** | -                 |\n| mean         | 107.41 ± 12.36   | 103.29 ± 3.68    | **101.07 ± 5.21*** | -                 |\n| noise        | 98.19 ± 6.92     | 108.20 ± 6.78    | **97.52 ± 2.59***  | -                 |\n| scaling      | 359.13 ± 756.87  | 402.42 ± 843.38  | **117.44 ± 27.10***| -                 |\n| no_failure   | 95.06            | 97.98            | 94.05              | **93.13***        |\n\n\\* Lowest error per row marked in bold.\n\nOur method enhances robustness to sensor failures in the driving domain and clearly improves robustness in estimating air quality in the Beijing PM10 dataset. Its general applicability to other domains remains an open question, which we plan to explore in future work. In response to your concern, we will clarify point 1), and include a discussion of point 2) along with an evaluation with the additional benchmark in the appendix.\n\n**W2**: Thank you for the suggestion, and we agree that the composite failure setting is both interesting and valuable to study.\n\nWe focused on individual sensor failures to isolate their effects on sensor outage robustness. However, our pretraining process does include composite failures. The masking strategy used during pretraining samples multiple sensor dropouts per sequence (see Sec. 4.1), and the model is trained under composite failure conditions, and therefore, to a certain extent, robust to several simultaneous sensor failures.\n\nTo directly address your comment and improve the evaluations, we conducted new experiments evaluating performance under composite failures at test time. We randomly select 1 to 4 sensors to fail in parallel and evaluate 10 configurations per failure count, to reflect the fact that certain sensor combinations can be more detrimental than others. Below, we report the average MSE under 10 different coupled sensor failure configurations - tested on 1-4 simultaneous failures.\n\n**Performance Across Coupled Sensor Failures**\n\n| Failing Sensors | Baseline | Baseline PT | Baseline Ours |\n|------------------|----------|--------------|----------------|\n| 1                | 0.1037   | 0.0920       | **0.0443***     |\n| 2                | 0.1775   | 0.1831       | **0.0994***     |\n| 3                | 0.1386   | 0.1866       | **0.0843***     |\n| 4                | 0.2525   | 0.2244       | **0.1430***     |\n\n\\* Lowest error per row marked in bold.\n\nOur method consistently improves robustness compared to the zero-masking baseline (except in the case of pure mean-value outages) across nearly all composite failure settings. As the number of failing sensors increases, all models naturally degrade in performance, but the proposed method consistently improves performance over the baselines in the other composite failure setting.\n\nWe appreciate the reviewer’s suggestion and will include the full evaluation setup and extended results in the appendix of the final version.\n\n**W3**: Our approach shares the goal of improving robustness, but differs from domain randomization in two key respects:\n\n1) Data source: Domain randomization typically perturbs simulated environments by randomizing physical or visual parameters. Our method applies structured corruptions to real-world data to mimic sensor failure modes, and is thus closer to data augmentation.\n2) Goal and training objective: Domain randomization is often used to train models that are robust to distribution shifts in downstream tasks (e.g., locomotion across varied terrains). Our approach uses a self-supervised pretraining scheme that explicitly trains the model to recover original signals from fault-corrupted inputs. This aligns more closely with denoising and masked modeling approaches than traditional domain randomization.\n\nWhile different, the two approaches are complementary and could be combined to further improve robustness. Additionally, we will clarify the distinction in the related work section by adding a paragraph about data augmentation.\n\n**W4**: We agree that with knowledge of failure modes probability or the downstream task and how various errors will affect it, we could tune or learn the weights on the different failure modes as you suggest. For generality's sake, we decided not to fine-tune the loss function in Eq. (3) to the downstream control objective. As reported in the response to W1, using the same loss yields significant robustness benefits in a radically different application of predicting air quality. While it would be interesting to investigate the impact of the weights in the loss in future work, current results already show significant improvements in robustness compared to prior work without bespoke tuning of the loss."}, {"review_id": "HAt20mOZm2", "reviewer": "Reviewer_UAV2", "summary": "The paper presents a reconstruction pretraining schema for car sensory data which uses different sensor failures to mask the inputs. The goal is to achieve representations that are more robust towards such sensor failures. They test their approach on two vehicle sideslip angle (VSA) estimation tasks: an open-source data set and a real-world autonomous driving setting. They compare against no pretraining and traditional mean masking pretraining and determine that their approach achieves substantially more robust and thus accurate results.", "questions": "**Q1** Why did you not compare yourself against other time series pretraining methods like SimMTM? \n\n**Q2** How does your approach compare against more traditional methods like linear imputating the failed sensor value or Kalman Filtering of all sensor data? \n\n**Q3** What other tasks besides VSA estimation would benefit from more robust sensor representations in (autonomous) driving? And why did you choose VSA estimation as the sole evaluation task in this paper?\n\nIf all three questions are answered adequately, meaning to be either argued as non-significant or respective additions to the manuscript are made, I am willing to increase my score to accept.", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths:** \n\n**S1** The paper is very well written in terms of quality and clarity. As somebody who is not extensively familiar with the autonomous driving domain, the paper gave the right overview of different sensor failures and why they are important. \n\n**S2** The perturbation strategy is well-argued and seems to work well for robust representations of car sensory data. The ablation studies are extensive. \n\n**S3** The evaluation of the approach is done on an open-source data set, which is beneficial for recreating the results, and in an autonomous driving scenario, underscoring its transferability to real-world applications (even though the car drives in a relatively simple circle). \n\n**Weaknesses:** \n\n**W1** Both baselines are either relatively weak (no pretraining) or a subset of the proposed pretraining approach (mean masking).  Another pretraining method for time series data such as e.g. SimMTM [1] would be a stronger baseline.\n\n**W2** The evaluation, while extensive, focuses only on one task: VSA estimation. \n\n[1] Dong, J., Wu, H., Zhang, H., Zhang, L., Wang, J., & Long, M. (2023). Simmtm: A simple pre-training framework for masked time-series modeling. Advances in Neural Information Processing Systems, 36, 29996-30025.", "was_revised": true, "final_justification": "Questions Q1 and Q2 are answered sufficiently and provide comparison baselines that do not specifically train for more robust representations. \n\nIn terms of Q3, the authors included an air quality dataset. I can not comment on how comparable sensory failures modes for autonomous driving and air quality measurement are and if these two applications are representable applications for the sensory data domain in general. But I value the inclusion of a different domain dataset to showcase the generalisability of the method. However, I would suggest the authors to focus either only on the autonomous driving domain or, if claiming more broader generalisability, argue why their evaluations cover all relevant types of sensor data. As I am no expert in this domain, I will keep my score.", "rebuttal": "We thank the reviewer for the constructive feedback, valuable questions, and positive assessment of the paper’s clarity and experimental design.\n\n### **Weaknesses**:\n\n**W1 and Q1**: Thank you for the suggestion. We agree that comparing against recent time-series pretraining methods offers valuable insight, especially regarding nominal performance and robustness.\n\nOur primary goal was to isolate the effect of adapting the pretraining task to handle sensor failures, rather than to outperform the best nominal models. To do this, we used the same architecture with and without robustness-focused masking, emphasizing the impact of our approach. Most existing methods, such as Sim-MTM and Ti-MAE, do not explicitly address robustness or sensor outages.\n\nNonetheless, we conducted additional experiments with Sim-MTM. Since it was originally evaluated on classification and forecasting, we adapted it for the state estimation setting. While Sim-MTM performs well under nominal conditions, our method shows superior performance under sensor outages, underscoring the value of robustness-aware pretraining. A summary of results is provided below, with full details to be included in the appendix of the paper. The table shows the MSE of the lateral velocity estimation (as in Fig. 3), averaged across all failing sensors for a specific failure mode.\n\n**Comparison of Our Method vs. SimMTM Across Failure Types**\n\n| Failure Type | Ours     | SimMTM  |\n|--------------|----------|---------|\n| Bias         | **0.0484*** | 0.1352  |\n| Mean         | **0.0536*** | 0.1203  |\n| Noise        | **0.0283*** | 0.0423  |\n| Scaling      | **0.0433*** | 0.1009  |\n| No failure   | **0.0259*** | 0.0400  |\n\n\\* Lowest error per row marked in bold.\n\nSim-MTM was evaluated using the original paper’s recommended hyperparameters. While further tuning might boost nominal performance (improving the absolute errors), we don’t expect significant gains in robustness, as the method isn’t designed for such scenarios. We hope this also addresses Question 1 sufficiently. \n\nWe appreciate the reviewer’s suggestion and believe these additions strengthen the paper.\n\n**W2 and Q3**: Robust sensor representations could benefit a range of tasks beyond VSA estimation, such as traction control (e.g., estimating wheel slip), road condition monitoring (e.g., classifying wet or icy surfaces), and tire or brake wear estimation. In general, our approach might be applicable to many sensors whose signal might degrade or be missing, including ride height sensors or pushrods in race cars, and position estimation with satellite information.\n\nWe chose VSA estimation because it is a well-established and challenging task in the literature, making it an effective and communicable benchmark. It also plays a central role in understanding vehicle dynamics. We will clarify this in the experiment section.\n\nIn response to your comment, we conducted additional experiments on the Beijing PM10 dataset from the Time Series Extrinsic Regression (TSER) benchmark. This dataset involves predicting PM10 air quality levels in Beijing based on various environmental measurements, such as pollutant concentrations, temperature, and pressure. Our results demonstrate that the proposed method drastically improves robustness to sensor failures in this setting. For comparison, we report the XGBoost RMSE from the TSER benchmark website, which represents the strongest performance on the dataset under standard (non-failure) conditions.\n\n**Beijing PM10 Benchmark - Mean RMSE ± Std across Sensor Failures**\n\n| Error Type   | Baseline         | PT Baseline      | Ours              | XGBoost (RSER BM) |\n|--------------|------------------|------------------|-------------------|-------------------|\n| bias         | 117.63 ± 21.06   | 127.22 ± 27.48   | **104.89 ± 6.29*** | -                 |\n| mean         | 107.41 ± 12.36   | 103.29 ± 3.68    | **101.07 ± 5.21*** | -                 |\n| noise        | 98.19 ± 6.92     | 108.20 ± 6.78    | **97.52 ± 2.59***  | -                 |\n| scaling      | 359.13 ± 756.87  | 402.42 ± 843.38  | **117.44 ± 27.10***| -                 |\n| no_failure   | 95.06            | 97.98            | 94.05              | **93.13***        |\n\n\\* Lowest error per row marked in bold.\n\nOur method enhances robustness to sensor failures in the driving domain and clearly improves robustness in estimating air quality in the Beijing PM10 dataset. Its general applicability to other domains remains an open question, which we plan to explore in future work. In response to your concern, we will add an evaluation with the additional benchmark in the appendix. We hope this also addresses Question 3 sufficiently.\n### **Questions**:\n\n**Q2**: This is a very good suggestion. Relevant linear imputation methods familiar to the authors hinge on knowing where data is missing. In our application, this is not known, and would necessitate non-trivial fault detection schemes.\n\nKalman filters (KFs) are more readily applicable, but make several strict assumptions, such as Gaussian i.i.d. noise statistics and the absence of unmodeled disturbances on the measurements. The sensor failures that we are considering violate both of these assumptions and can lead to significant performance degradation in the estimates.\n\nTo illustrate this, we run a KF on the data from the real-time experiments in Fig. 6. To implement the KF, we need to assume knowledge of a carefully tuned physics model, where the sideslip angle is a state and observable in measurements of longitudinal velocity, yaw rate and rear wheel speeds. For this purpose, we use bicycle dynamics with a Fiala tire model. We then implement a nonlinear KF and estimate the side slip angle using the disturbed wheel speeds in the experiment in Fig. 6. We obtain the following average and maximum errors in its reconstruction.\n\n| Method                   | RMSE     | MAE     |\n|--------------------------|----------|---------|\n| Baseline (no pretraining)| 0.047    | 0.462   |\n| KF (w/ physics model)    | 0.018    | 0.170   |\n| Proposed (w/ pretraining)| **0.006*** | **0.030*** |\n\n\\* Lowest error per column marked in bold.\n\nDespite not having access to the physics-based motion model, the proposed pretraining method yields better reconstruction results. We will include these additional experiments with details on the implemented KF and the vehicle model in the appendix of the final version, and thank the reviewer for suggesting the comparison."}]

Author rebuttal:

Meta-review: The submission initially made a mixed impression on the reviewers. After the authors’ responses and the discussion phase, the submission ended up with two borderline ratings (2x borderline accept), one rating of reject and one for acceptance. Ultimately, based on the content of the reviews (not only the ratings), and taking into account also the reviewers’ expertise, experience, and engagement, I find that there is enough support to accept the work to NeurIPS.
Decision: Accept (poster)