One Filters All: A Generalist Filter For State Estimation
Authors: Shiqi Liu
Wenhan Cao
Chang Liu
Zeyu He
Tianyi Zhang
Yinuo Wang
Shengbo Eben Li
Venue: NeurIPS 2025 poster ()
Keywords: State Estimation
Large Language Model
Bayesian Filtering
Primary area: other

Abstract: Estimating hidden states in dynamical systems, also known as optimal filtering, is a long-standing problem in various fields of science and engineering. In this paper, we introduce a general filtering framework, $\textbf{LLM-Filter}$, which leverages large language models (LLMs) for state estimation by embedding noisy observations with text prototypes. In a number of experiments for classical dynamical systems, we find that first, state estimation can significantly benefit from the knowledge embedded in pre-trained LLMs. By achieving proper modality alignment with the frozen LLM, LLM-Filter outperforms the state-of-the-art learning-based approaches. Second, we carefully design the prompt structure, System-as-Prompt (SaP), incorporating task instructions that enable LLMs to understand tasks and adapt to specific systems. Guided by these prompts, LLM-Filter exhibits exceptional generalization, capable of performing filtering tasks accurately in changed or even unseen environments.  We further observe a scaling-law behavior in LLM-Filter, where accuracy improves with larger model sizes and longer training times. These findings make LLM-Filter a promising foundation model of filtering.

One Filters All: A Generalist Filter for State Estimation
Shiqi Liu'
Wenhan Cao
Chang Liu²
Zeyu He' Tianyi Zhang'
Yinuo Wang'
Shengbo Eben Li
School of Vehicle and Mobility, Tsinghua University College of Engineering, Peking University College of AI, Tsinghua University 'Equal contribution
*Corresponding author
Abstract
Estimating hidden states in dynamical systems, also known as optimal filtering, is long-standing problem in various fields of science and engineering. In this paper, we introduce general filtering framework, LLM-Filter' which leverages large language models (LLMs) for state estimation by embedding noisy observations with text prototypes. In various experiments for classical dynamical systems, we find that first, state estimation can significantly benefit from the reasoning knowl- edge embedded in pre-trained L.MMs. By achieving proper modality alignment with the frozen LLM,, L.M--Filter outperforms the state-of-the-art learning-hai based approaches. Second, we carefully design the prompt structure, System-as- assPro (SaP), incorporating task instructions that enable the LLM to understand the es- timation tasks. Guided by these prompts, LLM-Filter exhibits exceptional gen- eralization, capable of performing filtering tasks accurately in changed or even unseen environments. We further observe scaling-law behavior in LLM-Filter, where accuracy improves with larger model sizes and longer training times. These findings make LLM-Filter promising foundation model of filtering.
Introduction
State estimation of dynamical systems crucial topic in various fields, including robotics [1], meteorology [2]. chemistry [3], and transportation [4]. The most comprehensive framework for [5], which performs online estimation by iteratively applying prediction and update steps. Popular online Bayes filters be categorized into Gaussian filters [6] and particle filters (PFs) [7]. high-dimensional non- non-Gaussian systems, Gaussian filters tend to produce significant emrors [8, 9]. whereas PFs are limited by their substantial computational demands [10, 11].
state estimation is Bayesian filtering
To address these limitations, there is growing trend toward learning-basse based filtering methods [12, 13, 14, 15]. Trained offline on well-curated datasets, these learning- based filters directly capture accurate system statistics rather than relying manual modeling, enabling high accuracy and efficient online estimation. However, these filters are typically tailored to specifie tasks and require retraining when applied to different dynamical systems, as illustrated in Figure 1. Without retraining, their performance often deteriorates when the system undergoes changes or transitions to entirely new environments.
The control domain faces analogous challenges when adapting to diverse tasks and environments [16, 17]. Large control models have been proposed to address these challenges, with examples
'The code and dataset publicly accessible https//gittub.oo//Anonnoooo User User- L3e722336/ --iiter....r...
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Conventional Learning based Filter
LLM-Filter
Observations
Observations
bsservations st
Observations R
S Filter"
Filter
Prompts
LLM-Filttr
Prompts
Estimates
Estimates
Estimates
Estimates
Comparisons of LLM-Filter with Conventional Learning based Filters. The left subfigure shows conventional learning- based filters, which rely on specific system data and training. The right subfigure shows LLM- which generalizes across different systems through prompt guidance, without requiring retraining.
Figure I:
like RT [18, 19] and OpenVLA [20]. By leveraging the large-scale pretraining knowledge of large language models (LLMs) and vision-languse models (VLMs), these large control models demonstrate effective generalization across diverse robots, tasks, and environments [21]. is worth noting that, control is intrinsically dual problem of filtering, first demonstrated in linear systems through Lagrangian duality [22, 23], and later extended to nonlinear systems [24, 25, 26]. In the filtering domain, to the best of the authors' knowledge, no filtering method has yet integrated LLMs to utilize pre-training knowledge, nor has a general filtering model been developed.
To fill this research gap, we propose LLM-Filter, generalist filter that reprograms LLMs for state estimation while preserving the integrity of the backbone model. The process begins by embedding noisy observations as text prototypes, which are subsequently processed by frozen LLM to harmonize disparate data modalities. To enable generalization cross systems, we augment inputs with System-as-Pront The outputs generated by the LLM are then mapped onto the final state estimates. By leveraging the pretraining knowledge of the LLM and guiding it through carefully designed prompts, LLM- Filter can outperform most ffate-of-the-as generalization, even when applied to completely new systems, demonstrated in our experiments. Our main contributions are as follows:
(SaP), which includes task-specific instructions and task examples.
learning-basse based filters and exhibit remarkable in-context
We demonstrate that state estimation can significantly benefit from the knowledge embedded in pre-trained language models. By establishing proper modality alignment between the filtering process and the inference mechanism of LLMs, we propose a generalist filter, called LLM-Filter.
We carefully design an in-context approach, SaP, to help the LLM understand the current task and adapt to the specific application system. Guided by SaP, LLM-Filter exhibits exceptional generalization, effectively performing filtering tasks in unseen scenarios.
In our experiments, LI-M-Fiter consistently outperforms the state-of-the-art learning-basee -aaed filters and demonstrates remarkable in in-context generalization in changed or entirely new systems without retraining fine-tuning. Additionally, observe scaling-law behavior, where accuracy improves with larger model sizes and longer training times.
2 Preliminaries
2.1 Formulation of State Estimation
We begin with a brief review state estimation and the primary filtering methods considered in this work. Typically, state estimation is modeled using Markovian state-space model:
xt+1 f (xx) + E y:- = h (x:) + Ce:
(Ia)
(lb)
for / E (1, Here f(.) and h(-) represent the state transition and observation models, re- spectively. The term a, E M defined as the state while y, E R denotes the observations. Furthermore, and ç, represent the transition noise and observation noise, respectively, with their

---

probability distributions generally assumed to be known, This state-space model description in (1) can be represented hidden Markov model:
26 ~ p(
p( 1);
y/- ~ ((y,/ar))
where, ((=1)) and p(y, zx)) are the transition and output probabilities, respectively. and p( denotes the initial state distribution. Essentially, both (1) and (2) provide different formulations of the same underlying system model. The objective of state estimation is to extract the information of state æ, on all available observations V1.t*
2.2 Online Bayesian Filtering
A principled framework for state estimation is Bayesian filtering, which computes (y..) recur- sively through two steps:
p = / p
(3a)
ply, ( 1
p( p(y, x (p(2) 11:111)
(3b)
where, (3a) is called prediction step while (3b) is called the update step. Under the assumptions of linearity and Gaussian noise, (3) can be analytically calculated online, which is the basis of Kalman filter (KF) [27], as detailed Appendix A. However, these filtering methods are highly dependent on accurate modeling f() and h(-), well the noise distributions of &, and S, specific to each system.
2.3 Learning-basse Filtering
Instead of performing online computations as in (3), learning- baased filtering methods are trained on well-collected data learn the correlation between observations and the underlying state. This learning is achieved by optimizing the following supervised learning loss to update the network parameters 0:
(()) = || d; 1y 0)1
(4)
where ă; ( 1.:: 0) represents the estimates, and æ; the ground truth. By capturing accurate system statistics from training data instead of relying on manual modeling. learning- bearning-based filtering methods often provide more precise estimates and greater efficiency. However, heavy reliance on task- specific datasets renders conventional learning- filters inflexible, requiring retraining whenever tasks change or transition to entirely new environments. To overcome this limitation, this paper aims to propose general filtering method that leverages in-context mechanism, enabling effective performance across multiple systems without retraining.
Methodology
The main structure of LLM-Filter is illustrated in Figure 2. We first align the modalities between the discrete tokens the L.M and continuous observations (Section.3. then design an in-context inference approach based on system information enhance generalization (Scction.3.2), finally obtain the estimates from the predicted tokens the LLM and define the loss function (Section.3.3).
3.1 Observation Embedding
Moving Horizon. As described in Section.2.1 the objective is to estimate the state æ; on all available observations 31.c- However, it is impractical to input an ever-growing sequence of observations. Inspired by the moving horizon estimation [28] and the context window of LLMs [29], we design sliding window with a fixed length T for estimation. The input observations are defined Y, ( E T and the corresponding output estimates are X; = { E R T with X, ( E RTxM representing the ground truth states, where N and M denote the dimensions of the observation and state variables, respectively.

---

In- Context Inference
State Projection
Observation Embedding
BEGINS
X
Segmening
[Tesk Instraction]
MSE
&
[Task Examples)
StatePriection
51
<ND
Pre-trained LLM
Nor
Pre trained LLM Tokenizer
ObsEmbedding
11N
Figure 2: Framework of LLM-Filter. (1) Observation Embedding: Observations are segmented and embedded into token embeddings. (2) In-context Inference: In-context tokens from the tokenized SaP context are concatenated with the observation embeddings. (3) State Projection: The LLM's predicted tokens are projected into the state space to obtain the final estimates.
Segment and Embedding. To align the continuous observations with the discrete tokens, segment the input observations and embed them into LLMs. A popular segmentation method uses single-series sequence format [30, 31J, which involves directly flattening all data dimensions. However, this approach can disrupt the inherent correlations between variables, such as the crucial relationship between position and speed, which vital for accurate position estimation. To preserve the inherent correlations, we embed Y, by segmenting it based on segment length while preserving its milti-dimensional structure:
T
Y = {s1 82
SN. N₂
where each segment 8; E L (for 1.2,..., N.) and |-] denotes the floor function. the division is not exact, padding is applied at the end to ensure all segments have the same length L. To leverage the pretraining knowledge and the intrinsic token transitions of the LLM, we freeze the parameters and eliminate the embedding and projection layers designed for language tokens. Instead, we introduce OlsEmbeddingg() : LRN + R DxN which independently embeds each normalized segment into the D-dimensional latent space of the LLM:
e, ObsEmbeddinggNorm
i= 1,2, ...... No.
3.2 In-context Inference
Traditional carning-based filters estimate the true system states solely from observational data, as discussed in Section 2.3. However, this re- liance limits their ability to identify underly- ing system dynamics, resulting in poor gener- alization to new systems. To address this, we build upon the in-context learning capabilities of LLMS [33] and introduce novel prompting strategy called SaP, which guides L.LM-Filter to adapt flexibly to different systems. SaP is grounded in two components necessary for craft- ing effective prompts for LLM-Filter: Task In- struction and Task Examples. Task Instruc- tion equips LLM-Filte with crucial contextual knowledge, which may differ across various do- mains. Task Examples guide L.M--Filter by showcasing example scenarios that clarify the task at hand. An example depiction of SaP can be found Figure 3. When inference, the SaP text will be fed to the pre-trained LLM tokenizer for processing language text:
<BEGIN>
[Task Instruction]
This task estimates the next state from a sequence of obscrvations, where each observation represents the system's input states over time. [Task Examples]
Input Observations (First 5):
[[-1.4645, 1.9252][[-1.4733 1.9189][-1.4811,
1.9124)[-144055, 1.9059).[-1.4998 1.8993]]
True Output States (First 5):
[[-1.3500 0.6519][-113646 0.6308)[-1.379, 0.6094)1-11.3911 0.5878], [-1.4069 0.5659]]
<END>
Figure An Illustration of SaP for Hopf [32] system.
(cr,--- [N_] = LLMoookenzerr(C),
where N. is the number of tokenized context tokens.

---

3.3 State Projection
Token Prediction. Prevalent LLMs, trained on diverse downstream tasks, excel at predicting the next token based on previous ones across various domains [3J, By inputting the guiding SaP and the embedding token of a previous observation, we aim to derive state estimation features for the next period. To achieve modality alignment, we utilize only the core layers of the LLM by excluding the original embedding and projection layers for language tokens. The tokenized SaP context and observation embeddings (c1,-.-, CN., e1. : eN,, are directly fed into the core layers, which then generate the corresponding output token embeddings {éi, e:.):
È LLLMCoreLayers (fc),-.. CN., e1 ,,,,,, fēr, .. (ev.) = E(NC+INN+NN..
(5)
Output Projection. Afterward, we employ the layer StateProjection(() R PxN + R to project the output embeddings {ei, ",,, onto the system state space, thereby obtaining the final estimates X²:
X, SlateProjection(((),
,,,,)):
where both ObsEmbeedding()) and StateProjectionns has been shown to effectively embed and project time series for L.M [30].
implemented using simple MLPs, which
Loss Function. Finally, the overall objective is to minimize the error between the ground truth X and the output estimations X, to optimize the parameters & of LLM-Filler Fe:
Co - - X,
x. FY 1)....
This loss function can be seen extension of (4), incorporating additional context input. Notably, to preserve the pre-trained knowledge and due to resource constraints, we freeze the LLM and only update the parameters of ObsEmbedding(() and StateProjection. The detailed configuration of LLM-Filter is provided in Appendix D.2.
Remark 1. When the SaP prompt is not available, LLM-Filter can still operate without in-context guidance, variant we refer as LLM-Filter-.. However, due the lack of contextual guidance, its filtering accuracy and generalization capability may degrade, as demonstrated in the comparison provided in Section
4 Related Work
Online Bayesian Filtering. The field of state estimation can be traced back to the pioneering development of the KF [27], which was famously applied to NASA's Apollo guidance and navigation systems at the Ames Research Center [36]. This milestone led to advanced methods like the extended Kalman filter (EKF) [37], unscented Kalman filter (UKF) [38], enabling applications in nonlinear systems. However, their reliance on linearization and Gaussian assumptions can introduce significant errors in complex scenarios. In response, particle-based based filtering methods were developed, including the PF [39] and the ensemble Kalman filter (EnKF) [40]. Unfortunately, particle-base based approaches face high computational demands and the curse of dimensionality.
Learning- Filtering, Recent advances in neural network integration have driven the devel- opment of tearning-based filtering methods. For instance, the KalmanNet [13, 41] adopts RNN to compute Kalman gains, significantly improving computational efficiency and accuracy. Similarly, MEstimator [14] and RStateNet [15] utilize multilayer perceptrons (MLPs) and long short-term memory networks (LSTMs) [42] to process historical state ensembles, enriching input information and achieving higher precision. Prr [43], on the other hand, employs attention mechanisms to model system dynamics in the latent space, enabling long-term estimations. However, these methods rely solely observational data input and require retraining when applied to systems. In contrast, our proposed LLM-Filter aligns the modalities language and state estimation, allowing generalize to unseen systems through in-context guidance without additional training.
Large Control Model. As the dual problem of state estimation, there is an increasing trend toward training multi-task generalist control models large, diverse datasets across different embodiments.
5

---

Recent work has incorporated LLMs and VLMs into control [16, 34], focusing primarily on high- level planning while underutilizing the knowledge from large-scale models. To address this, the RT model [18, 19, 21] directly train VLMS designed for open-vocabulary visual question answering to output low-level robot actions. Building on this architecture, Open VI.A [20] streamlines the model's 's parameters while improving performance and exploring further fine-tuning strategies for generalization. The rapid advancement of large control models drives our ambition to develop a generalist state estimation model that leverages pre-trained LLMs' knowledge.
LLMs for Time Series Forecasting. State estimation and time series forecasting both use sequen- tial data. However, state estimation estimates hidden states from noisy observations, while time series forecasting predicts future values from the same data, Recent studies have made significant breakthroughs, especially in zero-shot forecasting, by establishing mappings between LLM tokens and numerical data [44, 45]. Additionally, Timel.I. [46]has explored the in-context forecasting capabilities of L.Mss, leveraging an in-context learning to introduce a variety of forecasting tasks. AutoTimes [30] demonstrates how timestamp encoding techniques can unlock further potential for LLMs in time-series applications. While these models primarily predict future values based on past data, LLM-Filter focuses on estimating hidden states from noisy observations in dynamic systems.
5
Experiments
The goal of our experimental evaluations is to answer the following questions:
Can LLM-Filter learn to filter in canonical estimation task? (Section.5.1)
2. Can LLM-Filter effectively generalize to new systems? (Section.5.2)
3. Do the inherent characteristics of LLMs manifest within LLM-Filter? (Section.5.3)
Benchmarks. To comprehensively evaluate the performance of LLM-Filter, we conducted experi-
ments on two types of classical dynamical systems: (1) low-dimensional nonlinear systems, and (2)
high-dimensional chaotic systems, which are commonly used as benchmarks in previous studies [47,
48, 2]. Specifically, the din-dimeniional nonlinear systems includes Selkov [49], Oscillator [50],
Hopf [32], and Double Pendulum [51] systems. For high-dimensional chaotic systems, we performed experiments on Lorenz96 [52] and VL20 [53], both of which are widely used in atmospheric dynamics research. For the generalization experiments, we also include the Tracking system [5]. Detailed descriptions and configurations of each system are outlined in Appendix B. We utilize root mean square error (RMSE) and average runtime (ms/step) as the primary evaluation metries, with their precise definitions provided in Appendix D.3.
Baselines. We compare LLM-Filter with state-of-the methods, including online Bayes filters: the PF [10]. the EnKF [40], the average-particls ensemble Kalman filter (EnKFI) [48], the soft ensemble Kalman filter (EnKFS) [48], and the huber ensemble Kalman filter (HubEnKF) [54] as well as the
learning- based methods: MEstimator [14], RStateNet [15]. Pro'Tran [43], and KalmanNet [13]. For
fair comparison, the hyperparameters of all online Bayes filters are selected during the first trial using the Bayesian Optimization (BO) package [55]. Detailed descriptions of these methods are provided
in Appendix D. Unless otherwise stated, LLaMA-7B [56] is used as the base LLM in LLM-Filter.
5.1 Results for Canonical Estimation Task
In this experiment, we evaluate the fundamental filtering capabilities of LLM--Flter and LLM--Filte--O on canonical estimation tasks, comparing their performance to state-of-the-art methods including MEstimator, RStateNet, ProTran, KalmanNet, EnKF, and PF.
Low-dimensional Nonlinear Systems. We first conducted experiments on low-dimensional nonlinear systems. A summary of the results is presented in Table 1, where LLM-Fiiter consistently outperform all other methods across all tested systems. Notably, the RMSE of the LLM-Filter shows a maximum reduction of 32.00% across all systems. On average, it demonstrates an improvement of 17.93% over the best-performing learning-bass filtering methods. When compared to the top-performing online Bayes filter, LLM-Filter still achieves an average improvement of 21.65%. In addition, with the use of the in-context prompt, L.LM-Filte achieves 10.18% improvement over the data-only LLM-Filter-O. O, indicating that in-context prompting helps LLM-Filter better understand the system

---

and perform more accurate state estimation. These results highlight the strong filtering capabilitiess of the LLM-Filted in low-dimensional nonlinear systems, establishing its state-of-the-an performance.
Table RMSE of State Estimation on Classical Systems. The best performances are marked in bold, and the second-best performances underlined. Results labeled as NaN indicate cases where the models diverged during training.
Learning-Base Based Filters
Online Bayes Filters
Method
LLM-Filter
LLM-Filter-O
MEstimator
RStateNet
ProTTran
KalmanNet
EnKF
PF
Selkov
0.4061
0.6369
0.8864
0.7202
1,0219
1.1662
0.5978
0.6863
Oscillator
0.5247
0.5753
0.8347
0.8493
0.8933
0.5665
0.550S
0.6807
Hopf
0,5751
0.8180
0.8290
0,7282
0.7146
1.1984
0.6322
0.6801
Pendulum
0.8348
0.9218
0.9354
0,9180
0.8456
2.7140
1.4117
5,3788
Lorenz96
0.9149
0.9735
0.9649
0.9762
0,9975
NaN
6,6024
4.6289
VL.20
0.7717
0.8433
1.0014
0.9428
0,9902
NaN
5.8633
11,6980
Average
0.6712
0.7948
0.9086
0.8558
0,9105
NaN
2.6097
3.9588
Runtime (motcoe))
MStateNet
LLM-Filter
EnKF
2.51
2.22
2.94
200
0.88
0.63
0.96
0.95
0.93
0.05
0.13
2.1G
30-1
Selkow
Oscilator
Hopf
Pandullum
Average
Figure 4: Comparison of Estimation Runtime Between LLM-Filter, EnKF, and RStateNet. Compre- hensive results are presented in Tab 6.
High-dimensional Chaotic Systems. We then performed experiments on the Lorenz96 and VI.20 systems, both with 72 state dimensions, designed to simulate atmospheric dynamics. The results are summarized in Table 1. LLM-Filter achieves state-of-the-ard performance on the Lorenz96 and VL20 systems, with an average improvement of 11.7% over the best baseline filters. In contrast, the KalmanNet, as implemented in the official codebase". ultimately suffers from divergence during training. These results validate that LLM-Filter is capable of effectively addressing high-dimensional chaotic systems.
Running Time, The running time a crucial factor in practical filtering applications. For learning- based filters and online Bayes filters, we selected RStateNet and EnKF, respectively, as representative methods for comparison. The results are summarized Figure For all systems, the running time of the LL-M-Filter is slower than RStateNet, which has fewer parameters. However, LLM--Fiter is faster than the online EnKF, making it a highly practical and effective solution for real-time applications. As the system dimension increases, both EnKF and RStateNet show significant increase in computation time. In contrast, LLM-Filter maintains stable runtime around 1.0mss which highlights the LLM-Filter's efficiency in nigh-dime system estimation, offering clear advantage for such applications. Detailed information about the testing equipment configuration, model parameters, and peak memory can be found in Appendix D and Table 5.
5.2 Generalization Evaluation for Estimation Task
In this experiment, we assess the cross adastaadaptttion capability of our proposed Filter- its performance when applied modified system even entirely different system.
Model Mismatch. Model mismatch, common challenge in filtering tasks, occurs when the assumed observation noise variance differs from the actual noise variance [13, 48]. J. We evaluated LLM-Filter under model mismatch on the Selkov, Oscillator, and Hopf systems, using the observation covariance expansion ratio (OCER) to quantify mismatch severity. We compared LLM-Filter with the robust filters designed for model mismatch, including EnKFI, EnKFS, and HubEnKF. The results, shown in Figure The performance of the robust filters rapidly declined as the OCER increased, while
"The official open-source repository available at https://github.com Kalaeeooo//////aaaaaeaaaaaa optimizing the typeraarameters and network architecture, KalmanNet still struggles to filter high -iseeiimnsional systems, potentially due to the large dimensionality involved computing the Kalman gain.
TSP. Despite
7

---

LLM-Filter maintained consistent accuracy with minimal degradation. These findings underscore the LLM-Filter's exceptional robustness and adaptability with varying observation conditions.
EnKFI
EnKFS
HubEnkF
LLM-Filler
Selkav
Oscillatar
Hopf
3.0
Average
4
2.0
2.5
RMMS
3-
2-
1.5
2.0
1.5
2
1.0
1.0
0.5
0.5
25
50
75
100
25 50
75
100
25
50
75
100
1
25
50
75
100
Observation Corvariance Expansion Ratio
Figure 5: Comparison of RMSE Results across the Selkov, Oscillator, and Hopf systems as the observation covariance expansion ratio increases. The rightmost subfigure presents the average results across these systems. Complete results presented in Table 7.
Tracking Penduluum
Pendulum Tracking
Oscillator Hopf
Hapf Oscillator
5.09
3.86
2.54 2.75 3.02
LLM Filer
LLM-FIlte-O MEstimator
RMSE
2.74
2.11
RStateNet
1.26
0.92
0.94
0:80
1.42
0.93
0.833 0.73
0.73
0.83
ProTran
Figure 6: RMSE for Cross-System Scenarios. The hollow bars represent cross-system scenarios, while the solid bars correspond to cases where the model is both trained and tested on the same system. In cross coss-system scenarios, Tracking - Pendulum indicates that the model is trained on the Tracking system and evaluated on the Pendulum system. Comprehensive results are presented in Tab 9.
Cross Systems. this experiment, we evaluate the performance on completely different systems, presented in Figure 6. The cross-system results are shown with hollow bars, while solid bars represent the performance baseline models trained directly on the target systems. We also include the LLM- Filter-O as an ablation comparison. Guided by the SaP prompt, LLM-Filter achieves exceptional performance, demonstrating the lowest RMSE across all methods in cross-system scenarios. When compared with specific training baselines, L.LM-Filter delivers competitive performance, even achieving the best results in the Oscillator + Hopf scenario. Without the guidance the SaP prompt, LLM-Filter-O shows noticeable decrease in its generalization capability compared to LLM-Filter. Moreover, the minimal difference between the solid and hollow bars for LLM-Filter indicates that maintains strong performance even when applied to previously unseen systems. In contrast, the other learning-basse based methods experience significant decline performance, even losing their ability to provide accurate estimates when confronted with unseen systems. These results demonstrate the potential of LLM-Filter to generalize effectively across new systems.
5.3 Exploration of LLMs
In this subsection, explore the scaling law and fine-tuning methods of LLMS for LLM-Filter. Additionally, we conduct an ablation study on the LLM component to emphasize its crucial role in achieving the model's success.
Scaling laws represent empirical observations about how performance and efficiency metrics vary model size increases [57]. In our study, we explore the scaling behavior of LLM-Filter in state estimation tasks by evaluating models of different parameter sizes as the LLM backbone. To avoid inconsistencies caused by varying tokenizers across models, we did not use in-context prompting this experiment. We examined the average RMSE and training efficiency across all low-dimensional nonlinear systems and high-dimensional chaotic systems, with results illustrated in Figure Our findings illustrate the scaling behavior of L.M--Filte: as model parameters increase, RMSE decreases, boosting estimation accuracy, though the cost of longer
Scaling Behavior.

---

Table RMSE for LLM-Filter Using LoRA
Table RMSE for LLM Ablation Study of LLM-
or Full Fine-Turme Tuning.
Filter.
Method
LLM--iitte
+LoRA
+Full Finetune
Method
L.M--ilter
MLP
RNN
Transformer
Selkov
0.4061
0.4498
0.7103
Selkov
0.4061
0.9910
0.9862
0.8863
Oscillator
0.5247
0,5370
0.5513
Oscillator
0.5247
0.8176
0,9481
0.8566
Hopf
0,5751
0.6744
0,8444
Hopf
0,5751
0.9057
1.0114
0.8610
Lorenz96
0.9149
0.9147
0.9140
Lorenz96
0.9149
0.9661
1.0105
0.9721
V1.20
0.7717
0.8962
0.7027
VL.20
0.7717
0.9977
0.8043
0.9598
training times. Furthermore, our experiments underscore the adaptability of LLM-Filter, as it delivers strong performance across a variety LLM bases.
Full Fine- Tine-Tuning and LoRA. This experiment in-
vestigates tWo prominent methods commonly used
1.00
adapt L.Mss more effectively for downstream tasks:
full parameter fine-tuning [58] and low-rank adapta- tion (LoRA) [59]. For LLM-Filter, full fine-tuning in- volves updating all model parameters, whereas LoRA
0.95 GPT-2
Average RMSE
OPT-1.3B
only updates low-rank matrix within the model. A summary of the results is shown in Table 2. Inter- estingly, equipping LLM-Filter with LoRA led to a decline performance. We attribute this to the limita- tions of low-rank matrix adjustments, which may lack the expressive power needed for the high precision required in state estimation tasks. On the other hand, a fully mne-tuned LLM-Filter demonstrated improved performance in more challenging environments, such as Lorenz96, and VL20 systems. This comes at a cost: in simpler dynamic systems, the performance was worse, likely due to overfitting from extensive parameter updates.
7B
LLaMA-78
0.75
LLaMA-13B
0.70
25
50
75
100
125 150
Training Time (ms/iter)
Figure 7: Comparison of RMSE and Effi- ciency Across Various LLMs. Complete re- sults can be found in Table 10.
LLM Ablation Study. In this experiment, we ad- dress whether the use of L.MSs is truly essential for state estimation tasks, concern raised in recent studies on L.MSs for time series analysis [60]. To investigate this, replace the LLM in LLM-Filter with traditional neural network architectures, including MLP, RNN, and Transformer, while keeping all other components unchanged. The results summarized in Table 3. Our findings clearly demonstrate that LLM-Filter consistently outper- forms these traditional network architectures across all tested systems. Specifically, the RMSE of LLM-Filter shows average improvement of 31.66%, 32.43%. and 30. 73% over MLP, RNN, and Transformer, respectively. This significant improvement underscores the superior performance and essential role of LL in state estimation tasks, validating their necessity for achieving state-of-the-art results.
Conclusion and Future Work
Inspired by the success of large control models, we propose general large filtering model, LLM-Filter, designed to solve specific estimation tasks, including generalization tasks, by transferring knowledge from LLMs. The process begins by embedding noisy observations with text prototypes and then enriching the input with the SaP method, which incorporates basic task instructions and characteristics. These inputs are then processed by a frozen LLM. The generated outputs are projected onto the final estimates. Guided by prompts and leveraging pretraining knowledge, LLM-Filter outperforms specialized learning- based methods and demonstrates strong generalization across various systems.
The generalization ability of LLM-Filter is currently restricted to systems with the same dimension- ality, and its evaluation across diverse settings still limited. Future work will focus on training large-scale state estimation models with multi-source data, aiming to enable generalization across systems with varying dimensions. We hope that our contributions will lay the groundwork for addressing state estimation challenges in wide range of real-world applications.

---

Acknowledgements
This work was supported by the Tsinghua-Eroo Joint Research Center for EAI Computation and Perception, and by the Anhui Province Science and Technology Innovation Plan under the project "Breakthrough and Industrialization L4-Level Multi-Modal Perception End-to-End Model" (No.
202423d12050007)
References
[1]
Timothy D Barfoot. State estimation for robotics. Cambridge University Press, 2024.
[2] Kevin Course and Prasanth B Nair. "State estimation of physical system with unknown governing equations". In: Nature 622.7982 (2023), pp. 261-267.
[3] Denis Dochain. "State and parameter estimation in chemical and biochemical processes: tutorial". In: Journal of process control 13.8 (2003), pp. 801-818.
[4] Jinlei Zhang et al. PPhysics-iffrmee deep learning for traffic state estimation based on the traffic flow model and computational graph method". In: Information Fusion 101 (2024),
p. 101971.
[5] Simo Särkkä and Lennart Svensson. Bayesian and smoothing. Vol. 17. Cambridge university press, 2023.
[6] Kazufumi Ito and Kaiqi Xiong. "Gaussian filters for nonlinear filtering problems". In: IEEE transactions on automatic control 45.5 (2000), pp. 910-927.
[7] Zhe Chen et al. "Bayesian filtering: From Kalman filters to particle filters, and beyond". In: Statistics 182.1 (2003), pp. 1-69.
[8] Hamed H Afshari, S Andrew Gadsden, and Habibi. "Gaussian filters for parameter and state estimation: general review of theory and recent trends". In: Signal Processing 135 (2017), pp. 218-238.
[9] Ondřej Straka, Jindfich Duník, and Victor Elvira. "Importance Gauss-Hermite Gaussian Filter for Models with Non-Additive Non-Gaussian Noises". In: 2021 IEEE 24th International Conference on Information Fusion (FUSION), IEEE. 2021, pp. 1-7.
[10] Peter Jan Van Leeuwen et al. "Particle filters for high-dimensional geoscience applications: review". In: Quarterly Journal of the Royal Meteorological Society 145.723 (2019), pp. 2335- 2365.
[11] Ziyu Wan and Lin Zhao. "DiffPF: Differentiable Particle Filtering with Generative Sampling via Conditional Diffusson Models". In: arXiv preprint arXiv:2507. 15716 (2025).
[12] Yujie Tang et al. "Reinforcement learning compensated extended Kalman filter for attitude estimation", In: 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE. 2021, pp. 6854.
[13] Guy Revach al. "KalmanNet: Neural network aided Kalman filtering for partially known dynamics". In: IEEE Transactions on Signal Processing 70 (2022), pp. 1532-1547.
[14] Gwanghyeon Ji et al. "Concurrent training of a control policy and a state estimator for dynamic and robust legged locomotion". In: IEEE Robotics and Automation Letters 7.2 (2022), pp. 4630-4637.
[15] Pragyan Dahal et al. "RobustStateNet: Robust ego vehicle state estimation for Autonomous Driving". In: Robotics and Autonomous Systems 172 (2024), p. 104585.
[16] Michael Ahn et al. "Do as i can, not as i say: Grounding language in robotic affordances". In: (2022).
arXiv preprint arXiv:2204.0166
[17] Octo Model Team et al. "Octo: An open-source generalist robot policy". In: arXiv preprint arXiv:2405. 12213 (2024).
[18] Anthony Brohan et al. "Rt-1: Robotics transformer for real-world control at scale". In: arXiv preprint arXiv:2212.0688 (2022).
[19] Anthony Brohan et al. "Rt-2: Vasion--anguuge-ene models transfer web knowledge to robotic control". In: arXiv preprint arXiv:2307.158818 (2023).
10

---

[20] Moo Jin Kim et al. "OA: An Open-Source Ssion-Language-Actiinn Action Model". In: arXiv preprint 2xX1:22406.09246 (2024).
[21] Abby O'Neill al. "Open x-embodiment: Robotic learning datasets and rt-x models". In: arXiv preprint (0t::2330.088000 (2023).
[22]
Thomas Kailath, Ali Sayed, and Babak Hassibi. Linear estimation. Prentice Hall, 2000.
[23]
Graham Goodwin al. "Lagrangian duality between constrained estimation and control".
In: Automatica 41 11.6 (2005), pp. 335-944.
[24] Emanuel Todorov. "General duality between optimal control and estimation".. In: 2008 47th
IEEE conference on decision and control. IEEE. 2008, pp. 4286-4292.
[25]
Jin Won Kim. "Duality for nonlinear filtering". In: arXiv preprint
(2022).
[26] Yinuo Wang et al. "OsE-based Smoothing Neural Network for Reinforcement Learning Tasks". In: The Thirteenth International Conference on Learning Representations. 2025.
[27]
Rudolph Emil Kalman. "A new approach to linear filtering and prediction problems". In: (1960).
[28] Angelo Alessandri, Marco Baglietto, and Giorgio Battistelli. 'Moving-horizon state estimation
for nonlinear discrete-time systems: New stability results and approximation schemes". In: Automatica 44.7 (2008), pp. 1753-1765.
[29]
Josh Achiam al. "Gpt-4 technical report". In: arXiv preprint arXiv:2303.0874 (2023).
[30]
Yong Liu et al. "Autotimes: Autoregressive time series forecasters via large language models".
In: arXiv preprint arXi::2402.02370 (2024).
[31] Yong Liu et al. "Timer: Generative Pre-trained Transformers Are Large Time Series Models".
In: Forty-first International Conference on Machine Learning. 2024.
[32] Brian D Hassard, Nicholas D Kazarinoff, and Yieh-Hei Wan. Theory and applications of Hopf bifureatiin. Vol. 41. CUP Archive, 1981.
[33] Shukang Yin et al. "A survey multimodal large language models". In: arXiv preprint arXiv:2306. 13.53066.339999 (2023).
[34] Danny Driess et al. "Palm-e; An embodied multimodal language model". In: arXiv preprint vrXiv:2303.03377 (2023).
[35] Jiaming Han et al. "Onellm: One framework to align all modalities with language". In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024, pp. 06584-26595.
[36] Mohinder S Grewal and Angus P Andrews. "Applications of Kalman filtering in aerospace 1960 to the present [historical perspectives]". In: IEEE Control Systems Magazine 30.3 (2010), pp. 69-78.
[37] Gerald L Smith, Stanley F Schmidt, and Leonard A McGee. Application of statistical filter theory to the optimal estimation position and velocity board circumlunar vehicle. Vol. 135. National Aeronautics and Space Administration, 1962.
[38] Simon Julier, Jeffrey K Uhlmann, and Hugh Durrant-Whyte. "A new approach for filtering nonlinear systems". : In: Proceedings of 1995 American Control Conference-ACCC955, Vol. 3.
IEEE. 1995, pp. 1628-1632.
[39] Jun S Liu and Rong Chen. "Sequential Monte Carlo methods for dynamic systems". In: Journal of the American statistical association 93, .4443 (1998), pp. 1032-1044.
[40] Pavel Sakov and Peter R Oke. "Implications of the form of the ensemble transformation in the ensemble square root filters". In: Monthly Weather Review 136.3 (2008), pp. 1042-1053.
[41] Xiaoyong Ni, Guy Revach, and Nir Shlezinger. "Adaptive Kalmannet: Data-Driven Kalman Fil- ter with Fast Adaptation". In: ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE. 2024, pp. 5970-5974.
[42] S Hochreiter. "Long Short-term Memory".. In: Neural Computation MIT-Press (1997).
[43] Binh Tang and David S Matteson. "Probabilistic transformer for time series analysis". In: Advances in Neural Information Processing Systems 34 (2021), pp. 23592-23608.

---

[44] Tian Zhou et al. "One fits all: Power general time series analysis by pretrained Im". In: Advances in neural information processing systems 36 (2023), pp. 43322-43333
[45] Nate Gruver al. "Large language models are zero-shot time series forecasters", In: Advances in Neural Information Processing Systems 36 (2024).
[46] Ming Jin al. "Time-llm: Time series forecasting by reprogramming large language models". In: arXiv preprini /rX00:23/0.01728 (2023).
[47] Patrick N Raanes, Yumeng Chen, and Colin Grudzien. "DAPPER: data assimilation with Python: a package for experimental research". In: Journal of Open Source Software 9.94
(2024), p. 5150.
[48] Gerardo Duran-Martin et al. "Outlier-robust Kalman Filtering through Generalised Bayes", In: arXiv preprint arXiv:2405.0566 (2024).
[49] Evgeni E SEL'KOV. "Self-Oscillltiinns in Glycolysis 1. A Simple Kinetic Model". In: Euro- pean Journal of Biochemistry 4.1 (1968), pp. 79-86.
[50] Steven Brunton, Joshua Proctor, and J Nathan Kutz. "Discovering governing equations from data by sparse identification of nonlinear dynamical systems". In: Proceedings of the national academy of sciences 113.15 (2016). pp. 3932-3937.
[51] RB Levien and SM Tan. "Double pendulum: An experiment in chaos". In: American Journal of Physies (1993), pp. 1038-1038.
[52] Edward N Lorenz "Predictability: A problem partly solved". In: Proc. Seminar predictabil- ity. Vol. 1. 1. Reading. 1996.
[53] Gabriele Vissio and Valerio Lucarini. "Mechanics and thermodynamics of new minimal model of the atmosphere", In: The European Physical Journal Plus 135.10 (2020), pp. 1-21.
[54] Soojin Roh al. "Observation quality control with robust ensemble Kalman filter". In: Monthly Weather Review 141.12 (2013), pp. 4414-4428.
[55] Jacob R Gardner et al. "Bayesian optimization with inequality constraints." " In: ICML. Vol. 2014. 2014, pp. 937-945.
[56] Hugo Touvron et al. "Llama: Open and efficient foundation language models". In: arXiv preprint arXiv:2302. 13971 (2023).
[57] Jared Kaplan et al. "Scaling laws for neural language models". In: arXiv preprint arXiv:2001. 003361 (2020).
[58] Jesse Dodge al. "Fine-tuning pretrained language models: Weight initializations, data orders, and early stopping". In: arXiv preprint arXiv:2002. 06305 (2020).
[59] Edward Hu et al. "Lora: Low-rank adaptation of large language models". In: arXiv preprint
10X. 0966009..555. (2021).
[60] Mingtian Tan et al. "Are language models actually useful for time series forecasting?" In: arXiv preprint arXiv:2406. 16964 (2024).
[61] Alberto Carrassi et "Data assimilation the geosciences: An overview of methods, issues, and perspectives". In: Wiley Interdisciplinary Reviews: Climate Change 9.5 (2018), e535.
[62] John Butcher. "Implicit runge-kutta processes". In: Mathematics of computation 18.85 (1964), pp. 50-64.
[63] Xinshuo Weng et al. "Ab3dmot: A baseline for 3d multi-object tracking and new evaluation metries". In: arXiv preprini arXiv:2008 08063 (2020).
[64] R OpenAL. "Gpt-4 technical report. arxiv 3007.08774". In: View in Article 2.5 (2023).
[65] Guojian Zhan et al. "Bicriteria Policy Optimization for High-Accuracy Reinforcement Learn- ing". In: IEEE Transactions on Neural Networks and Learning Systems 37.1 (2026), pp. 312- 326.
[66]
Adam Paszke et al. "Pytorch: An imperative style, high-performance deep learning library"" In: Advances in neural information processing systems 32 (2019).
[67] James Bradbury et al. JAX: composable transformations of Python+NumPy programs. Ver- sion 0.3.13. 2018
[68] Ilya Loshehilov, Frank Hutter, et al. "Fixing weight decay regularization in adam". In: arXiv preprint arXiv:1711.05101 5 (2017).
12

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justifiicton right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No] ", it is perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
13

---

Justification:
The limitations of our work are discussed in Section 6.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approxinations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proof'
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer:
[NA]
Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide all necessary details to reproduce the main experimental results in Section 5.
Guidelines:
14

---

The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: Our experimental data and code have been made publicly available and are accessible at
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
Seuiides/oodSuumissee
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: / inips.cc/publin http:///wgeeeeeeeeeeeeceeee.eeee httppp/piiiigguiteeee/eeeeeubbii SooodeSubmsssionn.
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
15

---

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
The paper specifies all training and testing details, as well as hyperparameter analysis, in Appendix D and Appendix E.4.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: The paper presents the RMSE results obtained from experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Sufficient information on the computational resources is provided in Ap- pendix D.
Guidelines:
16

---

The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make into the paper).
9. Code ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics dtctps:/neuurpp: wwweeurpp.ccpppiiii httpppllic/ticscuiieeeeeeee
Answer: [Yes]
Justification: The research presented in this paper fully conforms, in all respects, to the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction..
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
The societal impacts of this work are discussed in Section 6.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications,
authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
17

---

Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging. and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question:
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Yes, the creators and original owners of assets are properly credited, and their licenses and terms of use explicitly mentioned and respected.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. conddttaeess has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is available online, the authors are encouraged reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
18

---

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our L.LM policy (https /ttp:///eeuu.peeeeeee wcc/ooneereeccss/22222222 for what should or should not be described.
19

---

A Bayes Filters
In this section, we introduce the classical Bayes filters, including Kalman filter (KF), Extended Kalman filter (EKF), Ensemble Kalman filter (EnKF), and Particle filter (PF). For detailed descriptions of advanced variants, readers are encouraged explore the relevant literature cited in Section 5.
A.1 Kalman Filter
Suppose the SSM is linear and Gaussian:
Dt+1 F + Ev:
yt = Ha, + Ce:
where &, ~ (E, Qt) and S,~ ~ (N, 10, R2 represent zero-mean Gaussian noise terms with covariance matrices Qe RMX and R₂ IR NNN,, respectively. The initial state is Gaussian distribution, given by To ~ (20l Σo) with known mean 140 E IR M and covariance Eo E RMXM According the Kalman filter [27]. the prior predictive distribution and posterior distribution remain Gaussian:
p(a, (1a-1 = -N@x; M1j-11 Eqj-1)
p( = -N(x μ4 E.).
The mean and covariance of the prior predictive distribution are denoted by:
M1j-11 1114-11
E1p-1 FFE,-1F. Qu:
The posterior mean and covariance are given by:
ΣT = 1/1 H R H
K, = E,H, Ri
Mcje-1 + Kr (ye- H.M4q-1)-
where K; the Kalman gain matrix.
A.2 Extended Kalman Filter
When faced with nonlinear systems, EKF introduces the first-order Taylor approximations of the nonlinear transition model f(.) and observation model h(-). We denote the Jacobians of these functions as F (-) and (+). Then the predict step of EKF is:
M4j111 = 5(1411)
Σ E1p-1 (x D F (1( Q.
The update step is
Σ, Eūj-1 + H Hx(M(p-1) ) Ri H Hal
K. H,H, , (14)1-1) R,,
p. Majo1 + Ki y/ -Ho((up-)))1--)))
A.3 Ensemble Kalman Filter
The EnKF was developed as an alternative to EKF for high-dimensional and chaotic systems, such as weather forecasting [61]. Instead of explicitly computing the covariance matrix, EnKF represents the belief state with ensemble of N₂ particles क RM where i 1.... .N. The predict step of EnKF obtains a - using the transition model from (3a). In the update step, predicted observations y are sampled for each particle as:
5 ((iih(()))))) R.) i= 1,2, ... Ne.
20

---

Next, the particles are updated as follows:
K; Cov(# 9.) V()
: u 1/111 Kiy, D(), i 1,2, ,Ne.
where Cov(,,)) and V-) denote the covariance and variance, respectively, Finally, the state estimate time / is caleulated as the average of all particles:
A.4
Particle Filter
The PF is filtering method based on sequential importance sampling [39], characterized by high precision but relatively low computational efficiency. The standard implementation of PF follows the bootstrap procedure outlined below:
(I) Sampling: Generate samples for all Np particles;
at p(æ,)=)-1). i i= 1,2,
(2) Weighting: compute weights of all particles and normalize;
wf (x
(3) Resampling: Resample particles based on their normalized weights.
Finally, the state estimate at time calculated as the weighted average of all particles:
Np
w w
Table 4: System Specifications for Experiments. The dimensions of the state and observation spaces are denoted by M and N, respectively. Dataset Size refers to the collected data from the system, specified as (trajectory length, number of trajectories, and state dimension). The Forward Euler method refers to first-order explicit discretization scheme, while RK4 represents the fourth-order Runge-Kutta discretization scheme [62].
System
Linear
Dataset
Q
A:
Discretization
Application Domain
Tracking
(200. 100,
0.11
0.1
Forward Euler
Target Tracking
Selkov
F
2
2
(200, 100, 4)
I
0.01
RK4
Glycolysis Process
Oscillator
F
(200, 100,
r
0.01
RK4
Oscillatory Motion
Hopf
F
2
2
(200, 100, 4)
I
0.01
RK4
Chemical Reactions
Pendulum
F
(200, 100,
I
r
0.01
RK4
Physical Dynamics
Lorenz
(200, 100, 144)
1001
0.01
RK4
Atmospheric Dynamies
VL20
F
72
72
(200, 100, 144)
I
0.01
RK4
Atmospheric Dynamics
B System Descriptions
The systems used in this experiment are based on real-world models that are continuous in nature and governed by ordinary differential equations (ODEs). To map these continuous systems into the corresponding Markovian space model (SMM), we discretize them using methods such as the forward Euler method and the fourth- oodrrr-order Runge- Kutta discretization [62]. For the uncertain noise terms (1), denoted as &, and Ç., we assume they follow standard Gaussian distribution, ensuring consistency across the models.
&, ~.~ NN,000 Q), S. ~ N(O
21

---

Figure 8: Illustration of Experiment Systems. (a) Tracking; (b) Double Pendulum; (c) Selkov; (d)
Oscillator; (e) Hopf; (f) Lorenz96 & VL20.
where Q E RMXM and R E I. NxN are the covariances, A summary of the essential characteristics of all the systems is provided Table and Figure 8. Following this, the specific background and mathematical modeling for each system are discussed in detail.
Tracking. The Tracking system, also known the Wiener velocity model, is canonical linear Gaussian system [5] commonly used for target Tracking [63]. The state represents the position and velocity of a moving object in two dimensions, expressed as r = [px Py Uyy where Ps, and Py denote the object's positions in the longitudinal and lateral directions, respectively, and er and Uy are the corresponding velocities. The observations are direct, noisy observations of the position components. The overall system can be described
0
0
0
1
ē
0
2.
0
0
0
0
1
0
y:
0
Ct + S2-
Selkov. The Selkov system models the dynamics of the glycolysis process through set of two coupled differential equations [49]. The state vector [0, 02] represents the concentrations of the metabolites involved the glycolytic cycle, where 0, and 02 correspond of the key metabolites. The constants 0.08 and b 0.6 are parameters related to the reaction rates within the system. The dynamics of the system and the observation process are governed by the following equations:
concentrations
[e]
01 + aB2 + 07002 b all 090₂
y: = m. + E.-
02
Damped Oscillator. The damped Oscillator refers a physical system in which an object experiences Oscillatory motion that gradually decreases over time due to the presence of a damping force [50]. In this system, the state vector i01 0z] represents the positions of the Oscillator along two axes, 0 and 0g. The overall system can be described as:
0
a0 + 693 003
y: x. + S.-
02
60
Here, a = 0.1 and b = 2 constants that define the strength of the nonlinearity, and the cubic terms represent the restoring forces in the system.
Hopf Bifurcation. The Hopf bifurcation describes dynamical system that undergoes spontancous oscillations as system parameter crosses critical threshold, transitioning from stable equilibrium to an Oscillatory regime [32]. This phenomenon is commonly observed in various real- world systems, such as chemical reactions, electrical circuits, and fluid dynamics. In this system, the state vector is represented as x 0, 0₂] The parameter μ = 0.5 controls the stability of the system, while = 1 and are constants that define the frequency of oscillations and the strength of the nonlincar interactions, respectively. The overall system be described as
01 02
100
w02 A0 (0² +
y. = x, + S..
w + 100z A0 03)
22

---

Double Pendulum. The Double Pendulum, often referred to as the chaotic pendulum, is a nonlinear dynamical system consisting of pendulum with another pendulum attached to its end [51]. The state vector a = [01 wi 0g wz] encapsulates the system's configuration, where 82 and 02 represent the angular positions, and wi and wy denote the angular velocities of the two pendulums. The system's observations are the noisy observations of the angles 81 and 82:
Mylaw (D)(D) (D)( MMG3sinnnneee sin( enst
sin (Mg)))ssn
W1
(M Mi++M2l4, Mr cos
0g
M. Mowi sin( cos() () + MM (1 cos D (M sin sin( (in(
W2
12 (M, M.MM++ii Ma) M1 co))
1
0
0 0
y.
0
0
1
Zt + S., D 02 θ1.
Here, M1 = 1 and M2 are the masses of the two pendulums, L1 1 and L2 = represent their respective lengths, and G 9.8 is the gravitational constant that governs the motion of the system.
Lorenz96. The Lorenz96 system, conceived by Edward Lorenz [52], offers simplified yet powerful model of atmospheric dynamics, often serving cornerstone in chaos theory and meteorology. This system described by a set of ordinary differential equations that capture the intricate interplay between atmospheric modes in periodic domain:
i (x+++1 x x((-2))ax(i-1) x11) F, y- Ses 1, ...,, M.
In this model, (d)) represents the jth state of the Lorenz96 system, while F 8 is the external forcing term that drives the system. The system operates under periodic boundary conditions, ensuring continuity across the domain: x (M/+1) = x (0) (MM) and x!-1) (dM-1), with M 72 representing the total number dimensions in the state space.
Vissio-Lucarini 20. The Vissio-Lucarini 20 model (VL20), introduced by Vissio and Lucarini [53]. presents coupled system that provides minimalist yet rich representation Earth's 's atmospheric dynamics. By extending the Lorenz framework to include temperature variables, the VI.20 model enables the development of more complex and nuanced behavioral patterns:
(oli+1) ( - = F.
( (1+t1) g(++)) 10 g + = 7066 + +G,
yt = H, St. 1, M'.
In this model, M' represents the number of discrete points in the system, and the state vector is given by ( with M 2M variates. The constants F 10, G 0, 1, 1 are defined in [53]. Furthermore, the system is subject to periodic boundary conditions, akin to those in the Lorenz96 model.
C Principle Analysis
In this section, we analyze the underlying connection between large language models (LLMs) and filtering, As illustrated in Figure 9, we present simplified comparison of the two processes. LLMs predict the probability distribution of the next token based on all preceding tokens, while filtering estimates the distribution of the current system state given all past observations. This fundamental similarity, where both approaches make predictions based on sequential historical information, enables natural alignment between them. With appropriate alignment, LLM-Filter effectively leverage the reasoning capabilities and rich pre- trainediee knowledge of LLMs for state estimation tasks.
Regarding a more detailed theoretical explanation, it is challenging to provide one at this stage. This limitation stems from the fact that the theoretical foundations of LLMs themselves remain an open research problem [29, 64]. Consequently, many applications of LLMs across diverse domains-suchh as time series forecasting [30, 44, 46] and control systems [18, 19, 20, 65), also lack rigorous mathematical justification. Similarly, our work focuses on empirical evidence and the practical alignment between language modeling and filtering, while acknowledging the need for deeper theoretical understanding future research.
23

---

The
cat
sat
on
the
mat
LLM
Y1
Y2
Yt-1
p(xeXrn))
A Alggmment
eD
&
Filter
Figure 9: Principle Comparison of LLMs and Filtering. The LLM predicts the next token metg based on preceding tokens, while a filter estimates the current state 2 using past observations.
D Experiment Details
Our model is built with PyTTrrch [66] and trained extensively the same server with the following hardware configuration: 4 x Intel@ Xeon@ Gold 6226R @ 2.90GHz CPUs and x NVIDIA H8800-00 GPU. The complete code is publicly accessible at
2367/LLM--ilee
D.1 Benchmark Settings
The online Bayes baseline methods are implemented by JAX [67] following [48]. The number of particles for EnKF, EnKFI, EnKFS, and HubEnKF is set to 1000 to balance efficiency and accuracy [47]. For fair comparison, the hyperparameters of all online Bayes methods are selected on the first trial using the Bayesian Optimization (BO) package [55] with parameter range of (1, 20). The KalmanNet [13] implementation is based their official open-source source repository. The MEstimator [14]. RStateNet [15]. and ProTTran [43] are implemented according to the methodologies outlined their respective papers. The parameter counts and peak memory for different models in Table 5. Since we froze the core layers of Llama-7B and only trained the ObsEmbedding and StateProjection layers, we distinguish between the total parameter count and the trainable parameter count.
Our experiments are conducted on the systems described in Appendix B. We gather 100 trajectories, each consisting of 200 time steps, from each system, as shown in Table 4. To prevent data leakage, the datasets are split chronologically into training, validation, and test sets in 7 ratio.
Table 5: Comparison of Model Parameters and Peak Memory
LLM-Filter (Total/Trainable)
Model
MEstimator
RStateNet
ProTran
KalmanNet
Parameter
6.61B /4.22M
2.12M
7.38M
15.77M
5.20K
Peak memory
27.43GiB
622.51MiB
742.57MiB
854.54MiB
582.93MiB
D.2 Implementation
In ILLM-Filter, only the ObsFanbeddingand StateProjection layers are updated via the gradient. These components implemented using a simple MI.,, which has been shown to effectively embed and project time series for LLM [30]. The default hidden dimension and number of MLP layers, batch size, window length 7', and segment length L are to 512, 16 ,40, and 20 respectively. We use the Adam W optimizer [68] with the initial learning rate of 10 and the weight decay of 10 10-5, where the model trained for 10 epochs. Unless otherwise specified, we use LLaMA-7B [56] as the default base LLM.
D.3 Evaluation Metrics
For evaluation, we use the root mean square error (RMSE) as the primary metric assess the accuracy of state estimation [5]. To evaluate efficiency, measure the inference running time per step in

---

milliseconds (ms/step). The calculations for these metrics are as follows:
Teros
Cost Time
RMSE
Runtime
Tiraj
Ttraj: Niest
where Tiroj is the trajectory length and Neett denotes the number of trajectories the test set.
Supplementary Results
E.1
Running Time
We evaluated the estimation efficiency (ms/step) of various methods on the same device, as described in Appendix D. The results are presented in Table 6.
Table 6: Comparison of Estimation Runtime (ms/step) Across Methods.
Method
LLM-Filter
EnKF
RStateNet
Selkov
0.8766
2.5095
0.1546
Oscillator
0.8299
2.6815
0.1553
Hopf
0.8245
2.5747
0.1267
Pendulum
1.2447
2.2246
0.0967
Lorenz96
0.9577
6.7377
0.4513
VL20
0.9503
2.9397
0.4521
E.2 Generalization Estimation Experiments
To assess the generalization capabilities of LLM-Filter we conducted extensive evaluation. First, we evaluated its performance on systems subjected to varying strengths of outliers, as detailed in Table 7. Notably, while the performance of other methods deteriorated rapidly with increasing number of outliers, LLM-Fille demonstrated minimal performance loss.
We also tested the classical inteevaatiooniittraamnn scenario [48], where robust filtering methods such as EnKFI, EnKFS, and HubEnKF are known to excel. As shown in Table 8, although LLM- Filter did not achieve the best results, it delivered performance comparable specialized robust algorithms. These findings highlight LLM-Filter's ability to generalize effectively even in the presence of significant outlier interference system turbulence, offering reliable solution across diverse and challenging scenarios.
In addition, provide the complete experimental results for the cross-system scenarios discussed in Section 5.2, as shown in Table
E.3 Scaling Behavior
We investigate the scaling behavior of LLM-Filter in state estimation tasks by evaluating models with various pretrained LLM backbones. Detailed results can be found in Table 10. Our analysis reveals that LLM-Filter follows scaling-law behavior: as the parameter size of the model increases, average estimation error tends to decrease, leading to improved estimation performance.
E.4 Hyperparameter Sensitivity
In the main experiments, the configuration of the LLM-Filter is provided in Appendix D.2. Here, we verify the robustness of the LLM-Filter with respect hyperparameters, including the sliding length T. the hidden layer dimension, and the layer number of MLP in ObsEmbedding and StateProjection. Experiments were conducted on the Selkov and VL20 systems, with the results summarized in Figure 10. Overall, LLM-Filter demonstrates consistent performance across different parameter settings, showing it is largely insensitive to hyperparameter variations. On average, a window length of 20 or 40 yields better results. This suggests that appropriately chosen window lengths strike windows risk losing crucial information, while longer windows may fail to capture local features effectively.
25

---

Table 7: RMSE Results for Observation Outlier Scenarios. Here R. denotes the true observation covariance with outliers, while Rg represents the assumed covariance.
Method
LLM-Filter
EnKF
EnKFI
EnKFS
HubEnKF
Ro
0.4061
0.5978
0.6703
0.6081
0.6676
25Ro
0.6898
1.8626
1.9874
1.6523
2.1998
Selkov
50
0.7034
2.5496
3.4248
2.6597
2.5318
75Rl
0.7074
3.0952
4.3829
3.1699
2.8636
100Ro
0.7090
3.5625
4.8417
4.3972
2.9850
Ro
0.5247
0.5505
0,5516
0.5548
0,5540
R5
0,5946
1.1630
1.2714
1.1674
1.1746
Oscillator
50Rg
0.6044
1.3267
1.3195
1,3575
1.4778
75Rg
0.6083
1.4701
3.1428
1.4757
1.8707
100Ro
0.6110
1.6154
1.6040
1.5793
2.1680
Ro
0,5751
0.6322
0,6983
0,6380
0,6982
25Ro
0.7142
0.9946
1.1299
0.9938
1.0976
Hopf
50Ag
0.7738
1.0549
1.1657
1.2035
L.1155
R5
0.8010
1.0898
1.1872
1,5071
1.1313
100Ro
0.8096
1.1146
1.1989
2.1321
1.1483
Table 8: RMSE Results of Observation- Dittutrron---tttuaaeeeeeee Scenarios.
Method
LLM-Filter
EnKF
EnKFI
EnKFS
HubEnKF
Selkov
0.4999
0.9223
0.9500
0.6079
0.6198
Oscillator
0.7451
1.3431
1,3611
0.5543
0.5364
Hopf
0.6827
0.8220
NaN
0.6379
0.7226
Lorenz96
0.9186
7.6225
7.7459
4.5833
3.1399
VL.00
0.8876
NaN
7.7777
11.5041
0,9684
Table 9: RMSE Cross- Pross-System Scenarios.
Method
LLM-Filter
LLM-Filter-O
MEstimator
RStateNet
ProTran
Tracking PPeddlum
1.2568
1.6533
3.6957
3.1052
3.1162
Pendulum--Trackinn
1,4214
1.8936
2.7407
5.0927
NaN
Dscillator-HHopp
0.6613
2.1125
NaN
3.8645
3.5668
Hopf-=Oscillator
0.9069
1.5624
2.5369
2.7542
3.0075
Table 10: RMSE Results of Various LLMs in the LLM-Filter Framework.
LLM
GPT-2 (124M)
OPT-1.3B
LLaMA-7B
LLaMA-13B
Selkov
1.0434
1.1713
0.6369
0.6200
Oscillator
0.9393
1.0702
0.5753
0.4441
Hopf
0.9611
0.8466
0.8180
0.7000
Lorenz96
0.9727
0.9029
0.9735
0,9147
VI.20
0.9931
0.5999
0.8433
0.7623
Average
0.9819
0.9182
0.7694
0,6882
Additionally, the segment length in LLM-Filter is hyperparameter that determines how many observations are grouped into a single token before being fed into the LLM. We conducted further experiments using different segment lengths while keeping the window length fixed at 40, as shown in Table 11. The performance of L.M--Filter remains relatively stable across different segment lengths, demonstrating its robustness to this hyperparameter.
26

---

T-10 10
7-
T-40
+
7-50
0.8
Selkov
0.8
Selkov
1.1
VL20
1.1
VL20
1.0
0.9
0.6
0.6
0.9
0.8
0.7
0.7
0.4
0.4
0.5
0.6
0.5
0.3
128
256
512
1024
128
256
512
1024
MLP Hidden Dimension
MLP Hidden Layer
MLP Hidden Dimension
MLP Hidden Layer
Figure 10: Hyperparameter Sensitivity of LLM-Filter on the Selkov and VL20 Systems.
Table 11: RMSE Comparison with Varying Segment Lengths
Segment Length
Base (20)
10
5
Selkov
0,4061
0.7024
0.3077
Oscillator
0.5247
0,5829
0.5298
Hopf
0.5751
0.6222
0,5779
Pendulum
0.8348
0,5028
0.8254
Lorenz96
0.9149
0.9200
0,9182
VL20
0.7717
1.0346
1.0221
Average
0.6712
0.7275
0.6969
27


Reviews:
[{"review_id": "wd1zb6rrXA", "reviewer": "Reviewer_aUbp", "summary": "The paper is addressing state estimation of dynamical systems (hidden states in markov chains). Previously in this field LLMs and VLMs have shown positive results. This paper is introducing a general filter to integrate LLMs into state estimation. The system is able to take in disparate data modalities, then uses inputs augmented with System-as-Prompt to run inference with an LLM - said inference is then mapped into a state estimate.", "questions": "1.Did you do any experiments playing with the levels of SaP? Such as omitting taks examples, varying the amount of examples provided. I know you included LLM-filter-O, but I would like to see a bit more experimentation with how SaP added to the efficacy of the filter.\n2. LLM-Filter optimizes MSE but does not output filter covariance: Can the authors: quantify residual error distributions for the Hopf system, and identify conditions where the filter diverges or large errors occur? \n3.Could you clarify a bit more about where the latency/memory overhead is coming from in the LLM filter?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Quality: Good idea seemingly done well. Good benchmarking, good results. Initially I was very skeptical of the SaP idea however [remark 1] mentions they tested without it and sure enough table 1 shows LLM-filter-O performing substantially worse in multiple tests. \n\nClarity: Very clear, easy to read paper. Some hyperparameter details were only mentioned in the appendix (like window seize, prompt length, etc) which could confuse some people\n\nSignificance: seems to be reasonably significant, to my knowledge they’re the first to repurpose LLms for generalist filtering. They outperform specialist filters. Some limitations like all experiments pressuring observation and state dimensionality or the latency of the filter remain for future work. \n\nOriginality: Novel system integration, SaP seems novel as well (albiet mostly just clever prompt engineering).", "was_revised": true, "final_justification": "My review stands for the reasons outlined in my comment. I think this is a solid paper with a solid goal - there is just nothing massive or novel enough for me to raise my review", "rebuttal": "Thank you very much for your insightful question. We appreciate the opportunity to clarify and hope the following explanation addresses your concerns.\n\n---\n\n**[Question 1]: Did you do any experiments playing with the levels of SaP? Such as omitting taks examples, varying the amount of examples provided. I know you included LLM-filter-O, but I would like to see a bit more experimentation with how SaP added to the efficacy of the filter.**\n\nThank you for the insightful suggestion. To better understand the role of SaP, we conduct an ablation study by varying the number of **task examples** in the prompt, while keeping the **task instruction** fixed. The results are summarized in the table below.\n\n| Example Num | LLM-Filter-O | 0      | 2      | 5  | 10     |\n| ----------- | ------------ | ------ | ------ | -------------- | ------ |\n| Selkov      | 0.6369       | 0.6321 | 0.5882 | 0.4061         | 0.4164 |\n| Oscillator  | 0.5753       | 0.5692 | 0.5241 | 0.5247         | 0.5099 |\n| Hopf        | 0.8180       | 0.8128 | 0.7382 | 0.5751         | 0.5514 |\n| Pendulum    | 0.9218       | 0.9145 | 0.8597 | 0.8348         | 0.8267 |\n| Lorenz96    | 0.9735       | 0.9806 | 0.9226 | 0.9149         | 0.9223 |\n| VL20        | 0.8433       | 0.8469 | 0.7970 | 0.7717         | 0.7708 |\n\nWe observe that adding a few task examples significantly improves performance, confirming their value as in-context cues. However, the marginal gain diminishes as the number increases. This is likely due to attention dilution, where too many examples reduce focus on the actual input segment.\nMoreover, the result for **0 example** (task instruction only) outperforms **LLM-Filter-O** (no instruction nor examples), highlighting that **task instruction alone provides a meaningful signal** for the model.\n\nThese findings suggest that both task instruction and examples are beneficial, but the number of examples should be balanced to avoid overwhelming the attention capacity of the model.\n\n---\n\n**[Question 2]: LLM-Filter optimizes MSE but does not output filter covariance: Can the authors: quantify residual error distributions for the Hopf system, and identify conditions where the filter diverges or large errors occur?**\n\nThank you for your question. As far as we know, existing learning-based filters [1-4] generally do not provide an explicit quantification of residual error distributions either. They are usually trained to directly predict outcomes (e.g., filtering decisions), but they don't come with a built-in probabilistic model that would allow for estimating uncertainty in the residual error.\n\nOne possible direction to address this limitation is to adapt uncertainty estimation techniques developed for LLMs. For example, some recent work has proposed using **Laplace approximation** [5] to estimate the variance of the model's output, which provides a principled way to quantify uncertainty. Incorporating such techniques into LLM-Filter could enable future extensions that include uncertainty estimates alongside state predictions. \n\n---\n\n**[Question 3]: Could you clarify a bit more about where the latency/memory overhead is coming from in the LLM filter?**\n\nThank you for the question. Below, we provide a detailed breakdown of the training time and GPU memory usage for each component of the LLM-Filter in the Selkov system. The '**Ops**' primarily consist of reshape, normalization operations. The **pre-allocated** memory includes storage for input data, model weights, and key-value (KV) caches, which remain mostly static during inference and training.\n\n| Component | Encoder | LLM Core | Decoder | Ops  | Total  |\n| --------- | ------- | -------- | ------- | ---- | ------ |\n| Time (ms) | 1.00    | 79.28    | 0.58    | 2.66 | 73.08 |\n\n| Component    | Encoder | LLM Core | Decoder | Pre-allocated | Total    |\n| ----------- | ------- | -------- | ------- | ------------- | -------- |\n| Memory (MB) | 4.19    | 12416.85 | 4.19    | 12647.80      | 25073.03 |\n\nThe latency and memory overhead primarily stem from the LLM core, while the remaining components and operations add only minor costs. This is consistent with the expected resource demands of LLMs during filtering tasks.\n\nWe hope this clarifies where the latency and memory overheads arise.\n\n---\n\n### **Reference**\n\n[1] Revach, G., et al. KalmanNet: Neural network aided Kalman filtering for partially known dynamics. IEEE Transactions on Signal Processing, 70, 1532-1547.\n\n[2] Dahal, P., Mentasti, S., Paparusso, L., Arrigoni, S., & Braghin, F. (2024). RobustStateNet: Robust ego vehicle state estimation for Autonomous Driving. Robotics and Autonomous Systems, 172, 104585.\n\n[3] Ji, G., Mun, J., Kim, H., & Hwangbo, J. (2022). Concurrent training of a control policy and a state estimator for dynamic and robust legged locomotion. IEEE Robotics and Automation Letters, 7(2), 4630-4637.\n\n[4] Buchanan, R., et al. Learning inertial odometry for dynamic legged robot state estimation. In Conference on robot learning (pp. 1575-1584). PMLR.\n\n[5] Yang, A. X., Robeyns, M., Wang, X., & Aitchison, L. Bayesian Low-rank Adaptation for Large Language Models. In The Twelfth International Conference on Learning Representations."}, {"review_id": "Sd42Rpi915", "reviewer": "Reviewer_HiuW", "summary": "**Summary**\n\nThis paper introduces a \"universal filter\" framework for dynamic system state estimation called **LLM-Filter**, which repurposes large language models (LLMs) as filters through three key steps: First, it segments continuous, noisy observation sequences and embeds them into the same representation space as LLM tokens using an ObsEmbedding module. Second, during inference, it organizes system/task descriptions and a few examples into a prompt, which is concatenated with the observation embedding as context tokens and input to the frozen LLM core layer, a process termed System-as-Prompt (SaP). Third, it maps the token representation predicted by the LLM back to the continuous state space using a lightweight MLP, training only the ObsEmbedding and StateProjection modules while keeping the LLM weights frozen. Across multiple benchmarks, including low-dimensional nonlinear systems and high-dimensional chaotic atmospheric models, LLM-Filter achieves significantly lower RMSE than state-of-the-art learning-based filters like KalmanNet and ProTran, as well as various Bayesian filters such as EnKF and PF, all while maintaining millisecond-level inference efficiency. Extended experiments highlight its robustness in model mismatch and cross-system generalization scenarios, with a clear scaling law showing error reduction as LLM size increases. The main contributions include proposing the universal LLM-Filter framework that enables direct state estimation with frozen pre-trained LLMs, designing the SaP mechanism for quick adaptation to new systems without retraining, verifying performance and generalization through substantial error reductions on multiple systems, and open-sourcing the code and datasets to provide a reproducible benchmark and promote LLM applications in engineering control.", "questions": "1. **Question about baseline parity and training budget**: KalmanNet is reported to diverge on two high-dimensional tasks, and all learning baselines are trained from scratch for every system, while LLM-Filter freezes 7 B parameters and tunes only ≈ 4 M trainable weights (Table 5). Could you please detail the exact wall-clock training time, compute, and early-stopping criteria for each baseline and clarify whether any hyper-parameter sweeps used validation data inaccessible to the baselines after split?\n\n2. **Question about observation-to-token embedding fidelity**: The ObsEmbedding groups L numeric readings into one token, and performance is claimed to be “stable” across window/segment lengths (Table 11), but Figure 10 shows marked variance on VL20 when hidden dims shrink. Could you please provide visualisations of reconstruction error when mapping observations → token → projection and discuss whether discretisation causes information loss for high-frequency dynamics?\n\n3. **Question about scaling-law interpretation vs. parameter count confound**: Figure 7 shows lower RMSE with larger backbones, yet the ablation replaces the LLM with a 4-layer Transformer of ~10 M params (Table 3). Could you please run a control where you keep ~7 B parameters but randomly initialise the core layers to disentangle pre-training knowledge from model capacity?\n\n4. **Question about runtime and memory claims**: LLM-Filter peaks at 27 GB memory yet runs at ~1 ms/step (Table 6, hardware: H800-80G), while RStateNet is two orders of magnitude smaller but reported slower on high-dim tasks (Table 6). Could you please specify whether baselines were executed on identical GPU kernels, precision (fp16/bf16), and batch sizes and include a throughput vs. latency plot for varying batch sizes?", "limitations": "Please see the Weekness.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n1. **Rigorous Methodology**: The paper instantiates a clear three-stage pipeline—ObsEmbedding, System-as-Prompt, and StateProjection—with frozen LLM weights and only two lightweight trainable modules. This modular design simplifies training and ablation studies, demonstrating careful experimental control. \n\n2. **Comprehensive Evaluation**: Benchmarks span low-dimensional nonlinear systems, high-dimensional chaotic atmospheric simulations, model-mismatch scenarios, and cross-system generalization. Metrics (e.g., RMSE) are reported against both classical Bayesian filters and state-of-the-art learned filters, showing statistically significant improvements. \n\n3. **System-as-Prompt Paradigm**: While in-context learning is well known in NLP, applying it as a “prompt-based filter” with formal embedding/projection modules is novel. It reconceives LLMs as black-box dynamical system solvers rather than pure sequence predictors.\n\n4. **Scaling Law Analysis**: The finding that filtering accuracy improves monotonically with LLM scale (and limited training) provides fresh insight into how model scale interacts with downstream control tasks.\n\n**Weekness**\n\n1. **Theoretical Justification**: The paper lacks a deeper theoretical analysis of why LLM latent spaces align so well with continuous state spaces. A formal connection or bounds on estimation error would strengthen the work’s foundations.\n\n2. **Limited Noise Models**: Experiments focus on Gaussian observation noise; robustness to heavy-tailed or adversarial noise is untested, leaving open questions about performance in more challenging real-world settings.", "was_revised": true, "final_justification": "The authors’ response has addressed most of my concerns, so I will retain my original score.", "rebuttal": "Thank you very much for your insightful question. We appreciate the opportunity to clarify and hope the following explanation addresses your concerns.\n\n---\n\n**[Question 1]: Baseline parity and training budget.**\n\nThank you for your question. We provide a detailed breakdown of the training protocols and implementation details of each baseline below to clarify training budget, early stopping, and tuning fairness.\n\n| Method     | Implementation                                                            | Early Stopping    | Training Time | HP Tuning              |\n| ---------- | ------------------------------------------------------------------------- | ----------------- | ------------- | ---------------------- |\n| KalmanNet  | Official repo [TSP KalmanNet](https://github.com/KalmanNet/KalmanNet_TSP) | ✘ (not supported) | \\~15 min      | Grid search on val set |\n| RStateNet  | Re-implemented                                                            | ✅ (patience=3)    | \\~13.69 mins      | Grid search on val set |\n| MEstimator | Re-implemented                                                            | ✅ (patience=3)    | \\~12.16 mins   | Grid search on val set |\n| ProTran    | Re-implemented                                                            | ✅ (patience=3)    | \\~11.03 mins   | Grid search on val set |\n| LLM-Filter | Ours                                                                      | ✅ (patience=3)    | \\~30.16 mins   | Grid search on val set |\n\n* For **KalmanNet**, we directly used the [official implementation](https://github.com/KalmanNet/KalmanNet_TSP). This version does not include built-in early stopping, and saves the best-performing checkpoint on the validation set and reports its test results.\n* For **RStateNet**, **MEstimator**, and **ProTran**, we re-implemented the architectures and applied the same early stopping strategy used for LLM-Filter (patience=3, based on validation loss).\n* For **all methods**, we conducted modest grid searches on core hyperparameters (e.g., learning rate, hidden size), using the same training and validation data. **Test data was strictly held out until final evaluation**.\n\nRegarding the divergence of KalmanNet on high-dimensional systems, we emphasize that this issue is not due to insufficient training time or suboptimal hyperparameter tuning. Instead, we believe it arises from **KalmanNet’s architectural reliance on Kalman structure** and a hidden state representation whose **dimensionality scales quadratically with system size**. \nThese structural choices, while suitable for low-dimensional linear systems, significantly limit scalability and representational capacity for high-dimensional nonlinear systems.\n\nWe hope this addresses your concern and assures a fair comparison was made across all baselines.\n\n---\n\n**[Question 2]: Observation-to-token embedding fidelity.**\n\nThank you for your question. While visual inspection might suggest instability, the actual numerical results show that **performance variance decreases as the hidden dimension shrinks**. The complete RMSE values across different segment lengths and hidden dimensions are provided in the table below:\n\n | Hidden Dim | 128    | 256    | 512    | 1024   |\n | ---------- | ------ | ------ | ------ | ------ |\n | \\$T = 10\\$ | 0.6825 | 1.0353 | 0.9307 | 1.0100 |\n | \\$T = 20\\$ | 0.6200 | 0.5610 | 0.4632 | 1.0823 |\n | \\$T = 40\\$ | 0.9834 | 0.6873 | 0.7509 | 0.4525 |\n | \\$T = 50\\$ | 0.7705 | 0.7226 | 0.9686 | 1.0144 |\n | Variance   | 0.0252 | 0.0406 | 0.0532 | 0.0861 |\n\nOverall, the performance variance remains within a small range (generally under 0.1), indicating that our method is **stable** across window lengths. Additionally, increasing the hidden dimension generally leads to improved performance, but also results in slightly higher variance across different window lengths. This is because larger hidden dimensions give the MLP more capacity to capture complex patterns from LLM embeddings, leading to better accuracy. However, the increased capacity can also make the model more sensitive to training data and hyperparameters, resulting in slightly higher variance.\n\nWe apologize if we misunderstood your request. We assumed that \"**reconstruction error visualization**\" refers to visualizing the residuals between the estimated states and ground-truth values.\nWe have plotted the estimation vs. ground truth trajectories for the Selkov system to analyze the residuals. Due to rebuttal constraints, we are unable to include figures here, but we will add them in the final version of the paper. We apologize for this limitation.\n\nWe would also like to clarify that our work focuses on the **discrete filtering problem** [2], where both the system and observations are discretized. \nFor each dynamical system, we apply discretization beforehand, as described in Appendix B, which introduces unavoidable information loss equally to all methods. To ensure fairness, all baseline methods we compare against are also discrete filtering approaches and are trained and evaluated on data collected from the same discretized system.\n\n---\n\n**[Question 3]: Scaling-law interpretation vs. parameter count confound.**\n\nThis is a great question, which also helps explain why large models are effective. We initially considered conducting an ablation study by randomly initializing the 7B-parameter LLM backbone to disentangle the effects of model capacity from pretraining knowledge. However, in practice, we found that training such a large model from scratch using our task-specific dataset leads to **divergence**, after trying all possible skills, such as warmup schedules, gradient clipping, and regularization techniques. \nThis instability may partly be due to our current hardware limitations, which make it infeasible to fully train a 7B-parameter model from scratch. More fundamentally, it aligns with prior observations that training large models such as LLaMA-7B typically requires extensive data, multi-stage curricula, and significant computational resources to achieve stable convergence [1].\n\nA promising direction for future work is to train a foundation filtering model from scratch by first collecting a comprehensive and unified dataset encompassing multiple dynamical systems. Following this, a multi-stage training procedure could be employed, beginning with large-scale pretraining on the diverse dataset to learn general filtering capabilities, and then fine-tuning on task-specific datasets to adapt to particular system dynamics.\n\n\n---\n\n**[Question 4]: Runtime and memory claims.**\n\nThank you for your question. All methods were evaluated on the same hardware: a single NVIDIA H800-80G GPU. For LLM-Filter, we used bf16 precision, while for the smaller baseline models, we followed PyTorch's default fp32 precision.\n\nWe would also like to clarify that the runtimes reported in Table 6 reflect per-step inference latency with batch size = 1, which is typical for real-time filtering tasks. As such, batch size does not influence the latency values presented in the table.\n\nIt is important to note that inference speed is not solely determined by model size. Differences in network structure are also crucial. For example, LLMs often benefit from optimized inference kernels, fused attention mechanisms, and high degrees of parallelism, which can result in faster execution even with larger parameter counts. In contrast, small custom models such as RStateNet may not take advantage of such low-level optimizations, potentially resulting in slower inference despite their compact size. \n\n---\n\n**[Weakness 1]: Theoretical Justification.**\n\nThank you for your insightful comment.  We acknowledge that establishing formal connections or deriving estimation error bounds would certainly strengthen the theoretical foundation of this work, and we view this as a promising direction for future research.\n\nWe also wish to clarify that the primary contribution of this work lies not in proposing novel theoretical analysis, but in demonstrating a new paradigm for filtering: leveraging the in-context learning capabilities of LLM to build a generalizable filter. This approach enables adaptation to a wide range of dynamical systems without requiring system-specific retraining, thereby addressing a major limitation of conventional learning-based filters that rely heavily on system-specific data and training procedures.\n\n---\n\n**[Weakness 2]: Limited Noise Models.**\n\nThank you for your comment.  We would like to clarify that a classical observation-disturbance scenario, featuring heavy-tailed noise, has been included in Appendix E.2.\n\nAs shown in Table 8, although LLM-Filter did not achieve the best performance in this setting, it performed comparably to specialized robust filtering algorithms. This result highlights LLM-Filter’s ability to generalize well even in the presence of significant outliers or system disturbances, making it a promising solution for diverse and challenging scenarios. \n\n---\n\n### **Reference**\n\n[1] Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M. A., Lacroix, T., ... & Lample, G. (2023). Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971.\n\n[2] Särkkä, S., et al. Bayesian filtering and smoothing (Vol. 17). Cambridge university press."}, {"review_id": "fN7kUoYwSj", "reviewer": "Reviewer_qPxC", "summary": "The work proposes adapting an LLM for state estimation. By designing an embedding scheme and a text-based prompt, the proposed LLM-Fiter outperforms other learning-based baselines. Notably, the LLM-Filter also performs robustly in model mismatch and unseen settings. Furthermore, performance scales with LLM base model size. In an ablation study, when the LLM is replaced with non-pretrained alternate neural models, the performance drops significantly.", "questions": "- For very long sequences it may not be feasible to feed the whole embedding sequence $\\mathbf{e}_{1:N_s}$ into the Transformer. Did the authors consider what are good ways to compress the context in this case? Also does the quality drop significantly?", "limitations": "yes", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n- The proposed work significantly outperforms previous SoTA for dynamical systems' state estimation.\n- Evaluation on different architectures / different base LMs provide strong evidence that the proposed method is effective when true state observations are available. \n\nWeaknesses\n(disclaimer: I am not an expert in the particular subfield of dynamical systems' state estimation, and am open to further discussion regarding comparison against previous work.)\n- The paper assumes that we have access to the true states $\\mathbf{X}$ in Eq 4. But I wonder if this is a very strong assumption.\n- Moreover, there have been prior papers on neural approximation of hidden state models (e.g. [Krishnan et al (2016)](https://arxiv.org/abs/1609.09869) and [Lin and Eisner (2016)](https://arxiv.org/abs/1804.10747)), which may be discussed in the paper. In particular, Lin and Eisner considered a smoothing scenario where the entire sequence of $\\mathbf{Y}$ is given before hand, which can likely further improve accuracy in this paper's setting as well.", "was_revised": true, "final_justification": "My concerns remain that the strong assumptions limit the impact of this paper.", "rebuttal": "Thank you very much for your insightful question. We appreciate the opportunity to clarify and hope the following explanation addresses your concerns.\n\n---\n\n**[Question 1]: For very long sequences it may not be feasible to feed the whole embedding sequence into the Transformer. Did the authors consider what are good ways to compress the context in this case? Also does the quality drop significantly?**\n\nThank you for your question. Indeed, we have adopted a sliding window mechanism, which corresponds to the commonly used Moving Horizon Estimation (MHE) framework [1] with theoretical guarantees, as described in Section 3.1 of the paper. This approach ensures that only a fixed number of recent observations are included in the input to the LLM, keeping the embedding sequence length bounded and computationally tractable.\n\nAdditionally, as discussed in Appendix E.4, we observe that increasing the window length beyond 20 provides diminishing returns in performance. With a window length of 20, the resulting embedding sequence typically contains fewer than 200 tokens, which is well within the effective context length of modern large language models [2, 3]. Based on this observation, we use a window length of 20 in all experiments.\n\n---\n\n\n**[Weakness 1]: The paper assumes that we have access to the true states in Eq 4. But I wonder if this is a very strong assumption.**\n\nIn practice, this assumption is not as strong as it may initially seem.\nSupervised learning-based filters typically require access to ground-truth states during training [4, 5, 6], which are often obtained using high-precision sensors or external tracking systems. This setup is standard in many real-world applications.\n\nFor example, in quadruped robot state estimation [7], ground-truth data is commonly collected using an indoor motion capture system to supervise the training process. Once trained, the filter can operate using only onboard measurements such as IMUs and leg odometry, while still producing accurate state estimates.\n\n\n---\n\n**[Weakness 2]: Moreover, there have been prior papers on neural approximation of hidden state models and Lin and Eisner, which may be discussed in the paper. In particular, Lin and Eisner considered a smoothing scenario where the entire sequence of is given before hand, which can likely further improve accuracy in this paper's setting as well.**\n\nThat's a good question. We would like to clarify that smoothing and filtering correspond to different problem formulations [8]. Smoothing estimates the current state based on both past and future observations, whereas filtering relies solely on observations available up to the current time to infer the current state. While smoothing can improve accuracy by leveraging future information, it is generally infeasible in real-time applications, Such as robotics control. For example, a leg-robot must make decisions (e.g., control actions, trajectory planning, collision avoidance) based on the current belief of its state, which must be inferred from past and immediately available sensor data. Future sensor readings are inherently unavailable during execution, making smoothing inapplicable in such real-time systems. Therefore, this is a typical filtering problem, and our paper is designed with this class of applications in mind\n\nWe also appreciate your pointing us to the work by Lin and Eisner. We acknowledge that our paper currently lacks a discussion of smoothing-based methods, and we will include a more complete review of related smoothing literature in the appendix to better situate our work. Thank you again for your thoughtful feedback.\n\n---\n\n### **Reference**\n\n[1] Alessandri, A., Baglietto, M., & Battistelli, G. (2008). Moving-horizon state estimation for nonlinear discrete-time systems: New stability results and approximation schemes. Automatica, 44(7), 1753-1765.\n\n[2] Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., ... & McGrew, B. (2023). Gpt-4 technical report. arXiv preprint arXiv:2303.08774.\n\n[3] Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M. A., Lacroix, T., ... & Lample, G. (2023). Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971.\n\n[4] Revach, G., et al. KalmanNet: Neural network aided Kalman filtering for partially known dynamics. IEEE Transactions on Signal Processing, 70, 1532-1547.\n\n[5] Dahal, P., Mentasti, S., Paparusso, L., Arrigoni, S., & Braghin, F. (2024). RobustStateNet: Robust ego vehicle state estimation for Autonomous Driving. Robotics and Autonomous Systems, 172, 104585.\n\n[6] Ji, G., Mun, J., Kim, H., & Hwangbo, J. (2022). Concurrent training of a control policy and a state estimator for dynamic and robust legged locomotion. IEEE Robotics and Automation Letters, 7(2), 4630-4637.\n\n[7] Buchanan, R., et al. Learning inertial odometry for dynamic legged robot state estimation. In Conference on robot learning (pp. 1575-1584). PMLR.\n\n[8] Särkkä, S., et al. Bayesian filtering and smoothing (Vol. 17). Cambridge university press."}, {"review_id": "1xRZTMZiZK", "reviewer": "Reviewer_EuEp", "summary": "This paper proposes a framework based on large language models to estimate hidden states of a dynamical systems. The approach consists of an encoder and a decoder which takes care of the modality alignment and a prompt to improve the LLM performance. The authors argue that this approach beats conventional methods in performance but also in generalization to cross systems and overall robustness without a significant increase in runtime.", "questions": "1. Your method significantly outperforms other learning-based filters. Have you evaluated it in a synthetic setting where the Kalman filter is known to be optimal? If so, what is the performance gap between your method and this optimal benchmark? This would be very useful to understand the limits of your approach.\n\n2. The paper would benefit from a discussion on sample efficiency. Could you clarify the data regime of your experiments (e.g., low-sample vs. high-sample) and analyze how the relative performance of your method against baselines changes with the number of available samples? Such an analysis could be particularly insightful, as it might also explain why fine-tuning with LoRA leads to performance degradation.\n\n3. I am surprised that frozen LLMs lead to this level of performance. Could you explain in more detail why you think LLMs can produce a representation that is only a simple MLP away from predicting the system's hidden state? In particular, would it be possible to run experiments to gain some understanding of what kind of representations the LLM produces?", "limitations": "Yes.", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 4, "significance": 2, "originality": 1, "strengths_and_weaknesses": "Strengths:\n\nThe approach is simple, and the experiments indicate strong performance. The adaptability of the approach is a strength. Overall, the use of large language models in dynamical systems is important and could lead to significant improvements in performance.\n\nWeaknesses:\n\nIt is generally unclear what the technical originality of this paper is. The proposed ObsEmbedding and StateProjection are standard for time series, as the authors mention. The prompting is also straightforward. Apart from the experimental evidence provided, I do not see any additional contributions. There is no in-depth analysis, explanation, or intuition regarding why the proposed approach works better than conventional learning-based filters or online Bayes filters.", "was_revised": true, "final_justification": "My initial judgment remains but I am increasing the score to 4 as the results are extremely impressive compared to baselines.", "rebuttal": "Thank you very much for your comments once again. I hope this clarifies your concerns.\n\n---\n\n**[Question 1]: Have you evaluated it in a synthetic setting where the Kalman filter is known to be optimal? If so, what is the performance gap between your method and this optimal benchmark? This would be very useful to understand the limits of your approach.**\n\nThank you for the insightful suggestion. We have added an experiment in a synthetic **linear Gaussian** system, the Tracking system, where the classical Kalman filter is known to be optimal. This system is part of the cross-system evaluation in Section 5.2, with detailed system dynamics provided in Appendix B. All training and data configurations follow those in the main paper. The results are summarized below:\n\n| Method| KF     | KalmanNet | LLM-Filter | LLM-Filter-O |\n| ----------------- | ------ | --------- | ---------- | ------------ |\n| Tracking          | 0.3231 | 0.3262    | 0.3247     | 0.3288       |\n\nAs expected, the Kalman filter achieves the best RMSE. LLM-Filter performs competitively, slightly outperforming KalmanNet and LLM-Filter-O. The improvement over LLM-Filter-O highlights the benefit of incorporating external prior knowledge via prompts.\n\n---\n\n\n**[Question 2]: The paper would benefit from a discussion on sample efficiency. Could you clarify the data regime of your experiments (e.g., low-sample vs. high-sample) and analyze how the relative performance of your method against baselines changes with the number of available samples? Such an analysis could be particularly insightful, as it might also explain why fine-tuning with LoRA leads to performance degradation.**\n\nThat's a good suggestion.  In our main experiments, all methods are trained using 20,000 samples per system, as detailed in Appendix B. This data volume was selected because performance across methods, including ours, saturates beyond this point. To better assess sample efficiency, we conducted additional experiments on the Selkov system by subsampling the training data to 10%, 40%, 70%, 90%, and 100% of the original size. The results are summarized in the table below:\n\n| Selkov         | 10%         | 40%        | 70%        | 90%        | 100%       |\n| -------------- | ----------- | ---------- | ---------- | ---------- | ---------- |\n| **LLM-Filter** | **10.0823** | **1.6534** | **0.4795** | **0.4072** | **0.4061** |\n| LLM-Filter-O   | NaN         | 1.8012     | 0.6837     | 0.6357     | 0.6369     |\n| MEstimator     | NaN         | 3.0156     | 0.9233     | 0.8952     | 0.8864     |\n| RStateNet      | 20.1987     | 2.8662     | 0.7410     | 0.7256     | 0.7202     |\n| ProTran        | 12.5344     | 5.1287     | 1.0511     | 1.0290     | 1.0219     |\n| KalmanNet      | NaN         | 2.2204     | 1.1783     | 1.1705     | 1.1662     |\n\nAs shown, performance degrades rapidly with reduced training data, and most methods fail to converge at the 10% level. However, LLM-Filter consistently outperforms all baselines across all data regimes.\n\nRegarding the observed performance drop when fine-tuning with LoRA, we attribute this to the limited expressiveness of low-rank parameter updates in capturing the precision required for accurate state estimation. \n\n---\n\n**[Question 3]: I am surprised that frozen LLMs lead to this level of performance. Could you explain in more detail why you think LLMs can produce a representation that is only a simple MLP away from predicting the system's hidden state? In particular, would it be possible to run experiments to gain some understanding of what kind of representations the LLM produces?**\n\nThank you for the thoughtful question. First, we observed that in the time-series forecasting domain, some studies [1, 2] have leveraged frozen LLMs. These works suggest that the pretrained knowledge embedded in LLMs can be effectively applied to forecasting tasks. This inspired us to ask: could pretrained LLMs also contain knowledge that is helpful for filtering tasks?\n\nTo investigate this, we attempted to isolate the contribution of pretrained knowledge. As suggested by Reviewer #HiuW in Question 3, we compared a pretrained LLM with a randomly initialized LLM of the same size. Training the randomly initialized model from scratch led to divergence, whereas the pretrained LLM successfully performed the filtering task and demonstrated generalization in the cross-system task. This suggests that pretrained LLMs encode useful knowledge and priors that benefit the filtering task.\n\nAdditionally, as shown in Figure 6, removing the SaP results in significant performance degradation. This suggests that LLMs are able to extract and utilize structured information about the system from the prompt, further supporting their utility in state estimation.\n\nDespite these observations, we acknowledge the difficulty in providing a system-level explanation of the internal representations LLMs produce. We tried various visualization techniques, such as attention heatmaps, hidden state visualizations, and adversarial perturbations. Unfortunately, none of these methods provided a clear explanation of how LLMs work in estimation tasks.\n\nWe have reviewed a broad body of related literature and found that even successful applications of LLMs to specialized domains (e.g., [1–6]) typically do not offer clear explanations of why LLMs perform so well. We would greatly appreciate any insights or suggestions you might have for deeper interpretability or representation analysis in this context.\n\n\n---\n\n\n**[Weakness 1]: It is generally unclear what the technical originality of this paper is. The proposed ObsEmbedding and StateProjection are standard for time series, as the authors mention. The prompting is also straightforward. Apart from the experimental evidence provided, I do not see any additional contributions. There is no in-depth analysis, explanation, or intuition regarding why the proposed approach works better than conventional learning-based filters or online Bayes filters.**\n\nThank you for the opportunity to clarify our contributions. The primary contribution of this work lies not in proposing novel architectural modules or theoretical frameworks, but in demonstrating a new paradigm for filtering: leveraging the in-context learning capabilities of LLM to build a generalizable filter. This approach enables adaptation to a wide range of dynamical systems without requiring system-specific retraining, thereby addressing a major limitation of conventional learning-based filters that rely heavily on system-specific data and training procedures.\n\nTo be honest, our motivation originated from practical challenges in robotic state estimation, particularly in quadruped robots. We initially employed KalmanNet [7], a representative learning-based filter. However, we observed that KalmanNet's performance degrades significantly when the environment changes (e.g., from indoor flat surfaces to outdoor sandy terrain), and adapting it to new settings required additional data collection and retraining, highlighting the limitations in generalization.\n\nSimultaneously, we observed that the control and time-series domains have explored the use of pretrained LLM knowledge to improve generalization. Inspired by OpenVLA [2], we initially attempted to represent continuous system states as discrete tokens within the LLM’s vocabulary space. However, we found that this discretization introduced significant precision loss, making it unsuitable for high-accuracy tasks like continuous-state estimation in filtering.\n\nFollowing insights from time-series forecasting works [4, 5], we then designed a modality alignment architecture to connect observations and states to the LLM. However, early versions of this setup struggled to generalize across systems, largely because the model lacked explicit knowledge of the underlying system dynamics. This led us to introduce the SaP, which encodes system-specific information in a format interpretable to the LLM. This component was crucial for enabling the LLM to effectively reason over both the observation history and the system characteristics.\n\nWe believe our work contributes both a practical solution and a new perspective on using pretrained LLMs for general filtering, which has not been previously explored in this form.\n\n---\n\n\n### **Reference**\n\n[1] Gruver, N., Finzi, M., Qiu, S., & Wilson, A. G. (2023). Large language models are zero-shot time series forecasters. Advances in Neural Information Processing Systems, 36, 19622-19635.\n\n[2] Liu, Y., Qin, G., Huang, X., Wang, J., & Long, M. (2024). Autotimes: Autoregressive time series forecasters via large language models. Advances in Neural Information Processing Systems, 37, 122154-122184.\n\n[3] Kim, M. J., Pertsch, K., Karamcheti, S., Xiao, T., Balakrishna, A., Nair, S., ... & Finn, C. (2024). Openvla: An open-source vision-language-action model. arXiv preprint arXiv:2406.09246.\n\n[4] Zitkovich, B., Yu, T., Xu, S., Xu, P., Xiao, T., Xia, F., ... & Han, K. (2023, December). Rt-2: Vision-language-action models transfer web knowledge to robotic control. In Conference on Robot Learning (pp. 2165-2183). PMLR.\n\n[5] Liu, X., Hu, J., Li, Y., Diao, S., Liang, Y., Hooi, B., & Zimmermann, R. (2024, May). Unitime: A language-empowered unified model for cross-domain time series forecasting. In Proceedings of the ACM Web Conference 2024 (pp. 4095-4106).\n\n[6] Shojaee, P., Meidani, K., Gupta, S., Farimani, A. B., & Reddy, C. K. (2024). Llm-sr: Scientific equation discovery via programming with large language models. arXiv preprint arXiv:2404.18400.\n\n[7] Revach, G., et al. KalmanNet: Neural network aided Kalman filtering for partially known dynamics. IEEE Transactions on Signal Processing, 70, 1532-1547."}]

Author rebuttal:

Meta-review: This paper introduces LLM-Filter, a framework that leverages large language models for state estimation in dynamical systems through modality alignment and a System-as-Prompt (SaP) mechanism. The approach is presented clearly and evaluated rigorously across diverse benchmarks, showing consistent gains over classical filters and learning-based baselines. Results highlight strong cross-system generalization, robustness under varied conditions, and a scaling effect with larger LLMs. These findings position the work as a compelling demonstration of pretrained LLMs as a technique for filtering.

The main concerns from reviewers were limited methodological novelty—since ObsEmbedding/StateProjection modules are standard from previous time series work and SaP could be viewed as prompt engineering—and the reliance on supervised training with ground-truth states. Theoretical justification for why LLM latent spaces align well with continuous states also remains underexplored. Nonetheless, reviewers agreed that the contribution lies in establishing a new paradigm rather than in architectural novelty. It is a borderline paper but given the breadth of experiments, clarity, and potential to shape future research directions, the paper perhaps offers a meaningful contribution. I recommend acceptance.
Decision: Accept (poster)