Filtered not Mixed: Filtering-Based Online Gating for Mixture of Large Language Models
Authors: Raeid Saqur
Anastasis Kratsios
Florian Krach
Yannick Limmer
Blanka Horvath
Frank Rudzicz
Venue: ICLR 2025 Poster (2025)
Keywords: Optimal filtering
LLMs
mixture-of-experts
time-series-forecasting
financial-market-movement
Primary area: foundation or frontier models, including LLMs
TL;DR: We propose a dynamic filtering algorithm for combining N expert LLM agents on online forecasting tasks with optimality guarantees of performing at least as good as the strongest expert.

Abstract: We propose MoE-F — a formalized mechanism for combining N pre-trained expert Large Language Models (LLMs) in online time-series prediction tasks by adaptively forecasting the best weighting of LLM predictions at every time step. Our mechanism leverages the conditional information in each expert's running performance to forecast the best combination of LLMs for predicting the time series in its next step. Diverging from static (learned) Mixture of Experts (MoE) methods, our approach employs time-adaptive stochastic filtering techniques to combine experts. By framing the expert selection problem as a finite state-space, continuous-time Hidden Markov model (HMM), we can leverage the Wohman-Shiryaev filter. Our approach first constructs N parallel filters corresponding to each of the N individual LLMs. Each filter proposes its best combination of LLMs, given the information that they have access to. Subsequently, the N filter outputs are optimally aggregated to maximize their robust predictive power, and this update is computed efficiently via a closed-form expression, thus generating our ensemble predictor.

Our contributions are:

- **(I)** the MoE-F algorithm — deployable as a plug-and-play filtering harness,

- **(II)** theoretical optimality guarantees of the proposed filtering-based gating algorithm (via optimality guarantees for its parallel Bayesian filtering and its robust aggregation steps), and

- **(III)** empirical evaluation and ablative results using state-of-the-art foundational and MoE LLMs on a real-world _Financial Market Movement_ task where MoE-F attains a remarkable 17% absolute and 48.5% relative F1 measure improvement over the next best performing individual LLM expert predicting short-horizon market movement based on streaming news. Further, we provide empirical evidence of substantial performance gains in applying MoE-F over specialized models in the _long-horizon time-series forecasting_ domain. Code available on github: https://github.com/raeidsaqur/moe-f

Published as a conference paper at ICLR 2025
FILTERED NOT MIXED: FILTERING-BASED ONLINE GATING FOR MIXTURE OF LARGE LANGUAGE MODELS
Raeid Saqur Anastasis Kratsios Florian Krach Yannick Limmer' Jacob-Junqi Tian*
John Willes Blanka Horvath Frank Rudzicz
1 Oxford-Man Institute for Quantitative Finance, Department of Mathematics, University of Oxford
2 "Department of Computer Science, University of Toronto
3 "Department of Mathematics, McMaster University
Vector Institute
$ Facully of Computer Science, Dalhousie University
'Department of Mathematics, ETH Zürich
toronto. edu,, rae racidsaqurées tosssuuures..toootoooooo.o.o
ca,
(limmeryy ry, horvath) 8maths.... ox.ac.ul uk, frank@dal.ca
ABSTRACT
We propose MoE-F formalized mechanism for combining N pre-trained expert Large Language Models (LLMs) in online time-series prediction tasks. MoE-F adaptively forecasts the optimal weighting of LLM predictions at each time step by leveraging the conditional information in each expert's running performance, en- abling the best combination of experts for the next step prediction. Diverging from static (learned) Mixture of Experts (MoE) methods, our approach employs time- adaptive stochastic filtering techniques combine experts. By framing the expert selection problem as finite state-space, continuous-ting Hidden Markov model (HMM), we can leverage the Wonham- Shiryaer filter. Our approach first constructs N parallel filters corresponding to each N individual LLMs. Each filter proposes best combination of LLMs, given the information that they have access to. Sub- sequently, the N filter outputs are optimally aggregated to maximize their robust predictive power, and this update is computed efficiently via a closed-form expres- sion, thus generating algorithm - deployable as plug-and-play filtering harness over any heterogenous mixture of L.LMs specialized models, (II) theoretical optimality guarantees of the proposed filtering-bases gating algorithm (via optimality guarantees for its parallel Bayesian filtering and its robust aggregation steps), and (III) empirical eval- uation and ablative results using state of the art foundational and MoE LLMs on a real-world Financial Market Movement task based on streaming news where MoE- F attains a 17% absolute and 48.5% 48.5% relative Fl-score improvement over the best performing individual L.LM expert. Further, we provide empirical evidence of sub- stantial performance gains with MoE-F over specialized models in the long-horizon time-series forecasting domain using electricity-gri datasets. Supplementary materials available https: //gttbub com http/raaidddaeeeee wtwp///reiiauuuuunneneeeeee f.
ensemble predictor. Our contributions are: (I) the MoE-F
INTRODUCTION
Mixture of expert models (MoEs), such as Liu et al. [2024a;b], Guo et al. [2024] the seminal Switch Transformers [Fedus et al., 2022], and more recently Mixtral [Jiang et al., 2024], Gemini [Google], DBRX [MosaicAI, 2024] and many others (e.g. [Saad et al., 2023, Chowdhury et al., 2023, Li et al., 2024, Puigcerver et al., 2024]), have taken a center stage the generative AI zeitgeist since they allow the number of parameters in large language models (LLMs) to be scaled-up while maintaining a roughly constant computational cost. This is due to the sparse activation strategy employed by MoEs, wherein several offline experts banked, and gating network routes each input small subset expert models when generating a prediction [Zhou et al., 2022]. Thus, only a few experts must be loaded into active memory at any given time, to generate a prediction from

---

Published as a conference paper at ICLR 2025
any novel imput. Currently, most MoE pipelines are designed for static tasks, i.c. tasks without temporal structure; consequentially these pipelines are limited to gating mechanisms which are constant-in constant-in-time. However, many prediction problems have temporal structure which is not being leveraged by these classical constant-iv -imeme pipelines; examples include time-series data appearing in physics [Mao, 2007], finance [Sharp, 1990], decision science [Feinberg & Shwartz, 2012], or most reservoir computing applications [Jaeger, 2002].
Dynamic prediction problems are fundamentally different from static ones, since each time new instance arrives a prediction is generated by each expert, hence, the user progressively gains more information to which expert models are the strongest predictors for given task. This information (observed measurements) can then be fed back to the MoE's 's pipeline and used to update its gating mechanisms to select the optimal combination of experts. The key challenge is that we cannot directly observe which is the true best expert(s) to use in predicting the target.
We consider the finite state-space process, which selects the optimal expert, as the unobservable signal process. The problem of best estimating this signal process, given the information from observable measurements, is precisely the continuous-
and finite state-space stochastic filtering problem initially studied by Wonham [1964]. The solution to a generalized version of that component of our problem, considered Liptser& & Shiriaev [1977], is a so-called stochastic (optimal) filter. It is a closed-fome recursion updating the best estimate of the unobservable signal process (the best expert to use in our case) given the observable measurements (the performance of each expert), formalized by precision for the conditional distribution of the signal process.
Our Mixture-of- -ffperr-of-Experts Filter (MoE-F) mechanism incorporates N stochastic filters, implemented in parallel, update the gating mechanism, which routes any new input of the mixture model an optimal combination of our N expert models given their measured historical performance. These N predictions generated by the parallel stochastic filters are then robustly aggregated into a single robust mixture prediction by a mechanism similar to those used in PAC-Bayes theory, e.g. Alquier [2008]. Rothfuss al. [2021]. In this way, our proposed MoE-F algorithm optimally predicts the best mixture of expert models while dynamically adapting itself to the observed performance of each expert model. We emphasize that this type of synnami--uddatng procedure only possible due the temporal structure of the time-series prediction tasks we interested in.
Contributions. We construct an online gating mechanism (Algorithm 1) for an online mixture of expert (LLM) models for time-series prediction tasks. Using tools from stochastic calculus and stochastic filtering theory, we prove the optimality of our online gating mechanism's 's first parallel Bayesian estimation stage (Theorem 1). Using tools from the theory of Markov chains and from Gibbs-measures we demonstrate the optimality of the robust aggregation phase (Theorem 2).
Outline. Section $2 covers the necessary background, including our probabilistic framework and continuous-time Markov chains theory. Section $3 explicates the MoE-F algorithm. Section $4 provides theoretical guarantees for the two phases of MoE-F algorithm, with proof's in the appendix. Section $5 demonstrates the large-scale application of our algorithm on real-world online classification and regression tasks. For classification, we evaluate the online prediction of financial market movements using the NIFTY dataset [Saqur et al., 2024] and concurrent SOTA LLM experts. For regression, we showcase the filter's 's utility in long-tent term time-series forecasting (LTSF) using the ETTh1, ETTh2 [Zhou et al., 2021] datasets with concurrent specialized LTSF models.
Figure 1: A visualization of MoE-F's 's application as filtering harness. Depicts seven SOTA LLMs predicting market movement direction over three randomly sampled windows of seven (trading) days across varying market regimes- left: mixed market with high fluctuations, middle: neutral, and right: bearish market. In all sub-plots, the ground-truth (market) trajectory in black, and the filtered trajectory depicted in dotted green. All other experts' (Table 2) predictions are overlayed as scatter-plot points. No values for non-tradin non-trading days.
2

---

Published as a conference paper at ICLR 2025
2 PRELIMINARIES
Notations for Markov Chains Matrices. For each N E N4, we use Py to denote the set of N x N row-stochastic matrices; i.e. P E Pr if each row of P is vector in [0, 1] whose entries sum to We use P to denote the set of "uniform row-stochastic matrices", meaning the set of P E PN
with identical rows. For each such N, we use QN to denote the set of N x N Markov Q/intensity matrices; that is, Q E 2N if n..m 2 0 whenever n + m, for ni m = ,,..,,,A N and the rows of Q sum to O. For any such Q, the matrix exponential exp(tQ) is row-stochastic matrix for every t 0.
The 8 norm of N x M matrix X will be denoted by || 0Z
def. max
[X
T=I....N
m=.......
Probability Theory. Ze [0, oc) + Fdd be a continuously differentiable path, and consider a complete filtered probability space (2,, def. (Fi)ezo,)) with right-
filtration F and supporting a 1-dimensional Brownian motion W. def. (W1)120- We want to predict a 1-dimensional targer (stochastic) process _ (Y2) using our MoE models F, as described in $ 1. Assume Y- evoolvss according to the dynamics in Equation equation (1). The process w. is assumed to be hidden Markov process, taking values in the standard basis {ex J.W of R.V. Its evolution is governed by the intensity (or Q) matrix Q [0, oc) -> NKN, which describes the rate at which the transition probabilities of the Markov chain w. change.
Formally, for i,j = L,...., N and t, A > 0 its (i. entry Qi det (Qt).. is determined by
P(wi+A = we eÿ) I== A Q.¹1 c(A).
We provide additional details on stochastic filtering and matrix operations in Appendix D, that are not required to formulate our main guarantee (Theorem 1) but are included for completeness of proofs.
THE MOE FILTERING (MoE-F) ALGORITHM
Our setting can be formalized follows. We consider an input signal z., d-dimensional smooth path, and the target process Y., a continuous-timg 1-dimensional stochastic process. We are given N auto-regressive, causal pre-trained expert models, f(),..., f(N) Urzo C(O0, t), R²) + E which map input paths such as x. to one-dimensional predictions in time such that their predictions do not depend on future states of x. Y. when predicting at any time 2 0.
Instead of treating the target as being static we allow it to evolve dynamically in time. We assume that there is Hidden Markov process dictating which expert best approximates Y., up some Brownian measurement noise W.; thus, we postulate that Y. evolves according to the stochastic differential equation (SDE) with stochastic drift given by the equation (I):
Y = Yo + f w F(I)o.)) ds +
L dW.,
Best Export Estimate
Idiosyncratic Rosidual Noise
where the Markov process w. randomly masks all but expert at any given time and concatenates the experts F( del (f(r) (zTo.))) The first term in equation (1), fo w F(Toos) ds, represents the true best sparse approximation of Y; i.e. the maximally sparse "single expert" - by the ensemble of experts F. The fo dW. term, is additive Gaussian noise with mean and variance t.
Thus, the mixture coefficients w. act as an unobservable signal which is indirectly observed through the measurements recorded by each expert's running performance (!r) def (el") )=20. where e!m) is the loss incurred by the th expert at time f. > 0 as quantified by the chosen loss-function é: IR. x IR + TR.
This online mixture of experts problem a two-fold problem which can be solved by our proposed MoE-F procedure. In the first phase, we solve N stochastic filtering problems in parallel, which optimally estimate w., any given time 2 0, given the measurement performance of any one expert )o The second phase of the MoE-F algorithm aggregates these optimal predictions.
Josest.
3

---

Published as a conference paper at ICLR 2025
3.1 THE MOE FILTERING
Our MoE-F procedure (Algorithm 1) summarized in Figure 2 operates in two steps, both of which efficiently computable due closed-form expressions.
Input Signal
Step 1 Optimal Parallel Filtering
Step 2 Robust Aggregation
Export
Expert
9000
Mechanism
Update Performance
Expert
(P-M)
(M0
Feedback
Figure 2: MoE-F Mechanism: conceptual depiction of an input signal J. evolving in IR" with N experts (T)).
In the first phase of our algorithm, we solve N distinct stochastic filtering problems in parallel. Each nh expert provides an optimal Bayesian prediction P((n) of the Y equation (1) based on their running performance e!a)) up to the current time 2 0. Additionally, each expert generates a ranking Ti reflecting the reliability of all experts' performance,including its own, up that point. The optimality of this update is guaranteed by Theorem 1.
Next, these optimal Bayesian predictions from each expert are robustly aggregated by central module. This module first computes robust version म by aggregating the reliability scores of each expert's filter {T(-)) j and then uses it compute robust ensemble prediction Σ #y (((1) for the target Y, Finally, the dynamics of the hidden Markov chain w. in equation (1), encoded by its so-called Q/intensity matrix, are re-estimated
robust fashion using the rumning performance of each of the N experts up until the current time 2 0. The optimality of this robust aggregation step provided by Theorem 2.
Loss functions. In this paper, we consider either the binary cross entropy (BCE) or mean-squared error (MSE) as loss functions. These are respectively defined by
(( y lo)()) (0) + +1 - y) log(1 - ú), BCE,
01.
MSE,
where y, ÿ belong to R in the regression case and to [0. 1] in the classification case. Though the setting in equation (1) directly applies to unbounded targets, like in regression, one can also use in classification by simply taking Y. to be the logits (inverse logistic transform) of the classification probabilities. Since the logistic function is a bi-muaaureble bijection, the filtration generated by the logits or the classification probability processes equal; thus, the filter does not change.
Helper Functions. For a concise presentation of our MoE-F algorithm (Algorithm 1), we now introduce few helper functions: A, A, B, F. Intuitively, the first two helper functions, A, and
A, compute the gradient and average the gradient of the nh expert's (H.! (")) prediction of the target path y. up to time f, accounting for the sensitivity of the th expert to changes in the input path at time f. Here, the averaging is taken uniformly over which "latent expert" is active. We rely on the dine-derivative of the loss function evaluated at the th expert's 's model, which is given in closed-form
(i- (rl"))) log [- (w) del (1- (1m)))))))))))) pintap))))
mmm] F(Mpp))))
if is BCE,
( (Iroa)))) (mn F(T - -A p (1poo)) +1)
if is MSE.
We will also use the concatenated and averaged gradients of the loss of each expert, denoted respectively by A, and As, and are defined as:
Arw Yi) def. (w,
(w, Yi)
IEASN and A,Th) (x) Me def.
( (es, (r)))s))
(3)
where the sensitivity of the th expert at time to changes in the input path x. is = (()))))))))))) (x))o_)))
A ( de. lim (f(")
p(n) (x70.0-1))
=10
4

---

Published as a conference paper at ICLR 2025
Helper function B quantifies the gradient of the loss function (BCE MSE) of the n expert's prediction w.r.t. the observed target y., ignoring changes in the input path x., and is given by:
p(m) (srooee)
log
i BCE,
B
( def
(=
(4)
2 (v:
((() (xT0.))) i MSE.
The helper function F concatenates the N experts F( def (xpo)))... ((M) (1).))))
As illustrated by Figure 2, our MoE-F (Algorithm 1) operates in two steps. The first step (lines 3-11) implements N parallel stochastic filters yielding optimal Bayesian predictions of Y. given the information available in the running loss function of each of the N expert models. The second step (lines 12 - 16) aggregates the N predictions of each expert model in robust manner, and it performs a bi-level optimization to compute our most robust Markov Q/intensity matrix supporting the running performance of each expert thus far.
Step 1 - Optimal Parallel Filtering, Lines 3-8 of our MoE-F algorithm is a time discretization of the stochastic optimal filtering equations in equation (5) according to the standard scheme, see e.g. Hutzenthaler & Jentzen [2015]. This is performable in parallel, with each parallel computing branch corresponding to a single expert. As in discrete-time filtering implementations, e.g. Grewal & Andrews [2014]. Kim et al. [2018], the discretized infinitesimal change in the running loss of the th expert is given by (o Rs 4, ((") provided that > A > 0. Thus, the innovations process W is the change in ((n) (n()(0))
over the time increment [t A minus A(((o.))) re-normalized by B; (), Importantly, this means that is computable online from the incoming larget process Y.. In line 8, each expert (still in parallel) proposes their best estimate Y," for Y, using their mixture weights, estimated only using their (local) information in their historical loss.
Finally, in line 10, the reliability of each expert is measured (still in parallel) by evaluating the loss between the target process Y, and their best (individual) prediction Y," y of it. This reliability metric is summarized by their score 8n of the th expert.
Step 2 Robust Aggregation. The prediction quality of each expert Y, )-(") has been compressed into a score
A Algorithm 1: MoE-F Algorithm
1 Initialize v, P, and Q:
Su. These scores used to aggre-
2 /* step 1: Optimal Parallel Filter x/
gate the expert predictions into a single
3 For n = ,....,N N in parallel
prediction Y. of Y. This is done us- ing the
drift + Q (-(=;
Gibbs- aggregation 5
AL + {(f (ff (To.O) (()) (1)) Y(-1):
approach with the softmin function (since lower loss implies better score here). Line 13 computes the aggregation weights म; which are then packaged into a uniform row-stockis cow-stochastic matrix whose
AW + (AL - A
(n) Yp.e- (1) (Yi-)):
diff: + (mx) (Ai-1 (ei, Yo.--1))
Á (n) Yuua-11)) /)//)))///// [ (Yg-1);
me < m + drift + dill AW:
rows are given by म. With these aggre- gation weights, our MoE-F algorithm generates a joint predictor in line 14. We associate P to a Markov Q/intensity ma- end
p((n) + (m,
(m)
F((p.));
trix Q. The matrix logarithm of P is
12 /* Step 2: Robust Aggregation
viable candidate for Q, since exp(1 - Q) # <- (e- / / ( )=)
N mi:
would be valid transition matrix; how- Y, del. del ( (V(1)) nn
ever, in general, the matrix logarithm is for n = I...., N do of an arbitrary row-stochastic transition matrix is not a valid intensity matrix. end for
Pr.. +
Upon regularizing P in line 18 to avoid 1s < (1 a) P alv:
pathologies, in line 19, we compute the Q ReI.U(log( P)) di ReL.U(log(P))) best intensity matrix approximation of return MoE-F Prediction y,; its matrix logarithm.
4 THEORETICAL GUARANTEES
Our MoE-F algorithm revolves around the following set of stochastic filtering equations. Each filtering equation corresponds to the best estimate of single expert on masking process w. given only
5

---

Published as a conference paper at ICLR 2025
the information available in their running performance. Interpreting "best estimate" in the L sense, each seeks to predict the conditional distribution of w. given the o-algebra J, ((") of (6) logest generated by the running loss process g!") def (E(Y,, Y; De.
(P(w)) (w ei )))))
An essential property of our filtering equations are that they provide a closed-form recursion for (R) def (T (R) 1220 depending only each of the expert models and the accumulating observations from the target process Y.. This key property is rare in stochastic filtering paradigm, voiding the need for any Monte-Carlo simulation.
4.1 GUARANTEES FOR STEP ONLINE PARALLEL FILTERING
Our main result comes in two variants, binary classification and a regression variant. Both versions of our guarantees operate under the following regularity conditions of the path æ..
Assumptions 4.1 (Regularity Conditions). The path z. : [0, oo) - R once continuously differen- tiable and there is constant C > O such that, for each t 2 O and i,j = 1..... d, (Qi)a. S. The loss function either the binary cross entropy loss or the squared-norm loss.
Theorem (Optimal Optimistic Prior for nh Expert). Under Assumption 4.1, the best a posteriori estimate of the thth expert, [m) satiisies the SDE
T (m:i) (Aolei, Yo..)) As(T!))) M...a))) dw(") (5)
(m:i) = To ((..) ds { Rt
(Y,))))))
where (Qt), denotes the 7th row of the transitions matrix Qs time t 0, x th P(wo e.). The --r-oBroinn motion
"innovations process" W 4tt (W: (W))) Jezo is the following
W dL (m)
Aa(Yo.)))
[0.u]
du,
(6)
Ba(Yoovi)
and each of the n. "running loss processes" (m) satisfy:
(i) Classification: If is the binary cross entropy loss:
(Y, (( g)) (((0()))))) log ((m(())))) (())))))))))))
(amp,)))
dI (n) [00)
(w)] F( de log
dW.
(1 (oroer))))))) ((11))))))))
(())))))))
(ii) Regression: E is the squared-norm loss
an -(Y (Iw) F(aao)))) ( celn(o")) +2(X) (00) (10..)))))) dWi.
4.2 GUARANTEES FOR STEP ROBUST AGGREGATION
The following bi-level optimality guarantee justifies the second step of our MoE-F algorithm.
Theorem 2 (Bi-level Robust Updates the Q-Matrix). Let Y ,..,, VIN) Y R and é TR2 IR. be Borel measurable. Then def. (#)) IT; where del Softmin (A (e(y Y)) minimizes
min PEPU
Pmn, e( Yi)
Pm,, log(P P NN).
(Inner)
m- I....N
Predictive Poweer of Ensemble
Entmpic Regularization
Let a E 1) then pa detef (1 (). 7) + alN is a row stochastic matrix and, for a < 1 large enough, (og is well-defined. Furthermore, Q def ReLU [log(")) is minimizer of
mingee | log(?? (lo
(Outer)

---

Published as a conference paper at ICLR 2025
We close our guarantees section by investigating the effect of regularizing the row stochastic matrix P obtained in the inner-level optimization equation (Inner) to ensure its invertibility and the well- posedness of its matrix logarithm. We ask whether the probability distributions defined by the rows of Ph (whose state-space is (en, n==1 depend continuously on a. Indeed, the next result shows that the KI.-divergence of the rows of Pt and those of P. (i.e. π) necessarily very close when is small. Proposition 1 (Stability of Perturbations). In the setting of Proposition 2, we have that log(T (Tinn)
max (L() S 2a ( (- 1/TTiin g
log((1 a) kmin)
1/((1 a) mmiu) where Tmin miniii,...,, T(> 0) and (P.); denotes the tth row of Pf
EXPERIMENTS
5.1 FINANCIAL MARKET MOVEMENT (FMM) TASK ON NIFTY USING MOE-F
We show the efficacy of our proposed MoE-F algorithm by using various SOTA class large language models on the financial market movement prediction task.
Task. The Financial Market Movement (FMM) prediction task for experts' evaluation can be defined as a ternary or binary market movement direction classification task among the labels' set 'Fall', 'Neutral', 'Rise' conditioned on history (or, expert memory) of window size H (i.e., on the time window |t |t- H + 1, ()) similar to the auto-regressive or causal generative language model (causal LM) training objective.
{ C==
Experts. We consider a diverse list of SOTA general purpose instruction-tunel LLMs as experts for the experiments our proposed MoE-F algorithm. For single LLM experts, weights models: Llama-2 2 (7B, 70B), Llama-3 (8B, 70B) [Touvron et al., 2023]. For mixture of experts (MoE) architecture models, we use SOTA open-weights models: Mixtral (8x7B) [Jiang al., 2024] - which is a mixture of 8 Mistral (7B) [Jiang et al., 2023] models - and DBRX-Instruct [MosaicAI, 2024] with 132B total parameters and a mixture of 16 (fine-grained, smaller, 65x more combinations of) experts. For evaluation, we deployed these open-weights models as vLLM [Kwon et al., 2023] OpenAl compatible API endpoints and ran the dataset queries against them. We use API/model configurations like guided-choic and max-tokens to format class label converged expert responses alongside specific prompt instructions. We use the closed-source latest variants of the GPT-4 [OpenAI, 2023] class of models: GPT4o, using the OpenAl API. These experts are leading foundation models on current performance benehmarks on language understanding (MMLU [Hendryeks et al., 202 1 }), programming (HumanEval [Chen et al., 2021]), I]). math (GSM8K [Cobbe et al., 2021]) tasks and other concurrent LLM benchmarks [Song et al., 2023, Liang et al., 2022].
Meta's 's open-
Datasets. For real-world experiments the defined FMM task, we use the US equities market movement (NYSE ticker: $SPY) dataset NIFTY (DLM) [Saqur et al., 2024]. The test split statistics which we used are recorded in Table 1.
Table 1: Statistics of NIFTY test split
Each sample of the DLAS contains high-quality, processed (one-turn) conversational queries for an expert instruction fine-tuned LLM, where query, sry. comprises of prompt zrf and response xx,, i.e., x4 = (at,; 2t)) corresponding to day (or, time-step) t.
Category
Statisties
Number of days ( (T) increment (At)
317/1 I
Lahel support (Fall / Neutral / Rise)
73 / 143 101
Date range (start to end)
2019-02-13 to 2020-09-21
For evaluation, each time step f, an expert LLM prompted (x/)) to predict the market movement the following day (i.e., ++ 1), based on the market's current contextual information (relevant financial news headlines and the market's financial numerics like the standard OHICV and common technical indicators - from past few days capturing trends). Fig. 3 depicts a snapshot of an expert prompt at for clucidation. Please see Fig. 6 in Appendix $C.1 for details.
For our ablative experiments, we utilise three additional datasets, namely StockNet aka. ACL18, BigData22 and CIKM datasets [Xu & Cohen, 2018, Soun et al., 2022, Wu et al., 2018]. These datasets have similar overarching (FMM) task design, but the prompted contextual information (e.g. social media opinions) and targeted asset classes (c.g. individual stock tickers, or price of gold) differ. We delegate the full details of these datasets in C.3 of Appendix &C Datasets.
7

---

Published as a conference paper at ICLR 2025
Lauder Goob Coranaviuus
Conumptiin | Hedge-Furt Billonoire's Deal Collopses Fed's Response Coranoviuus
Shows Coronovinus
Curba
Current Stance Interesl Roles Approprials
Revenue
Deport Layoff;: INYSE Owner
Topped
Potenticl ebay Deal TMobll Projeds More Customer Goiss 2020| Auroro Cannabis Chiel
Rolly Publishing Digess
GenConne
bankouptey | Risky Corporate Swamps Asio, Pashing
Stage
Floge Warnin-light Problem Poultry
Weight
Shares NYSE
liters bank Raily Caphal Group KKllogg
2020 York Times
Subacription Growth
Mouchin
Be lass
AroalorMinal Posls Earnings Deaphe Timas Stealnoters Canadian Antirue Profi,
Officiah Probe Giants Zantac Recall Weight Sanaft's Earninge
Figure 3: Example snapshot epidemic (the text colors here convey negative and positive sentiments). An expert policy, TLM prompt composed of task instruction as prefix, concatenated with the market context, and this news value concatenated: s.t. Ip ( Feontent; Tnews).
'news' component on 2020-02 02-0002--6222222 at the upstart the global coronavirus
Openyll
0,253
Mittral Instruct v6.1
0.225
Lime-S-tB-ttarruu
0,200
Experts
-0.20
samai22 2-720 chat-hf
00.75
0.75
Lima-d-- cop-inatrad
0150
0.150
DBRK Instruct-
0.125
0.10
20t 1p 2018 aife 40-43-221
2019-03-25
2019-12.6
200-1-11
200-47-13
2000-7--2
202-47
200-4722
22
206-7-227
201
Figure 4: Heatmap of expert weights and subsequent rankings for the sampled windows in Fig. 1.
Table Results of applying MoE-F seven SOTA experts on the NIFTY (test splif). All values mcan of (random) runs except GPT-4o. Best value in each metric row is in bold. Confusion matrices appendix Fig.
LLM Experts
Experts Filter
Llama-2
Llama-2
Llama-3
Llama-3
Mixtral-8ae -x7BBl
DBRX Instruct
OpenAl GPT-4o
MoE-F
Metrics
7b-chat
70b-chat
8B-Instruct
70B-Instruct
Instruct-v0.1
(ours)
F1
0.22
0.33
0.35
0.20
0.34
0.34
0.31
0.52
Acc
0.27
0.37
0.39
0.30
0.33
0.34
0.37
0.57
Precision
0.35
0.33
0.31
0.32
0.36
0.36
0.33
0.61
Recall
0.27
0.37
0.39
0.30
0.33
0.34
0.37
0.57
5.1.1
RESULTS: MoE-F ON MIXTURE OF (MIXTURE OF) EXPERTS
Table 2 shows the results of running MoE-F on the collection of SOTA experts. While none of these remarkable experts out-performs others exceedingly or has an overall outstanding performance on the FMM task, filtering the expert decisions using MoE-F yields remarkable performance both in terms of overall task results and gains with 17% % real and 48.5% relative F1 measure improvement over the next best performing expert values (= 35%). Fig. depicts our MoE-F mechanism's decisions trajectory overlayed on the true market movement and expert decision trajectories.
Ablations. We use the Llama [Touvron et al., 2023] instruction-tuned, open-weights class of experts - specifically, Llama-2 7B, Llama-3 8B variants ablation experiments, we create 4 additional expert variants (as standard LoRA [Hu et al.. 2021] LLM adapters) by fine-tuning the base expert with the stock movement datasets, namely: (nifty, acl1 11l18, 8, c kml 8, bi gddata22 and run them against the same task and evaluation test split from before. Table shows the combined results Llama and 3 along with the 4 expert variants.
further analyse MoE-F work. For
Table 3: Performance of 2lama-2-766 2-7-chht and Llama-3-88 Instrrutttt base models with (SFT LoRA adapter)
variants the NIFTY Stock Price Movement Prediction Task (lest split).
Llama-2-7b-thh
-lama-3-8B-Instree
Metries
Base
+ailly
+ael18
+bigdata22
+
MoE-F
+aifly
+ac118
+bigdata22
+
MoE.F
F1 Score
0.22
0.28
0.20
0.29
0.27
0.43
0.34
0.36
0.19
0.23
0.24
0.43
Accuracy
0.27
0.45
0.25
0.29
0.27
0.45
0.39
0.41
0.26
0.26
0.28
0.47
Precision
0.35
0,20
0.36
0.32
0.31
0.44
0.31
0.56
0.44
0,28
0.31
0.48
Recall
0.27
0.45
0.25
0.29
0.27
0.45
0.39
0.41
0.26
0.26
0.28
0.47
8

---

Published as a conference paper at ICLR 2025
Table 4: Llama-2, experts and MoE-F's performance decomposition by class labels on NIFTY test split.
Fall (support: 73)
Neutral (support: 143)
Risc (support: 101)
Expert
F1
Precision
Recall
F1
Precision
Recall
F1
Precision
Recall
Llama-2-7b-chee
0.35
0.23
0.75
0.19
0.40
0.13
0.17
0.34
0.11
nifty
0.00
0.00
0.00
0.62
0.45
1.00
0.00
0.00
0.00
acl18
0.28
0.20
0.45
0.08
0.46
0.04
0.32
0.28
0.39
cikm18
0.21
0.18
0,25
0.34
0,43
0.29
0.29
0,26
0.32
bigdata22
0.21
0.19
0.23
0.31
0.42
0.24
0.31
0.26
0.38
MoE-F
0.30
0.27
0.33
0.60
0.54
0.66
0.30
0.43
0.23
3-8t-Instructt
0.00
0.00
0.00
0.49
0.46
0.51
0.38
0.31
0.49
nifty
0.00
0.00
0.00
0.48
0.46
0.51
0.42
0.35
0.54
acl18
0.15
0.17
0.14
0.04
0.30
0.02
0.41
0,29
0.70
cikm18
0.13
0.16
0.11
0.27
0.42
0.20
0.39
0.29
0.58
bigdata22
0.14
0.17
0.12
0.20
0.38
0.13
0.40
0.29
0.62
MoE-F
0.12
0.45
0.07
0.56
0.56
0.57
0.48
0.39
0.62
Discussions. As before, using MoE-F delivers superior results for each the two cases. However, notice that the overall performance the FMM task is lower than from earlier: 43% compared to 52% (Table 2). This aligns with our proposed theoretical performance guarantee relative to experts. Increasing the quality and quantity experts with specialized capabilities improves MoE-F results. Intutitively, in presence of vastly superior expert, MoE-F will levy higher weight on it for its decisions. If any under-perforr pedddrrrefoomming (specialized) expert does better over a certain window (say during bearish market regime), it percolates up the decision weighting map accordingly.
Decomposing Expert Performance by Class Labels. Table 4 tabulates experts' performance by each of the three movement labels detailing another insight and verification of MoE-F's mechanism at work. While some experts have overall equivalent performance (e.g. the F1 score of '+nifty' and '+bigdata22' adapters), examining their label-specific results show these experts performance emanates from different decisions. To elucidate, the Llama-2 base expert was far better in predicting market 'Fall' than Llama-3, however, the latter handily outperforms when predicting 'Rise', Similarly, we also see some experts are degenerate for some label predictions, like the '+nifty' expert that only tends to predict 'Neutral' (the imbalanced class label with highest support).
5.2 TIME-SERIES FORECASTING (TSF) USING MOE-F
The mainstream practice of using the classic Mean Squared Error (MSE) in the TSF field makes problems in the domain suitable test-bed for our regression-best based theorem (Theorem 4).
Task. Letting represent the length of some historical observation window, and H the prediction horizon, generic TSF problem can be formalized
[[c+14H]] = =
where zit- 1:t]] LxC and +1+1++01 E #R Here, is the number of distinct features or
channels of an agent's observation; the (MSE) loss measures the discrepancy between the predicted (i)
values (1)
[+++++]] and the ground truth Vit+1: as = Σ 1
(6) _10) P[t1itH
5.2.1 EXPERIMENTAL SETUP
Experts and Datasets. Table presents few, among plethora of concurrent models in the busy TSF research area that come in a variety of flavours, from specialized TSF tasks only models like N-HiTS, N-BEATS [Challu et al., 2023, Oreshkin et al., 2019] 9] customized or fine-tuned LLMS like LLMTime [Gruver et al., 2024]. Immediately recent works in both these categories like the SAMFormer [Ilbert et al., 2024] and MoiRai [Woo et al., 2024] - pushes the envelope on ideas and performance further. The goal of the TSF experiments here is not to achieve superior performance, but to show the application
our online MoE-F harness over any N arbitrary expert models and effectively combine their strengths in inexpensive, heuristic manner. Thus, for applying filter, we chose three recent, well-adopted models with standardized implementation: DLinear, PStchTST, Sparse'TSF [Zeng al., 2023, Nie al., 2022, Lin al., 2024] using the author provided code from the latter to replicate the models' predictions over varying horizon H.

---

Published as a conference paper at ICLR 2025
Figure S: Electricity Transformer.
Similarly, we picked the most common and widely adopted datasets among a slew of mainstream TSF datasets (e.g. Electricity [Dua & Graff, 2017], Traffic [California Department of Trans- portation, 2023]) from concurrent literature in the LTSF domain: ETTh1&ETTH [Zhou al., 2021]. Fig. 5 presents the summary of these TSF datasets.
Datasets
ETThl & ETTh2
Channels
Frequency Timesteps
hourly
17,420
Results. Table presents MoE-F applied to a set of concurrent TSF experts. While our replicated performances of the bottom three experts, used for filtering, are lower than published results, the filtered predictions using MoE-F gains strong performance improvement using only three experts as-is and without any hyperparameters tuning (like optimal lambda, alpha values).
Table S: MSE results of multivariate Long-term Time-series Forecasting comparing contemporary SOTA expert models and applying our proposed filtering harness MoE-F. Best results are highlighted in bold. Second best results underlined. Replicated performance results are presented using decimal places. Other performances quoted from respective papers. Only the bottom three models were used to generate MoE-F results.
Dataset
ETThl
ETTh2
Horizon
96
192
336
720
192
336
720
Informer [Zhou et al., 2021]
0.865
1.008
1.107
1.181
3.755
5,602
4.721
3.647
Autoformer [Wu et al., 2021]
0.449
0.500
0,521
0.514
0.645
0.788
0.957
0.792
Pyraformer [Liu et al., 202
0.664
0.790
0.891
0.963
0.645
0.788
0.907
0.963
FITS (2024) [Xu et al., 2024]
0.375
0.408
0.429
0.427
0.274
0.333
0.340
0.374
PatchTST |Nic ct al., 2022]
0.6897
0.6676
0.5997
0,6873
0.4273
0,4964
2.6450
0.4324
DLinear [Zeng et al., 2023]
0,3788
0.4212
0.4520
0.5230
0.2908
0.4091
0.5320
0.7430
SparseTSF [Lin et al., 2024]
0.3631
0.4000
0.4346
0.4238
0.2945
0.3399
0.3595
0,3831
MoE-F (ours)
0.3630
0.4165
0.4178
0.4157
0.2911
0.3294
0.4840
0.3917
RELATED WORK
Closed-Form Finite Dimensional Filters. Our proposed MoE-F utilizes miite-dimensional closed-form filtering equations via the Wonham-Shiryaen filter Wonham [1964], Sirjaev [1965]. Such closed-form filtering equations rarely exist as the general stochastic filtering theory is infinite- dimensional Stratonovich [1959; 1960]. Zakai [1969] and thus computationally intractable in practice. filter, there only a handful of such fnite-dimensional filters with closed-form recursions. E.g. the Kalman-But
Other than the Wonham-Shing
filter Kalman [1960], Bucy & Joseph [2005] or the Benes filter Beneš [1981], both of which require highly rigid assumptions. Otherwise, one does typically have access to interacting systems of particles, so-called particle filters Djuric et al. [2003], Del Moral [2013], which are computationally intractable in high dimensions.
Bayesian Mixture of Experts. In the context of mixtures of experts, or large foundation models, one typically relies on gating mechanism, or learned routing among expert models Jacobs et al. [1991], Jordan & Jacobs [1994]. These gating mechanisms are often using static Bayesian optimization approach via Gibbs posterior mixtures; c.g. Alquier [2008], Alquier et al. [2016], Andrychowicz et al. [2016], Rothfuss et al. [2021; 2023]. What the gating mechanisms involved in these approaches have in common that they all are static in the sense that they do not learn in an online manner from dynamically arriving inputs and feedback. In contrast to these, our MoB-F model is online/dynamic Bayesian optimization algorithm which dynamically generates posteriors using a different (L²) notion of optimality rather than the extremal version used to define (static) Gibbs posterior mixtures.
Learned Routing vs. Ours. Most concurrent MoE of LLMs build on the idea of trainable/learred experts mixing, and use a routing layer Zhou et al. [2022] as introduced by Jacobs et al. [1991], Jordan & Jacobs [1994] in the 90s. Their efficacy was later shown Shazeer al. [2017] and numerous models and variants of this core idea have been successfully showcased in the current LLM context Fedus et al. [2022], Jiang et al. [2024], Dai et al. [2024], Gale al. [2023]. Our work is orthogonal to this approach since this framework has no dependency on learned routing. Our online heuristic approach alongside with formal optimality guarantee allows online adaptation like adding, removing or hot-swapping experts on the fly.
10

---

Published as a conference paper at ICLR 2025
7 CONCLUSION
We introduced a filtering- based gating mechanism for dynamically adapting mixing procedures to incoming data, unlike static classical MoE methods. Our algorithm enjoys optimality (Theorem 1) and robustness guarantees (Theorem 2) and effectively performs TSF in two orthogonal settings.
Limitations & Future Work. Our numerical experiments focused solely on LLM experts, excluding random structures like neural SDEs Kidger movements and stochastic effects. Future work will integrate both LLM and SDE-based experts to capture news-driven and technical market dynamics.
[2021], potentially missing technical market
ACKNOWLEDGMENT
Anastasis Kratsios acknowledges financial support from an NSERC Discovery Grant No. RGPIN- 2023-04482 and their McMaster Startup Funds. Raeid Saqur is supported by Canada NSERC CGS-D Doctoral Grant. The authors acknowledge that resources used in preparing this research were provided, in part, by the Province of Ontario, the Government of Canada through CIFAR, and companies sponsoring the Vector Institute https:/vectorinstiuute httpttttaai.aaateesssssseeeieeee prattpp/ partners/. The authors would like to thank Marshall Wang for helping with reference code for computing DBRX experiments.
ETHICS STATEMENT
This research on stochastic filtering-bsedd gating mechanisms for online MoE foundation models presents significant advancements, with broad plausible societal implications. Specifically, this technology can enhance adaptability in scientific predictions using complex dynamic systems, which important in various disciplines including (but not limited to) financial markets, healtheare, and climate science for example. The ability to integrate multiple expert models dynamically can lead to more informed decision-making that is less constrained than conventional methods. However, this technology is further refined, will be important to prevent misuse in situations where advantage may be unfairly balanced, as high-frre frehuerrgnncyy trading. This technology should be used in transparent, accountablc, and regularly audited frameworks to ensure its responsible deployment.
We note that access to highly-capable pre-trained foundation models is publicly available through open-weight LLM repositories, such as HuggingFace. The MoE-F filtering harness can be utilized enhance the performance of many downstream tasks using ensemble of off-the-shelf expert models, without the need for costly fine-tuning, However, there is growing concern about the lack of transparency concerning the training data used in these models. Consequently, any inherent biases in the experts could be exacerbated by the filtering process. For example, biased mortgage risk evaluating model can persistently rank higher due predicting a majority/minority label.
REPRODUCIBILITY STATEMENT
In our commitment advancing open science and fostering transparency, we strive to make all aspects of our research fully accessible to the broader scientific community. To ensure the reproducibility of our work, we have included relevant datasets, detailed results, and links code repositories (or anonymized links, where appropriate) as supplementary materials. These resources have been provided to facilitate the replication and verification of our findings, in alignment with the principles of the double-blind review process.
'Supplementary material, including test results and scripts to replicate the results presented here, is available https::////..........tt //g ttw.........t.t.////////t..... thub. com/ rasi dsaqur rasquu/net tsaqur/noe-f.
11

---

Published as a conference paper at ICLR 2025
REFERENCES
Pierre Alquier. Pac-bayesian bounds for randomized empirical risk minimizers. Mathematical Methods of Statistics, 17:279-304, 2008.
Pierre Alquier, James Ridgway, and Nicolas Chopin. On the properties of variational approximations of gibbs posteriors. Journal of Machine Learning Research, 17(236):1-41, 2016.
Marein Andrychowicz, Misha Denil, Sergio Gomez, Matthew W Hoffman, David Pfau, Tom Schaul, Brendan Shillingford, and Nando De Freitas. Learning to learn by gradient descent by gradient descent. Advances in neural information processing systems, 29, 2016.
M. Bartlett. An inverse matrix adjustment arising in discriminant analysis. Ann. Math. Statistics, 22:107-111, 1951. ISSN 0003-4851. doi: 111121///om/111777759000/ org/10. 1tt./////////17722/00
URL https: //ooi.
Václav E Bene3, Exact finite-dimensional filters for certain diffusions with nonlinear drift. Stochastics, 5(1-2):65-92, 1981.
Olivier Binette. A note reverse Pinsker inequalities. IEEE Trans. Inform. Theory, 65(7):4094- 4096, 2019. ISSN 1118-9948,1557-966 doi: 21119//T URL https: //doi. 1tt.bb..0r/0..111/1111.
2019. 2896112.
Richard S Bucy and Peter D Joseph. Filtering for stochastic processes with applications guidance. volume 326. American Mathematical Soc., 2005.
California Department of Transportation. Caltrans Performance Measurement System (PeMS).
https: / pems. dot. dot.ca...... gov/. /, 2023. Accessed: 2023-10-01.
Cristian Challu, Kin Olivares, Boris Oreshkin, Federico Garza Ramirez, Max Mergenthaler Canseco, and Artur Dubrawski. Nhits: Neural hierarchical interpolation for time series forecasting. In Proceedings of the AAAI Conference on Artificial Inielligence, volume 37, pp. 6989-6997, 2023.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained code. arXiv preprint arXiv:2107.03374 2021.
Mohammed Nowaz Rabbani Chowdhury, Shuai Zhang, Meng Wang, Sijia Liu, and Pin-Yu Chen. Patch-level routing in mixture-of experts is provably sample-efficient for convolutional neural networks. In International Conference on Machine Learning, pp. 6074-6114, PMLR, 2023.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.1416 141668 2021.
Samuel N. Cohen and Robert Elliott. Stochastic calculus and applications. Probability and its Applications. Springer, Cham, second edition, 2015. ISBN 978-1-4939-2866-88 978-1- 4939-2867-5. doi: 11110077778---499991 978 1- 4939-9992220 2099-2867-5.
URL https: http:///oooooog//...
Damai Dai, Chengqi Deng, Chenggang Zhao, RX Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Y Wu, et al. Deepseekmoe: Towards ultimate expert specialization in mixture- of-experts language models. arXiv preprint arXiv:2401.0660 060666 2024.
E. B. Davies. Embeddable Markov matrices. Electron. J. Probab. 15mo. 47, 1474-1486, 2010. ISSN 1083-6489. doi: 10.1214/EIR.15-733.. URL https: /tt.///oii/o//////// wtt.oo..../////........ 12144/EJ1 733.
Pierre Del Moral. Mean field simulation for Monte Carlo integration, volume 126 of Monographs on Statistics and Applied Probability. CRC Press, Boca Raton, FL, 2013. ISBN 978 778-1-4655-0405-9
Petar M Djuric, Jayesh H Kotecha, Jianqui Zhang, Yufei Huang, Tadesse Ghirmai, Mónica F Bugallo, and Joaquin Miguez. Partiele filtering. IEEE signal processing magazine, 20(5): 2003.
12

---

Published as a conference paper at ICLR 2025
Dheeru Dua and Casey Graff. UCI machine learning repository. https://archive..tt uci. /tt::////daattttsssseae oat.:a/sseeeeetttttiooooaaaaaaaa
Nelson Dunford and Jacob T. Schwartz. Linear Operators. I. General Theory, volume Vol. 7 of Pure and Applied Mathematics. Interscience Publishers, Inc., New York; Interscience Publishers Ltd., London, 1958. With the assistance of W. G. Bade and R. G. Bartle.
William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(120):1-36 2022.
Eugene A Feinberg and Adam Shwartz. Handbook of Markov decision processes: methods and applications, volume 40. Springer Science & Business Media, 2012.
Trevor Gale, Deepak Narayanan, Cliff Young, and Matei Zaharia. Megablocks: Efficient sparse training with mixture-of-experts. Proceedings of Machine Learning and Systems, 5, 2023.
Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, Jason Phang, Laria Reynolds, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation. x. September 2021. doi: 10.528 1222oodoooo URL https: //oii. org/10. www1/221/22/ .ttp://22ooodd..122255. 5371628.
Google. Gemini. Google. URL https:/:: https:/:::///emt //ga:ii////iii//////// www..iii.oooo.o.oooooooo
Mohinder S Grewal and Angus P Andrews. Kalman filtering: Theory and Practice with MATLAB. John Wiley & Sons, 2014.
Nate Gruver, Marc Finzi, Shikai Qiu, and Andrew Wilson. Large language models are zero-shot time series forecasters, Advances in Neural Information Processing Systems, 36, 2024.
Daya Guo, Qihao Zhu, Dejian Yang. Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Yu Wu, YK Li, et al. Deepseek-coder: When the large language model meets programming-the rise of code intelligence. arXiv preprint arXiv:2401. 14 2024.
Lu Han, Han-Jia Ye. and De-Chuan Zhan, The capacity and robustness trade-off: Revisiting the channel independent strategy for multivariate time series forecasting, IEEE Transactions on Knowledge and Data Engineering, 2024.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Mcasuring massive multitask language understanding. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685 2021.
Martin Hutzenthaler and Arnulf Jentzen. Numerical upproximations of stochastic differrntiaal equa- fions with non-globally Lipschitz continuous coefficients, volume 236. American Mathematical Society, 2015.
Romain Ilbert, Ambroise Odonnat, Vasilii Feofanov, Aladin Virmaux, Giuseppe Paolo, Themis Palpanas, and levgen Redko. SAMformer: Unlocking the potential of transformers in time series forecasting with sharpness-aware minimization and channel-wise attention. In Ruslan Salakhutdinov. Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver. Jonathan Scarlett, and Felix Berkenkamp (eds.), Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 20924-20954. PMLR, 21-27 Jul 2024. URL https: pttpp/eiiceeng...coccpppo ml. wrw.s/pessseo..... / v25 ilbert24a html.
Robert A Jacobs, Michael I Jordan, Steven J Nowlan, and Geoffrey E Hinton. Adaptive mixtures of local experts. Neural computation, 3(1):79-87, 1991.
13

---

Published as a conference paper at ICLR 2025
Herbert Jaeger. Adaptive nonlinear system identification with echo state networks. Advances in neural information processing systems, 15, 2002.
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b, arXiv preprini arXiv:2310.06825, 2023.
Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arXiv preprint 2Xiv:2401.040885 2024.
Michael I Jordan and Robert A Jacobs. Hierarchical mixtures of experts and the em algorithm. Neural computation,
1994.
R. E. Kalman. A New Approach to Linear Filtering and Prediction Problems. Journal of Basic Engineering, 82(1):35-45, 03 1960. ISSN 0021-9223, doi: 10. 10.1115/1.3662552 URL https: //doi. www//0.g.......oo..o wtlgg//..11/////.........1.11
Patrick Kidger, James Foster, Xuechen Li, and Terry J Lyons. Neural sdes as infinite-dimensional gans. In International conference on machine learning, pp. 5453-5463. PMLR, 2021.
Youngjoo Kim, Hyochoong Bang, et al. Introduction to kalman filter and its applications. Introduction and Implementations of the Kalman Filter, 1:1-16, 2018.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
Pingzhi Li, Zhenyu Zhang, Prateek Yadav, Yi-Lin Sung, Yu Cheng, Mohit Bansal, and Tianlong Chen. Merge, then compress: Demystify efficient SMoe with hints from its routing policy, In The Twelfth International Conference on Learning Representations, 2024. URL https: // //opereview net. tiee/ffooe wtt///ooummmi/e wwwprummiddeFGyyyy33
Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, al. Holistic evaluation of language models. ArXiv preprint, abs/221 11//2211.00 109110, 2022.
Shengsheng Lin, Weiwei Lin, Wentai Wu, Haojun Chen, and Junjie Yang. SparseTTSF: Modeling long- term time series forecasting with * #Ik* parameters. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (eds.), Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 30211-30226. PMLR, 21-27 Jul 2024. URL https: http://pccceeen...e/ee.. mlr.prm wwrr.press//e v
2.3 i 224 ht.m1
Robert S. Liptser and Albert N. Shiryaev. Statistics of random processes. II, volume 6 of Applications of Mathematics (New York). Springer-Veriie Berlin, expanded edition, 2001a. ISBN -540-63928- 4. Applications, Translated from the 1974 Russian original by A. B. Aries, Stochastic Modelling and Applied Probability.
Robert S. Liptser and Albert N. Shiryaev. Statistics of random processes. volume of Applications of Mathematics (New York). Springer-Verlag. Berlin, expanded edition, 2001b. ISBN 3-540-63929-
2. General theory, Translated from the 1974 Russian original by A. B. Aries, Stochastic Modelling and Applied Probability.
Robert Shevilevich Liptser and Albert Nikolaevich Shiriaev. Statistics of random processes: theory, volume 394. Springer, 1977.
General
Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao, Chengqi Dengr, Chong Ruan, Damai Dai, Daya Guo, al. Deepseek-v2: strong, economical, and efficient mixture-of- experts language model. arXiv preprint 2ati::2420044444 2024a.
14

---

Published as a conference paper at ICLR 2025
Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXfv:2412.19437, 2024b.
Shizhan Liu, Hang Yu, Cong Liao, Jianguo Li, Weiyao Lin, Alex X Liu, and Schahram Dust- dar. Pyraformer: Low-complexity pyramidal attention for long-range time series modeling and forecasting. In International conference on learning representations, 2021.
Xuerong Mao. Stochastic differential equations and applications. Elsevier, 2007.
MosaicAI. Introducing dbrx: A new state-of-the-ard open Ilm. https:/ / www. databrickk cks.
coom/blog/nttrouucete dbb-- new state- aetee--rtt-pettte -1m 2024. Accessed: 2024- 05-21.
Yuqi Nie, Nam Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. time series is worth 64 words: Long-term forecasting with transformers. arXiv preprint arXiv:2211.147:5 2022.
OpenAl. Gpt-4 technical report, 2023.
Boris Oreshkin, Dmitri Carpov, Nicolas Chapados, and Yoshua Bengio. N-beats: Neural basis expansion analysis for interpretable time series forecasting, arXiv preprint 1905 10437,
2019.
Joan Puigcerver, Carlos Riquelme Ruiz, Basil Mustafa, and Neil Houlsby. From sparse to soft mixtures of experts. In The Twelfth International Conference Learning Representations, 2024. URL https: /tte:://pppeee /appnreeeeeieee net/ forumm? id= jxpsA
Nils Reimers and Iryna Gurevych. Sentence-hert: Sentence embeddings using siamese hert-networks, 2019.
Jonas Rothfuss, Vincent Fortuin, Martin Josifoski. and Andreas Krause. Pacoh: Bayes-optimal meta- learning with pac- mucc-uaranttes. In International Conference on Machine Learning, pp. 9116-9126. PMLR, 2021.
Jonas Rothfuss, Martin Josifoski, Vincent Fortuin, and Andreas Krause. Scalable pac-bayesian meta-learing via the pac-optimal hyper-posterior: From theory practice. Journal of Machine Learning Research, 24:1-62, 2023.
El Mehdi Saad, Nicolas Verzelen, and Alexandra Carpentier. Active ranking of experts based on their performances in many tasks. In International Conference Machine Learning, pp. 29490-29513, PMLR, 2023.
Claude Sammut and Geoffrey I. Webb (eds.). TF-IDF, pp. 986-987, Springer US, Boston, MA, 2010. ISBN 078-0-387--0014-8 doi: 10. 10110077-----8877 3-17---------------8-8888- 8_22664-882...11 832. URL https://dot
900000/99770-00 0077788-0387-311111 -xc// 30_-3-8881112... 832.
Raeid Saqur, Ken Kato, Nicholas Vinden, and Frank Rudzicz. Nifty financial news headlines dataset,
2024. Manuscript under review.
Keith P Sharp. Stochastic differential equations in finance. Applied mathematics and Computation, 37 1990.
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experti layer. arXiv preprint wwXiv:1701.065..0 2017.
Yueqi Song, Catherine Cui, Simran Khanuja, Pengfei Liu, Fahim Faisal, Alissa Ostapenko, Genta In- dra Winata, Alham Fikri Aji, Samuel Cahyawijaya, Yulia Tsvetkov, et al. GlobalBench: A benchmark for global progress in natural language processing. ArXiv preprint, abs/2305.145 14716, 2023.
Yejun Soun, Jaemin Yoo, Minyong Cho, Jihyeong Jeon, and U Kang. Accurate stock movement prediction with supervuue learning from sparse noisy tweets. In 2022 IEEE International Conference Big Data (Big Data), pp. 1691-1700. IEFF, 2022.
15

---

Published as a conference paper at ICLR 2025
Ruslan Leontyevich Stratonovich. Optimum nonlinear systems which bring about separation of signal with constant parameters from noise. Radiofizika, 2(6):892 190,,, 1959.
Ruslan Leontyevich Stratonovich. Application of the markov processes theory to optimal filtering. Radio Engineering and Electronic Physics, 5:1-19, 1960.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint wrXiv:2322139971 2023.
A. N. Sirjaev. Some explicit formulae problem on "disorder". Teor: Verojamost. Primenen, 10: 380-385, 1965. ISSN 0040-361x.
Renjie Wang, Cody Hyndman, and Anastasis Kratsios. The entropic measure transform. Canadian Journal of Statistics, (8(1): ):97-129, 2020.
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self-generated instructions, 2023.
Henry Wolkowicz and George P. H. Styan. Bounds for eigenvalues using traces. Linear Algebra Appl., 29:471-506, 1980. ISSN 0024-3795. doi: 00.00166002443755888000255---- URL htttst://ddii.
101100011111000-111111 (80) 90258-X.
W Murray Wonham. Some applications of filtering. Journal of the Society for Industrial and Applied Mathematics, Series Control, 2(3): 347-369, 1964.
stochastic differential equations to optimal nonlinear
Gerald Woo, Chenghao Liu, Akshat Kumar, Caiming Xiong, Silvio Savarese, and Doyen Sahoo. Unified training of universal time series forecasting transformers. Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (eds.), Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 1414400-1144 PMLR, 21-27 Jul 2024. URL https: ppopeedingggss
mlr. wul.rppesss/.. v235 / woo24a. html.
Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting. Advances in neural information processing systems, 34:22419 14:22419-222300,, 2021.
Huizhe Wu, Wei Zhang, Weiwei Shen, and Jun Wang. Hybrid deep sequential modeling for social text- driven stock prediction. In Proceedings the 27th ACM international conference on information and knowledge management, pp. 1627-1630, 2018.
Qianqian Xie, Weiguang Han, Xiao Zhang, Yanzhao Lai, Min Peng, Alejandro Lopez- and Jimin Huang. Pixiu: A large language model, instruction data and evaluation benchmark for finance. arXiv preprint arXiv:2306.0544 05443, 2023.
Yumo Xu and Shay B Cohen. Stock movement prediction from tweets and historical prices. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 1970-1979, 2018
Zhijian Xu, Ailing Zeng, and Qiang Xu. Fits: Modeling time series with 10k parameters. In The Twelfth International Conference on Learning Representations, 2024.
Moshe Zakai. On the optimal filtering of diffusion processes. Zeitschrift für Wahrscheinnichkeiissheee rie und verwandte Gebiete, 11(3):200-243, 1969.
Ailing Zeng, Muxi Chen, Lei Zhang, and Qiang Xu. Are transformers effective for time series forecasting? Proceedings of the AAAI conference on artificial intelligence, volume 37, pp. 11121-11128, 2023.
Haoyi Zhou, Shanghang Zhang. Jiegi Peng. Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pp. 11 11106-11115, 2021.
16

---

Published as a conference paper at ICLR 2025
Yanqi Zhou, Tao Lei, Hanxiao Liu, Nan Du, Yanping Huang, Vincent Zhao, Andrew M Dai, Quoe V Le, James Laudon, et al. Mixture-of-experts with expert choice routing, Advances in Neural Information Processing Systems, 35:7103-71144 2022.
17

---

Published as a conference paper at ICLR 2025
Appendices
Appendix Contents
The MoE-F Algorithm
19
Proofs
20
B.1 Proof of Theorem
21
B.1.1 Case I: Binary Cross Cooss-Entoopy Case
B.1.2 Case II: Squared Loss
23
B.2 Proof of Theorem
24
B.2.1
Completion of The Proof of Theorem 2
27
B.3 Proof of the Stability Guarantee in Proposition
28
C Datasets and Benchmarks
29
C.1 NIFTY Dataset
29
C.1.1 NIFTY-LM: SFT Fine-tuning Dataset
29
C.2 NIFTY-RL: Preferences Dataset
30
C,3 FLARE Benchmark Datasets
31
Additional Background Material
D.I Matrix Logarithms
31
D.2
The Shirayev-Wonhan Filter
E
Additional Discussions
33
E.1
F1-Measure in FMM Task
33
E.2
Time-Series Forecasting Experiments: Additional Details
33
E.2.1
Forecasting granularity in Line-Series Forcasting: IMS vs. DMS
33
E.2.2
Channel Independent Strategy
34
E.2.3
Our Setup
34
18

---

Published as a conference paper at ICLR 2025
A THE MOE-F ALGORITHM
This section records a detailed version of Algorithm 1, which can also be rolled-forward online as the target process Y. dynamically observed.
Algorithm 2: The MoE-F Algorithm
Input: A time-horizon 7 E N+, N (pre-trained) experts f(),..., ((N), hyperparameters À > 0, E (0, 1) and E N. target (Y2)==0 and input signal r(o:T-1]- Output: A posterior Mixture Weights Wi
/ Initialize
Initialize π = (T (mx))) N n,i=1 f (1/N)
Q + (1/(N - 1 - 1 11;=))
(I(")) n=1 + 0
for / = ..,
T do
For ,..., N in parallel
- A (m) (T()))) Yi)
B xxx) (Yi) (Y,) +
L(a) < { (((()))))))
AL L(n)_ I
10
AW + AL-A
B
(Tn) + L(n)
12
/o Update components of n² expert's posterior (x(*))
13
14
for I,..., do
At A1 (e,, Yi)
drift (())
diffusion t (mii) (A (A-A) A)/B
17
(m=i) + (maa + drift + diffusionAW end for
18
19
(n)) +- x(1)/ / E; क (mi)
20
Sn + ((Y,, ) F(z)0=))
21
(m) - (n)) T F(
22
// Calculate expert scores
23
end
f- Ae- /( 1EN1 e > x
24
Get Expert Scores
Y, del T (Y (n))
25
==1
// Get time I prediction
26
/- Update Q
*/
Q-Q
28
for n I.....N do
p(n) +
29
end for
P (1- P
31
Q ReL.U(log(P)) P)) - diag(1 ReI.U(log(P)))
32
33 end for
return Sequence of Mixture Predictions (-))
19

---

Published as a conference paper at ICLR 2025
B PROOFS
This section contains the proofs of our main result, generalizations thereof, and variants which apply the quadratic (squared) loss. the latter case, the necessary modifications the algorithm and the overall proof structure are relatively similar but with key technical differences.
Mild Generalizations and Further Discussion. We will consider the slightly more general case where the target process Y. follows the generalized dynamics
Yo [ w. Fxxx) ds+
o dW.
Best Expert Estimate
(Generalized) Idiosyneratic Residual
where, there are constants a, C with C' S such that: for each 2 0 one has or Ce By the Itô-isometry, see [Cohen & Elliott, 2015, Lemma 12.1.4), we have that the variance of So 0N dW, given by
c def (T. as dw,,
C (1 - (1----22)(0) (2ax)
if Ta>0 0 if
(8)
Observe that, if then the variance of Jo a, dW, asymptotically stabilizes at 1, as becomes arbitrarily large. In contrast, the variance of fu a, dW, diverges in the case where 0 (which is the case considered the main body of our paper).
Intuition behind the choice of assumed fluctuations/difftusinn. The intuition behind this modelling choice for the diffusion coefficient is based ideas behind concentration of measure. Consider the case where a > in equation (8). Since we will be considering classification applications, then we will not want the idiosyncratic residual Jo a, W, to push fluctuate outside the unit interval [o, 1), or rather the probability that any fluctuation of f o, W, "large" should be small. Since fo o, W, has a Gaussian distribution, then note, by standard Gaussian concentration inequalities, we have that
P (l% o. W. 1/2) < 01/2)/ = -0/0 w21)) -a/(Caa) aa-a
We can control the probability that any fluctuation is "large", meaning larger than 1/2, by setting the right-hand side of equation (9) to be a prespecified "small" value E (0, 1] and solving for the required a > 0 parameter in terms 5 yields the specification set C
In(1/8*). If d > 2, then we may
/ purely for convenience in simplifying expressions below.
In this case, for any hyperparameter 8 S the quantities in Theorem become
wh + f (Q.) T( ds f T (m:) (A,(e;, (o,a) A. (T. o
T (na)
(")
(10)
B. (Y,.))
where (Qt), denotes the th row of the transitions matrix Q at time 2 w/ P(ws ex) and where the "innovations process" w!!)) det (w(" )r20 is the following (P, JP,J..)-Broniil motion
dL (n) [0.m] Au
W ot
du,
0
Ba (Y
where the "stochastic differential" dr (n) [0.a] is given by
(I)
dI (n)
2((Y. p(m) (x.w) V (xx) w. F(x.)) e -2111(1665)
[0,N]
23/2
+ d (Ya - (((1))) (xu)) +ln(1/10)) dW..
=ln(1/04)
20

---

Published as a conference paper at ICLR 2025
B.1 PROOF OF THEOREM 1
We are now ready to state and prove two versions, one of which generalizes, our first main result (Theorem 1). We consider two cases. We obtain main result by customizing Theorem 1 to classification problem, where = 1 and the range of each expert is in {0, 1] C R, and setting o be specific constant (0, xo). For convenience, if we postulate that o 1; i.c is a constant function of the path Vl,,1) and of time t 2 0. Note that, by the Itô-isometry, see [Cohen & Elliott, 2015, Lemma 12. 1.4] the idiosyncratic residual term So as (Y(o))))) dW, equation (1) has centred normal random distribution with variance Jo 0% ds t.
B.1.1 CASE I: BINARY CROSS-ENS POOSS-ENTROPY CASE We now state and prove mild generalization of Theorem 1.
Theorem 3 (Optimal Optimistic Prior for n Expert Squared Loss Case). Consider the binary cross-entropy loss
( del log ( + (1-y) log(1 - 9)
and fix continuously differentiable path x. E (IR).
Under Assumptions 4.1, I, the best posteriori estimate of the th expert, (1), satisfies the following stochastic differential equation
it (mx) mo + f (Q:). ds {
Tta (ma) (A, (o,,s)) - (T((I)), , dw!! (n)
(11)
(Y,ol))))
where (Qt), denotes the 7th row of the transitions matrix Qt at time
2 0, म - P(wo ez), f(l) (I(
(Y - f(r) (zTo.))) A ( log
A(n) (w, M(oo)) det.
[w=. F(
(1 (1- ((m) (x))))))
((m) (i7p.)))
d
1 :)ooo) def Σ
Acle, (o,s)) (m=))
(x70,)))
(210.)) 2 log
eeln(ot)
(z)
F(F(O.))) dnd (ED)))
((N) (
Win (W: (n)
and the "innovations process"
1:2 is the following (P, ) motion
dl (n) 10.0
A.
W(n)
du.
(Y
where
(Y, -(- A ( + log 1 f(r) (x)
((w) ) [w] F(
dl (n) [0.u]
de,,,, (z(oo))
(1 (() ((a)
f(m) f(m
log
(ll(6")) dW.
Proof. Fix a continuously differentiable path x. E (R). Define Af( del. (x((0)))) and set L RR 3 yyn + (y (y-f(zoaa)) f(
21

---

Published as a conference paper at ICLR 2025
First, observe that: for all 0, all I. C' C²(R) and all R one has
(v)) (y log( of ((m) (ITo.s))) + (1-11 y) log(1 (x(o.)))
ol.
d1 (v) (y (x A (zT00))))
f(r (z(00)) (1- - (2
oli (v) log
(((())))))
(12)
dy
1- (z)(e))
0l (y) O. 0y²
Since & C (IR4 x RR), , we assumed that the path æ. E (10, oo), R2) then this, together with the postulated dynamics on y:!n) imply that Ito's Lemma/Formula, see [Cohen & Elliott, 2015, Theorem pre-composed with Y. applies. Whence, our and the assumed dynamics on Y., postulated in equation (1), imply that the
14.2.4), used on the map e!") oo) TR. 3 (1, - (u - fxx)
process (14) def. p!t)) (Ye satisfies the following stochastic differential equation
fd ae!")
L (n)
(Ya)
as
0e!)) (n)
(Y.) [m! F(
Oy
1 k2
(Y,) 2c ln(6"))
ds
2 Oy²
+ / )(())
(Ys)e² =nn00 dW,
dy
/ (-1) - (2)p,))) A (xpon))
(13)
((-(()) f(r) ((10)1)))))
(()00)))
(01)
(w] F(
0 ds
(x(o,)) 1 (()
+ (-1) log
e In((6*) dW.
(14)
(Y, ((moee)))
A (
(w)00))
) [w] F(xp.,))) ds
+
J (1 (m(oo))) (in)
log
1 1 (()
f(m) (ITo.))
- log
e slu(6")) dW.
Synchronizing our notation with [Liptser & Shiryaev, 2001a, Equation (9.1)), we write
(Y, - (( A (
((n) [w.] F(
Aw V(..)) det
log
(1 ((po.))) ((
1 1- f(
p( (2
Be(y)o,a) det loog
1 (())oo))
(15)
Under Assumptions 4.1, we may apply [Liptser Shiryaev, 2001a, Theorem 9.1] to deduce that Then the posteriori probability x (n:i) dal (me)i. satisfies a system of equations
(n:i) =/ (+((((1)) (Q.). (m) ds /" n (m:1) (As Moo,a) (A( Yu,si))
dW
(16)
B.(Yo.))
22

---

Published as a conference paper at ICLR 2025
where (Qt), denotes the tthh row of the Qu/transitions matrix Qi at time t O, mb del P(we e4). and
the "innovations process" is the F.F.. Brownian motion given by
dL (n)
(m))) (o..))
W (n)
del.
10.
du
(17)
B., (1,)oo))))
and where
d A.(e;, (o.))) x(ni)).
(z y(0.d))
This completes the proof.
Remark Setting d in the previous derivation yields the formulation of Theorem found in the main body of the paper.
B.1.2 CASE SQUARED LOSS
Theorem (Optimal Optimistic Prior for th Expert Squared Loss Case). Let E(g. v) del. and fix a continuously differentiable path x. E C (TR).
8)
Under Assumptions 4.1, the best posteriori estimate the n expert, T("), satisfies the following stochastic differential equation
(mx)) Tt =wb + [f. (Q:).. (((())T ds fo"
TT' (nc)) (Aales, Yo,a)) As( (ooo))) dW u B.(Y (0.9)
(18)
where (Qt); denotes the tth now of the transitions matrix Qe at time 1 2 0, w/ del PP = ez).
A0 (w, (10.4)) 2(yr p(m (x70.1)) (w, F(m00.)) ( + elu(6"))
A (m),,, 3)0.0)) Aclei, Yoot)) (mn))
B!"h (3)0.1 del 23/2 (Ht f(m) ()e
F( del (F() ..... f(N) (2
W (W, (n)
and the "innovations process"
120 is the following ) motion
dL (n)
Ao(Yoo))))
( dt
00.l
du,
Bu (Yoou))
where
dL [.0)
de(Y,, f f(0) (x))0) =2(Y; ((m) f(m) (2)o.1))) (ml) F( (( (x + ll(66))) + f(()) (z lu(6") dW..
Proof. Fix a continuously differentiable path x. E (R). Define (f( ddt (x(00)))) and set 8: By (y- f(
First, observe that: for all 2 0, all r. E C² (R) and all y e R one has
n (y) (y f (TTo)))))
ol, (v) =2(y f(v) (m)oo)) ( A (m)) (mp,0)))
Øt
Oe, (v) 2(y f(1) (z)).))),
(19)
Dy
844 (g) =2 0y²
23

---

Published as a conference paper at ICLR 2025
Since & C (IR4 x ). assumed that the path C' ((0, Re), then this, together with the postulated dynamics on (!!") imply that Ito's Lemma/Formula, see [Cohen Elliott, 2015, Theorem 14.2.4], used on the map e(m)) [0, oo) x TR. e (t,y) (y ((r) (zx E R pre-composed with Y. applies. Whence, the computations in equation (19) and the assumed dynamies on Y., postulated
in equation (1), imply that the process L," (m) det. ())) (Yi) satisfies the following stochastic differential equation
delm)
)p())
dy (Ys) [m! F(r + 1 pP Dy² (Y,) 20=ln(6")
(n)
(m)
(Y,)
Ös
de(m)
(Y,) Va)e=ln(6") dW.
+
0 dy
(m)
2(Y, (z (lw - Afm)) (ITo.s) + c (20)
% 2(Y, flr Ve dW.
Synchronizing our notation with [Liptser & Shiryaev, 2001a, Equation (9.1)), we write
(w, V(o.e)) det 2(v - f( (x10.)) (w, F(Mo))) - (x(a))))))) + e*lu(61)) (v(o,))) t 23/2 (y) 1 (m) (e
(21)
Under Assumptions 4.1, may apply [Liptser Shiryaev, 2001a, Theorem 9.1] to deduce that Then the a posteriori probability nn:0 (mt)o. satisfies a system equations
Tt (n:t) wó + S.' (QUT a(
ds ((m
(ms)) (As(es, (ooa)) As(m(()) (o,a))) dW (Y(().)))
(22)
where (Qt); denotes the jth row of the Quttrnnitions matrix Qr at time / 2 0, wa del and the "innovations process" is the (IP, F.) -Brownian motion given by
P(wo = es).
dl (x( Yoo..) [0.u)
dI (6)
det [o
du
(23)
Bu(Yoouu))
and where
(x( (1004) ,10.4) def.
As(ei, (o.i) (m=))
This completes the proof.
Remark 2. Setting 8 = in the previous derivation yields the formulation of Theorem found in the main body of the paper.
B.2 PROOF OF THEOREM 2
The proof of Theorem 2 relies the following result. Briefly, this result guarantees for the validity of the perturbation to the transition probability defined by
pa pa (1 a: aIN
(24)
for arbitrary N E N+, P. PE (0. 1), and where IN is the N N identity matrix.
In what follows, will use An del (w E [0, 11 >EA wa} denote the probability N-simplex; which corresponds to the probability (measures) distributions supported on N points. Here, these N points are the experts themselves, and the probability of selecting any expert interpreted as the relative credibility we ascribe to its historical predictive power.
24

---

Published as a conference paper at ICLR 2025
Proposition 2 (Regularity of Perturbations). Let N N+, A O. $1 ... &N R, T An be given by
π del. Softmin (A and P. [(T)
For every a E 1), the matrix P. in equation (24), is invertible and (row) stochastic. If, moreover, all its real eigenvalues are non-negative, then (og(P well-defined and its rows sum O. In particular, setting a 2 11-11/1 1/N guarantees that log (P)) exists, if Pe has real eigenvalues.
We will now show our second main result, and the intermediate lemmata leading up to it. The next lemma states that if (row) stochastic matrix is constructed by filling each of its rows with an element of the probability simplex, then shining it by an arbitrarily small abound and growing its diagonal proportionally yields a (row) stochastic matrix, which is necessarily invertible.
Lemma 1 (Invertible Perturbations). Let N E N4 let т E AN. If P is (row) stochastic matrix, then. for any (0, 1). the matrix (1 P + alN an invertible (row) stochastic matrix.
Proof of Lemma 1. Let 1N RN be such that: for each i I,....N N we have (1x); (i.e. 1N a matrix of ones). By construction P (T....,..) T.. Therefore, P can be written as an outer product via
P= 1N π Therefore, for any E (0, 1), the perturbed matrix (1 - a)P + A/N can be expressed as
(25)
(1-a) ((1 aa) 11) 1x)=,,
(26)
Le. (1 11-al a.) P can be expressed as outer product of vectors in RV; namely of ((1- a)) 1N) and Consequentially, our matrix of interest can be written as
a) P AIN ((1 a)) I aIN- a1
(27)
Note that, a IN is invertible since det(aIN) = a 0. Thus, the main result of Bartlett [1951] can be applied, which yields the condition: if aIN is invertible (which it is) and if
(aIN) #0
(28)
then alN + ((1 a) 1 invertible. Thus, we only need to verify that the condition holds in our case. Simplifying equation (28) yields
1 + (a
(29)
1 N
(30)
a
N
1
Σ
Tn
(31)
a n=1
1 1 a
(32)
where equation (32) held since E AN. Consequentially, the identity in equation (27) and the computation equation (29)-equation (32) imply that (1 1- a)) P + T invertible if a # -1
Finally, since P is (row) stochastic and so is IN then, for each i = 1, ..... N, we have that
N Σ (1 = -1-a) Σ j=1
P.j + al = (1- +
f=1
Whence, (1 P + aIN is (row) stochastic also.
We now provide set of sufficient condition on guaranteeing that the principal logarithm of P. is well-defined. Furthermore, this lemme also shows that for a E (0, 1) large enough, as function of N. the matrix log( P(t) is necessarily a valid candidate for Markov transition matrix (i.e. each of its rows sum to 0). Lemma 2 (Sufficient Condition for Existence). Ifa 2 NZ then if either of the following holds:
25

---

Published as a conference paper at ICLR 2025
(i) Real Case: Pt has complex eigenvalues,
(ii) Complex Case: P is doubly stochastic (i.e. row and column stochastic),
then (Pg( exists and its rows sum to O.
Proof of Lemma 2. Since Pe is an N X N real (thus complex) matrix with real cigenvalues, then define
m tr(P and =
del. tr((P))))NN - m².
(33)
First, observe that since the entries of P. (in particular its diagonal elements) are all positive then
m tr(P))/N
//N
(34)
1
(11 (1-a) a) t. a))
(35)
N
N
(1 (L-o) a) Σ n; + a
Σ
(36)
N
(1-a)+
(37)
(1 11aa(N 1)).
(38)
Next, we compute 5². By Lemma I, we have that Pe is stochastic matrix and, therefore, so is its square (as the product of stochastic matrices stochastic). Note that
tr (T(())))) S
tr(s) tr(IN) = SEStoch(N)
max
(39)
where Stoch(N) is the set of N N stochastic matrices. Therefore, we bound s², defined in equa- tion (33) using the "extremal trace bound" in equation (39) via
s² tr((P))/N m²
(40)
S N/N - m
(41)
- 1 N2 (1 + -NN-a((N)))) -
(42)
That is
8 2 - (1
(1 a(N - 1)) 1))3
N2
Now, using lower-bound on the minimal eigenvalue of square complex matrix with real eigenvalues using m and s in [Wolkowicz& & Styan, 1980, Theorem 2.1] we have that
Amin (PP) 2m s(N 1)2/2
(43)
1
+ a(N - -1)) - (1 (1- (1 a(N (ON(NN1))) -1))3) 1/2 N2
1
(N - 1)
(44)
N
1
(I [+a(NN - -1) - (N22 - -NN2=(1+ (1+a(N-1))* 1/2 1/2 (N- 1)2/2
(45)
N
(L+ - 11)
(46)
N
(a - 1)3) 1/2
+2 - 1)N () -1) (alaa)
(47)
If 2 1- No and > (which is always the case) then
(1 (1+a(N-1)) 11) (N 1) (a (1-a2) + 2a(a 1)N - (a - -1)]]] 11/2)
> 0.
(48)
26

---

Published as a conference paper at ICLR 2025
Therefore, equation (48) together with equation (46) imply that
min (?))
whenever 2 N2.
Therefore, [Dunford Schwartz, 1958, Theorem VII.1.10] implies that log(P()) exists, since Pf is matrix whose spectrum does not contain (-o0, 0). Moreover, [Davies, 2010, Lemma 1] guarantees that the rows of (og(P:) sum
Finally, we note that since IN is doubly stochastic then so is Pf" provided that P is. Therefore, Pf is also stochastic matrix and so is the product Pt pre (as the product of (row) stochastic matrices again a (row) stochastic matrix). Whence
del (PP m² S
N² (1+ 11+a(N-11 1)))
(49)
and the same argument may be applied with s² in place of s² upon using [Wolkowicz & Styan, 1980, Theorem 3.1] in place of [Wolkowicz & Styan, 1980, Theorem 2.1]: however, in this case we do not need assume that the eigenvalues of P. real. In either case, this concludes our proof.
B.2.1 COMPLETION OF THE PROOF OF THEOREM 2
Proof of Theorem 2. Step 1 Minimizer of Inner Problem equation (Inner):
Since the elements the set of x uniform stochastic matrices P/ all have identical rows then, P is optimizer of equation (Inner) if and only if its first row is minimizer of
N
min PEPE
Σ Pi E()! (1) Y.) + ,Y)
1
Σ
Pi.m log(w,/N).
(50)
Since the matrices in Pk are row-stochastic, then all their rows belong to the N simplex AN- Therefore, P minimizer of equation (50) if and only if its first row, which we denote by π
E AN is minimizer of
N
N Σ 1=1
min TEAN
Σ
TH E(Y
Yx)
Σ
nn log(w,,/N)
(51)
n=
Since the elements of AN in bijection with the set of probability measures the N N-point set (I...., NJ. À O, and > Ta log(w,/N) is the KL-divergence (relative entropy) between the probability measure En nn and the uniform measure Σ 1 then [Wang al., 2020, Proposition 1] the unique minimizer of equation (51) is given by
N dn (both on [1....,N] N))
Softmin (> (((Y(a), Y))
Consequentially, the matrix P PV whose rows are is minimizer of equation (Inner).
Step 2 - Minimizer of Outer Problem equation (Outer):
By Proposition 2, for every a00, 1) the matrix
pa def. (1-a)P+aly
1). 1), the matrix Po has is row-stochastic and for "large enough"; meaning for E (1-1/N,1 all its eigenvalues in (0, oo). Therefore, by [Davies, 2010, Theorem 12] there is a minimizer
of equation (Outer) and it is given in closed-form by
Q def. ReLU (log(P!)).
This concludes our proof.
Proof of Proposition 2. By Lemma 1, the matrix Pa is (row) stochastic and invertible. Thus, it has no eirreeiiens..ues... If, moreover, the assumption holds that P. has no negative eigenvalues then [Dunford & Schwartz, 1958, Theorem VII.1.10] guarantees that the (principle branch) of the matrix logarithm of P exists. Consequentially.. [Davies, 2010, Lemma 1] applies from which we deduce that the rows of (og) sum O. The last claim follows directly from Lemma (i). n
P.))
27

---

Published as a conference paper at ICLR 2025
B.3 PROOF OF THE STABILITY GUARANTEE IN PROPOSITION 1
The following generalizes, thus implies, Proposition 1. Lemma 3 (Maximal KL Divergence for Perturbation Lemma 1). Let x E AN, a [0, 1).
i ,, and let for each i 1, ,,...,,VV. .N let ga. (1 a) + ars where {e is the standard basis of RRN If Pmin min,=1,.... Pt > 0 then log(Pmin)
max (T,(T)==))) S 20 (- 1/pmminn A= 1
log((1 (x) Pinin) 1/((1 ax) Punin)
Proof of Lemma 3. By the (sharp) reverse Pinsker inequality in [Binette, 2019, Theorem 1], as formulated in [Binette, 2019, Example A], yields the bound
(log(1/min))
log(1/Pmin)
(T(()==*)) (1
(52)
1/Pmin
1/Pmin -
where TV is the total variation distance between ι and xnd, and Pmin m N no. construction
By
Pm)i S ort S (1 ar Pmin + a.
Whence,
and 1/p 1/((1 a) Pm) + +a)= -1
(53)
(1 ampmm
Priin
Incorporating equation (53) into equation (52) yields
< TV(T, nntt) (- (Pggn)
log((1 (1 - ax) Pimiu)
1/Pmin -1 1/((1 - a) Pmnin)
[T 7 (-
log(Pmin)
log((1 a)) Prain)
1/Pmin
1/((1 a) Pinin)
log(Pmin)
log((1 - ai) Pmii)
|ort + adj=iil
1/Prnin -1 1/((1 ()) Pmmin)
Σ 1=1
J + (x)
(-
log(Pmin)
log((1 a)) Punin)
1/Pmin
1/((1 - -
log(pmin)
log((1 a)) Pmiin)
|T3 + +a)
1/Pmin
1/((1 a.) Puiin)
j=1
N
log(pmin)
log((1 ax) Pminn)
Ty. + ax)
(54)
1/Prain
1/((1 ax) Peniu)
j=1
(- log(pmin)
log((1 a)) Pmiin)
<aa + +a)
(55)
1/Pmin -1 1/((1 - a) Pmii)
log(Pmin)
log((1 a)) Pmii)
-2a
1/Pmin -I 1/((1 - (x) pmin)
where equation (54) held since AN and therefore. tion (55) held since >
0 for each i I,..,,N, and equa-
Di=1 म; 1 again due to the fact that π E AN.
28

---

Published as a conference paper at ICLR 2025
C DATASETS AND BENCHMARKS
C.I NIFTY DATASET
The Informee Financial Trend Vield (NIFTY) dataset Saqur al. [2024] is processed and curated daily news headlines dataset for the stock (US Equities) market price movement prediction task. NIFTY is comprised of two related datasets, NIFTY-LM and NIFTY-RL. In this section outline the composition of the two datasets, and comment on additional details.
Dataset statistics.
Table 6 and Table 7 present pertinent statistics related the dataset.
Table Statistics and breakdown splits sizes
Table Date Ranges news headlines splits
Category
Statistics
Split Num. Samples
Date range
Number of data points
2111
Train
1477
2010-01-06 to 2017-06-27
Number of Rise/Fall/Neutral label
558 /48/ 1122
Valid
317
2017-06-28 to 2019-02-12
TrainTest/Evaluation split
1477 / 317 / 317
Test
317
2019-02-13 to 2020-09-21
Anticipate the direction of the $SPY by analyzing market data and news from 2020-02-06.
(a) Instruction component of a TLM policy query Tेq.
date, open, high, ... pot_change, macd, boll_ub, boll_lb, rsi_0,, ... close_60_smo
2020-01 27, 323.03, 325.12, ... -0.016, 2.89, 333.77, 319.15, 56.26, ... 317.40 3 2020-01-28, 325.06, 327.85, ..., 0.0105, 2.59, 333.77, 319.55, 59.57, 317.78
2020-02-04, 328.07 330.01, 0.0152, 1.3341, 333.66 60, 321.26, .** 319.41
2020.02:05, 332.27, 333.09, ., 0.0115, 1.7247, 334.15, 321.73, .,*, 319.82
(b) The market's 's history is provided the past days of numerical statistics like the (OHLCV) price (in blue) and common technical indicators (in orange) (e.g. moving averages) data.
Figure 6: BBreaking down the instruction or prompt prefix, and market context components of prompt, Zy.
C.1.1 NIFTY-LM: SET FINE-TUNING DATASET
The NIFTY-LM prompt dataset was created finetune and evaluate LLMs on predicting future stock movement given previous market data and news headlines. The dataset was assembled by aggregating information from three distinct sources from January 6, 2010, to September 21, 2020. The compilation includes headlines from The Wall Street Journal and Reuters News, as well as market data of the SSPY index from Yahoo Finance, The NIFTY-LM dataset consists of:
Meta data: Dates and data ID.
Prompt (xp)): LLM question (Equestion); market data from previous days (Icomtezt), and news headlines (Enewos))
Response: Qualitative movement label (I,)) E { (Räse, Fall, Neutral), and percentage change of the closing price of the SSPY index.
To generate LLM questions, (Iquestion), the authors used the self-instruct Wang et al. [2023]
framework and OpenAl GPT4 to create 20 synthetic variations the instruction below:
Create 20 variations of the instruction below.
Examine the given market information and news headlines data on DATE to forecast whether the SSPY index will rise, fall, remain unchanged. If you think the movement will be less than 0.5%, then return 'Neutral'. Respond with Rise, Fall, or Neutral and your reasoning in new paragraph.
29

---

Published as a conference paper at ICLR 2025
Where DATE would be substituted later, during the training phase with a corresponding date.
Context. The key 'context' (oonnteatt was constructed to have newline delimited market metrics over the past T (Rs 10) days (N.B. Not all market data for the past days for were available and therefore prompts might have less than 10 days of market metrics.).
Table 8 show the details of financial context provided in each day's sample.
Table Summary of dataset columns with their respective descriptions.
Column Name
Description
Date
of trading session
Opening Price
Stock's opening market price
Daily High
Highest trading price of the day
Daily Low
Lowest trading price of the day
Closing Price
Stock's closing market price
Adjusted Closing Price
Closing price adjusted for splits and dividends
Volume
Total shares traded during the day
Percentage Change
Day-over-day percentage change in closing price Momentum indicator showing the relationship between moving averages Upper boundary of the Hollinger Bands, set at two standard deviations above the average Lower boundary, set at two standard deviations below the average Momentum oscillator measuring speed and change price movements Indicator identifying cyclical trends 30 days Indicates the strength of price trends over 30 days Average closing price over the past 30 days Average closing price over past days
MACD
Bollinger Upper Band
Bollinger Lower Band
30-Day RSI
30-Day CCI
30-Day DX
30-Day SMA
60-Day SMA
News Headlines. (Xnews): Final list of filtered headlines from the aggregation pipeline. The non-finance related headlines were filtered out by performing a similarity search with SBERT model, MIl-MiniLMM-L6-1 Reimers & Gurevych [2019]. Each headline was compared to set of artificially generated financial headlines generated by GPT-4, with the prompt "Generate 20 financial news headlines". Headlines with a similarity score below 0.2, were excluded from the dataset. To respect the prompting 'context length' LLMs, instances where the prompt exceeded a length of 3000 words, further refinement process was employed. This process involved the elimination of words with a if-idf Sammut & Webb [2010] score below 0.2 and truncating the prompt to a maximum of 3000 words.
is also important to note that the dataset does not encompass all calendar dates within the specified time range. This limitation emanates from the trading calendar days, and absence of relevant financial news headlines for certain dates.
Label. (xr)): The label determined by the percentage change in closing prices from one day to the next, as defined in equation 56. This percentage change is categorized into three labels: (Rise, Fall, Neutral), based the thresholds specified in equation 57.
Closing Price, - Closing Price,-1 ;-1
PCT change
100%
(56)
Closing Price,-1
Fall
if PCTehange < -0.5%
Er Neutral
0.5% S PCTeeange I 0.5%
(57)
Rise
if PCToaangs > 0.5%
C.2 NIFTY-RL: PREFERENCES DATASET
The preference dataset a variation of the fine-tuning dataset and is designed for alignment training of L.LMs using reward model. In NIFTY-RI, labels omitted and replaced with chosen and rejected
30

---

Published as a conference paper at ICLR 2025
results. The chosen result is label corresponding rise, a fall or neutral movement in the stock market and is equivalent to the response in NIFTY-LM. The rejected result a random label not equal the chosen label.
Metadata: Includes dates and data identifiers.
Prompt (zp): Includes LLM instruction (Iquestion), preceding market data (Fcontezt), and relevant news headlines (Inews).
Chosen Result: qualitative movement label (x+) from (Rise, Fall, Neutral) indicating the predicted market trend.
Rejected Result: A label (T,)) randomly selected from {Rise, Fall, Neutral, Surrrederrrrr Mær}. representing an incorrect market prediction.
C.3 FLARE BENCHMARK DATASETS
Datasets and Tasks: Flare-SM tasks. FLARE proposed by Xie et al. [2023], extends to include one financial prediction task - the CIKM dataset Wu et al. [2018] as an evaluation task among (four) other general financial NL.P tasks. Under the hood, this benchmark is fork of the 'Im-eval" harness Gao et al. [2021] with addendums. Other stock price movement prediction from social dataset include what referred to as ACL18 (or, 'acl18') in this paper is essentially the StockNet Xu & Cohen [2018] dataset which comprises of stock tweets of 88 stock tickers from financial market industries from Twitter over two years (from 2014-2015) aligned with their corresponding historical price data. BigData22 [Soun et al., 2022] is another more recent tweets dataset comprising of tweets about 50 stock tickers during the period 2019-07-05 to 2020-06-30.
Stock Movement Prediction
Table Summary of Flare stock price movement datasets. The 'Stocks' column indicates the total numher of different stock tickers referenced. The 'Tweets' and 'Days' columns represent number of tweets and days respectively in each dataset.
Data
Stocks
Tweets
Days
Date
End Date
ACL18
106,271
696
2014-01-02 -02
2015-12-33
BigData22
50
272,762
362
2019-07-05
2020-06-30
CIKM18
38
955,788
352
2017-01-03
2017-12-28
D ADDITIONAL BACKGROUND MATERIAL
an effort to keep our paper as self-contained as possible, this section contains additional background material used in our derivations and in the formulations of our technical results.
D.1 MATRIX LOGARITHMS
The (principal) logarithm of an N x N matrix A whose spectrum does not contain (-oo, 0J in Cs defined by
2Ti / log(z) (zIN A)
1
log( A)
where log(2) is the principal logarithm of z (in the complex plane) and r is closed curve in (-oc, containing the eigenspectrum of A.
D.2 THE SHIRAYEY- WONHAM FILTER
To keep our paper as self-contained as possible, included some brief background on stochastic filtering. Namely, this appendix contains background material on the Shirayev- Wonham (stochastic) filter, studied in [Liptser & Shiryaev, 2001b, Chapter 9].
Shirayev-Wonham
Consider complete probability space (2, P) equipped with a non-decreasing sequence of
right-continuous sub-o-algebras J,, S L S Let 0 (0,, Fr), S < T. denote real
right-continuous Markov process taking values in the countable set E = (a. B 7 ....). Additionally,
31

---

Published as a conference paper at ICLR 2025
let W = (W,,FF). 0 T, be standard Wiener process independent of 9. and let Eo be a Fo measuururebee random variable independent of 0. We assume the existence of nonanticipative functionals A,(c, zz) and (x) that define
de. A.(0,, Be(E)dW,
(58)
and satisfy the following conditions.
Af(e,, S1 //¹ + (1+ L2(1 -01
(59)
0CCS (x) 1 L ( [((1 (1+ (1+x2)dK(()) Iz211 + x2),
(60)
( - Ac(e, y)12 [B((z) Be(y)1² S L1 S (z. (2. L2( y1)²
(61)
where C, L,,,o La are certain constants, K(s) is non-decreasing right continuous function, s (s)) I 1, Cr,yE E Cr, E E,0 S s T.
Along with Equations (59) to (61) it will also be assumed that
Ma 00,
(62)
and
T M
J /' 8 < 00.
(63)
Define
(a(() def P(O, = 3),
Psa (1, def. P(O, = 310,- = a), 0 058<15 8, E,
and assume there exist a function (tn((), S t T, a, 8 E, that continuous overt (uniformly over a, 3)
(64)
(tas(t) - K
(65)
[paa (t + A,t) - ((3 aa (t) A S o(A),
(66)
where 8(8,a) is a Kronecker's symbol and the value ((A)/A A + as A + 0 (uniformly over a, B).
Let the Equations (59) (66) be fulfilled. Then the a posteriori probability TT))), E, satisfies system of equations
(1( = pp((0) / L'mp(u)du ( TT(()) Au - As(S)) dW m
Ba(E)
where
L L2 = Σ (1s(())T(()) and Au(g) = Σ Au(7,8)m,,
YEE
7EE
and W = Fi) a Wiener process with
W. ( dg, Au(E) du.
B.(E)
32

---

Published as a conference paper at ICLR 2025
E ADDITIONAL DISCUSSIONS
We have updated the amonymous git repo with all the experiment results and added script to easily replicate the main results of the paper in Table 2.
E.1 FI-MEASURE IN FMM TASK
For evaluation of model performances on the financial market movement (FMM) task using ternary class labels, we use the scikit-lcarn library methods. For multi-class, n., labels (with n 2), the choice of averaging is important. The tabulated results were evaluated with default averaging set "weighted" where metrics for each label were calculated, then average weighted by their corresponding support (the number of true instances for each label). This alters the global averaging 'macro' (equal weight) for label imbalance. Imbalanced label support can result an F-score that lower or not in between precision and recall.
Llama-2-7o-dhat-tt
Llama-2-0b--caa-ttff
Meta-Llama-3-6
41
Noie
Netter
(a) Llama-clas models.
Mixtral-xx7B-oi 9x1t-lnstruct-0..1
doneiinttuuct
OpenAl oot-lpt----o
Predicted
Predicted
(b) Mixture-of-experts class models and the state-of-the-art GPT-4 model.
Figure 7: Confusion matrices for Table 2. The first row highlights the Llama-class models, and the second row Tocuses mixture-of-experts and GPT-4 models.
E.2 TIME-SERIES FORECASTING EXPERIMENTS: ADDITIONAL DETAILS
This section provides additional discussion in support of the long-horizon time-series forecasting (LTSF) experiment covered in $5.2.
E.2.
FORECASTING GRANULARITY TIME-SERIES FORCASTING: IMS vs. DMS
In LTSF works, the decoding granularity is dichotomized in the following two categories:
Iterated/Incrementa Multi-step (LMS). This is auto-regressive language-modeling or gen- erative style prediction decoding where each step in prediction horizon H is iteratively predicted: fr-1, and is used for the prediction for the subsequent time.tep.. The common limitations of such forecasting granularity is 'error accumulation' over time the decoder builds on the error from previous steps while iteratively making subsequent predictions. Additionally, the run-time complexity is to the order of the length of the horizon H.
II) Direct Multi-step (DMS). As the name implies, in this decoding or forecasting approach, the entire forecasting horizon H is predicted one go: #1+14 While computationally more attractive
33

---

Published as a conference paper at ICLR 2025
(much faster than IMS), this approach may not incorporate seasonality/erriddictty in the time-series. Modern TSF specialist models, especially the transformer- baseer---eedd TSF architectures, tend to follow this scheme to avoid the quadratie cost (to the length of input) associated with attention.
E.2.2 CHANNEL INDEPENDENT STRATEGY
Recent advancements in Long-Term Series Forecasting (LTSF) have increasingly embraced Channel Independent (CI) approach for handling multivariate time series data Han et al. [2024]. The CI strategy simplifies forecasting by isolating each (channel or feature as) univariate time series within the dataset, allowing the model to focus on predicting individual channels independently. Unlike traditional methods that leverage the entire multivariate historical data to make forecasts, the CI 1+ E IRL - {(()) 1: E RR for each univariate series, providing streamlined model for each channel and reducing the need to account for inter-channel dependencies.
approach seeks a shared function / (1)
E.2.3
OUR SETUP
For our experiments, the historical observation window (aka. look-back window or lag period), L, is kept constant at 720 stess to be consistent and comparable with the literature. We follow the Shannel-independen strategy similar to the three expert models used for filtering giving us C number of distinct features channels of agent's observation. The (MSE) loss is then measured as the discrepancy between the predicted values (.(1)
(6) 1411+ and the ground truth 11+1:1+ as
1
( 3l
x(0
c
1it4
2411+ 2'
(67)
34


Reviews:
[{"review_id": "VMjXzjWU1c", "reviewer": "Reviewer_8aRL", "summary": "This paper presents an online algorithm, called MoE-F, to adapt a mixture of pretrained large language models (LLMs) for time-series prediction tasks. A filter is used for each LLM, and the learning algorithm will learn those filters from data. The authors then provide an analysis about their method to reveal some nice properties of the solution. Finally, an extensive experiment has been done to evaluate their proposed method, using many big pretrained models, e.g., Llama-2, Llama-3, Mixtral, DBRX, ... Two types of problems are used in their evaluation, classification of financial market movement, and time-series forcasting. The results suggest that the new method can perform significantly better than the baselines for the first classification task. For time-series forcasting, their method also exhibits competitive performance.", "questions": "Can the authors explain about the concern before? Does it only a writing mistake?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "**Originality:**\nThe proposed method seems novel and work well with time-series forcasting in an online fashion. \n\n**Quality:** \nThe performance of their proposed method seems good, compatitive with the recent baselines. Their method is supported by some theoretical analysis.\n\n**Clarity:**\nThe writing is readable with soem efforts for new readers.\n\n**Significance:**\nPlease refer to other reviewers for this aspect, as I am not familiar with the topic of this paper.", "weaknesses": "The experimental results pose a concern about their way of evaluation or measurement. This concern comes from the strange behavior of F1 score in Table 2 and Table 3. In some columns (e.g., most columns in table 3), the value of F1 is smaller than both Precision and Recall. This is really strange. In my understanding, F1 is a hamonic mean of Precision and Recall, and hence it can not be less than both Precision and Recall.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "m8SW43no48", "reviewer": "Reviewer_qJNc", "summary": "The paper proposes a filtering algorithm to combine predictions from several experts in sequential decision making problems. At each time-step each of the experts provide their prediction and their reliability based on their local estimates. Then based on the historical losses of the expert the algorithm comes up with mixture weights to combine the experts at time t and the process continues. The paper provides optimality guarantees for their method under a hidden markov model process environment. Then they show two applications: (i) combining LLM experts for the task of market direction prediction based on evolving text market data (ii) combining time-series experts for forecasting. They show remarkable gains in both applications.", "questions": "Asked in the weakness section", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The problem is interesting. The dynamic filtering method is easily implementable. \n\n2. Both the applications are well selected and real world. The method shows a remarkable 17% gain over individual experts for the financial market direction task. This is pretty significant for a noisy problem.", "weaknesses": "1. The writing can be improved in some sections, specifically the implications of theorem 1 and 2 need to be explained better.\n\n2. In the TSF application it is unclear at what granularity of time-steps is the filtering applied. For instance if the horizon is 720, do the individual experts predict every 60 time-steps and then their 60 step look-ahead predictions are combined and then we move to the task of predicting the next 60 time-steps. Or do they predict one step at a time for 720 time-steps?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Dj2zxqNnnD", "reviewer": "Reviewer_5hye", "summary": "This work proposes to combine the predictions of multiple (large language) models in order to predict univariate time series. To optimise the weightings associated with each model based on past performances, the authors employ a continuous-time finite-state hidden Markov model (and an associated filtering algorithm).", "questions": "1. Regarding the loss functions discussed in Lines 129--134, can you explain how the classification scenario works? This would require $Y_t$ to be in $[0, 1]$. But I don't see how this can be guaranteed with the Brownian motion component in Equation 1.\n\n2. Can you explain why we have \"$w_t$\" rather than \"$w_s$\" in Equation 1 and Line 101?\n\n3. Why does the model need to be specified in continuous time (thus complicating the methodology). For the applications considered in this work, wouldn't a discrete time model, i.e. a simple discrete-time hidden Markov model, suffice?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 1, "presentation": 1, "contribution": 3, "strengths": "**Novelty:** To my knowledge, this methodology is novel.\n\n**Significance/contribution.** Making use of multiple (large language) models and using a filtering algorithm to decide in an online manner how to weight the predictions from each of these seems like a sensible idea.", "weaknesses": "**Major comments:**\n\n1. **Clarity needs to be drastically improved.** I believe Sections 1, 2 and 3 need to be substantially rewritten to improve clarity. I do not believe that this manuscript is ready for publication in ICLR based on the presentation alone. For instance:\n\n    - There seems to be no clear structure/order to the explanation of the model/methodology in Sections 1--3.\n    - Key aspects of the methodology (including some notation) are explained in a figure caption (of Figure 2) and this figure is then not even referenced until the following page.\n    - Important parts of the model, e.g., the processes $W$, $w$ or $x$ are used for several pages, before being defined in the middle of a generic-seeming paragraph titled \"Probability Theory\" at the end of Section 2.\n    - Section 2, in particular, is a collection of disjointed paragraphs lacking a clear structure.\n    - Sections 1 and 2 switch between continuous time and discrete time without explanation. Indeed, a time-discretisation is briefly mentioned in Section 3 but without sufficient detail.\n    - Enumerating a bunch of \"helper functions\" (Equations 2--4) at the reader without much intuition or context makes for a difficult read.\n\n    I think these sections should start with a high-level explanation of the (Wonham-Shiryaev) filtering algorithm and the model that this filtering algorithm is targetting. The specification of the model should be consolidated in one place.\n\n\n2. **Undefined/poorly defined notation.** Related to the previous point, there is quite a bit of undefined or poorly defined notation (or notation that is only defined much later after it has already been extensively used). For instance:\n\n    - In the caption of Figure 2 (and also in Line 9 of Algorithm 1), what are \"$\\pi_n$\" and \"$\\pi^n$\"? Is this the same as \"$\\pi_t^{(n)}$\"?\n    - In Line 178, what does \"$Q_t^{n:i,j}$\" mean?\n    - In Line 5 of Algorithm 1, what is \"$\\ell$\"? Page 2 defines a loss function $\\ell$ but this has two arguments rather than one.\n    - In Line 15 of Algorithm 1, what does \"$P^{(n)}$\" mean?\n    - In Line 305, what does \"$P_{w_{t+1}|w_{t-H:t}}$\" mean?\n\n3. **Soundness.** The presentational issues outlined above have made it impossible for me to verify that the proposed algorithm is valid in any meaningful sense. Hence I have to give a low score here.\n\n\n\n**Minor comments:**\n\n* Abstract: \"remarkable\" does not seem very objective.\n\n* Figures should not appear in the text before they have been referenced. For instance, Figure 1 seems not to be referenced at all; Figure 2 is only referenced a page after it is displayed.\n\n* The legends/annotations in some of the figures (e.g. Figure 1) are too small.\n\n* Avoid \"forward\" referencing of equations, i.e. referencing equations before they appear in the text (see, e.g., Lines 81, 88, 191).\n\n* Switching between bold and non-bold symbols for some of the mathematical quantities in Section 2 is confusing.\n\n* The sentence \"Each $n$th expert additionally provides their best ranking $\\pi_t^{(n)}$ of the reliability of each expert’s performance thus far\" in Lines 83--84 is very ambiguous (but seems central to understanding the proposed methodology).\n\n* L284: The last sentence (starting with \"Furthermore, ...\") does not seem to contain a statement.\n\n\n**Typos:**\n\n* Abstract: Wohman-Shiryaev -> Wonham-Shiryaev\n\n* L81: \"predictions\" -> \"prediction\"\n\n* Eq. 2: Presumably, the time subscript $s$ (rather than $t$) is a typo?\n\n* L163: The name \"transition matrix\" is confusing since this would normally be understood to mean $\\mathcal{P}_N$. Presumably, \"transition-rate matrix\" or \"intensity matrix\" or \"$Q$-matrix\" is meant here.\n\n* L165: Shouldn't this be \"sum to 0\" instead of \"sum to 1\"?\n\n* L173: \"Eq. equation\"\n\n* L220: missing space", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6vmhx1Dc2l", "reviewer": "Reviewer_bkmt", "summary": "This paper introduces MoE-F, a novel mechanism for adaptively combining multiple pre-trained Large Language Models (LLMs) for online time-series prediction tasks. The key innovation is framing expert selection as a finite state-space continuous-time Hidden Markov model and leveraging stochastic filtering techniques to dynamically weight expert predictions.", "questions": "How does the performance of MoE-F degrade with increasing number of experts, and what are the computational scaling implications?\n\nWhat is the impact of different market regimes (high volatility vs. low volatility) on the filtering performance?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The paper demonstrates significant originality in its approach to combining LLMs through the lens of stochastic filtering theory. This represents a creative fusion of classical control theory with modern machine learning. The theoretical foundations are rigorous, with formal optimality guarantees derived for both the parallel filtering and robust aggregation phases.", "weaknesses": "While the theoretical analysis is strong, there are some limitations in the experimental validation. The financial market experiments focus primarily on a single dataset (NIFTY), and broader validation across different types of time series problems would strengthen the claims of generality. The ablation studies, while informative, could be expanded to provide deeper insights into the relative importance of different components of the MoE-F architecture.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)