SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Consider again the generative model
$$
z _ {1} \sim \mathcal {N} \left(\mu_ {0}, \Sigma_ {0}\right), \quad z _ {t} \sim \mathcal {N} \left(A _ {t} z _ {t - 1} + b _ {t}, Q _ {t}\right) \tag {63}
$$
Recall from Eq. (47) that the  $q(z;\omega)$  will have some parameters from the expected log prior and some parameters from the recognition potential. We separate out the parameters into individual factors, starting with a prior factor on the initial state:
$$
\omega_ {0} = \left[ \begin{array}{c} h _ {0} \\ - \frac {1}{2} J _ {0} \end{array} \right] t (z _ {1}) = \left[ \begin{array}{c} z _ {1} \\ z _ {1} z _ {1} ^ {T} \end{array} \right] \tag {64}
$$
Where  $h_0, -\frac{1}{2} J_0$  are the expected statistics of an NIw prior. Further, we place a transition factor on each adjacent pair of states, given by Eq. (49). To simplify notation, we write
$$
t \left(z _ {t}, z _ {t + 1}\right) = \left[ \begin{array}{c} z _ {t} \\ z _ {t} z _ {t} ^ {T} \\ z _ {t} z _ {t + 1} ^ {T} \\ z _ {t + 1} z _ {t + 1} ^ {T} \\ z _ {t + 1} \end{array} \right], \quad \omega_ {t + 1} = \left[ \begin{array}{c} - h _ {1, t} \\ - \frac {1}{2} J _ {1 1, t} \\ J _ {1 2, t} \\ - \frac {1}{2} J _ {2 2, t} \\ h _ {2, t} \end{array} \right] = \left[ \begin{array}{c} - \mathbb {E} \left[ A _ {t} ^ {T} Q _ {t} ^ {- 1} b _ {t} \right] \\ - \frac {1}{2} \mathbb {E} \left[ A _ {t} ^ {T} Q _ {t} ^ {- 1} A _ {t} \right] \\ \mathbb {E} \left[ A _ {t} ^ {T} Q _ {t} ^ {- 1} \right] \\ - \frac {1}{2} \mathbb {E} \left[ Q _ {t} ^ {- 1} \right] \\ \mathbb {E} \left[ Q _ {t} ^ {- 1} b _ {t} \right] \end{array} \right] \tag {65}
$$
For the SLDS, each of these parameters is outputted by the mean field message passing function and is a convex combination of the corresponding parameters for each cluster, weighted by  $\mathbb{E}_{q(k_t)}[t(k_t)]$ . For the LDS, these values are time-homogeneous.
Note that there is also a factor outputted by the recognition network output at each time step:
$$
\lambda_ {t} = \left[ \begin{array}{c} r _ {t} \\ - \frac {1}{2} R _ {t} \end{array} \right] t (z _ {t}) = \left[ \begin{array}{c} z _ {t} \\ z _ {t} z _ {t} ^ {T} \end{array} \right] \tag {66}
$$
Because the parameters are expectations, there may be no values of  $\hat{A}_t,\hat{Q}_t,\hat{b}_t$  which satisfy
$$
\left[ \begin{array}{c} - \hat {A} _ {t} ^ {T} \hat {Q} _ {t} ^ {- 1} \hat {b} _ {t} \\ - \frac {1}{2} \hat {A} _ {t} ^ {T} \hat {Q} _ {t} ^ {- 1} \hat {A} _ {t} \\ \hat {A} _ {t} ^ {T} \hat {Q} _ {t} ^ {- 1} \\ - \frac {1}{2} \hat {Q} _ {t} ^ {- 1} \\ \hat {Q} _ {t} ^ {- 1} \hat {b} _ {t} \end{array} \right] = \left[ \begin{array}{c} - h _ {1, t} \\ - \frac {1}{2} J _ {1 1, t} \\ J _ {1 2, t} \\ - \frac {1}{2} J _ {2 2, t} \\ h _ {2, t} \end{array} \right] \tag {67}
$$
And as a result, it might not be equivalent to any single LDS with fixed parameters. This prevents us from using general Kalman smoother algorithms, so we must return to the definition of exponential families and belief propagation.
The goal of inference is to compute the marginals  $q(z_{t})$ , draw samples, and compute  $\log Z$ . Using the belief propagation algorithm, we know  $q(z_{t})$  is proportional to the product of incoming messages from factors which involve  $z_{t}$ . In belief propagation, messages pass from factors to variables and back. Thus we write
$$
q \left(z _ {t}\right) \propto m _ {\omega_ {t}, z _ {t}} \left(z _ {t}\right) \cdot m _ {\omega_ {t + 1}, z _ {t}} \left(z _ {t}\right) \cdot m _ {\lambda_ {t}, z _ {t}} \left(z _ {t}\right) \tag {68}
$$
Where  $m_{\omega, z}(z)$  is the message from the factor with parameter  $\omega$  to variable  $z$ . We similarly define  $m_{z, \omega}(z)$  to be the message from variable to factor. Note that all  $m_{..}(z_t)$  are provably Gaussian, and can therefore each message function can be minimally described by the Gaussian natural parameters.
Messages from variables to factors are the product of incoming messages, e.g.
$$
m _ {z _ {t}, \omega_ {t + 1}} \left(z _ {t}\right) = m _ {\omega_ {t}, z _ {t}} \left(z _ {t}\right) \cdot m _ {\lambda_ {t}, z _ {t}} \left(z _ {t}\right) \tag {69}
$$
While messages from factors to variables are computed by integrating over incoming messages times the factor potential, e.g.
$$
m _ {\omega_ {t}, z _ {t}} \left(z _ {t}\right) = \int_ {z _ {t - 1}} m _ {z _ {t - 1}, \omega_ {t}} \left(z _ {t - 1}\right) \cdot \exp \left\{\left\langle \omega_ {t}, t \left(z _ {t - 1}, z _ {t}\right) \right\rangle \right\} d z _ {t - 1} \tag {70}
$$
We attempt to match our notation to the typical Kalman smoother algorithm, by defining filtered messages  $J_{t|t}$ ,  $h_{t|t}$  as the natural parameters of  $m_{z_t,\omega_{t+1}}(z_t)$ , i.e. the distribution which has accumulated the first  $\omega_{1:t}$  and  $\lambda_{1:t}$ , which correspond to the first  $t$  "observations". We similarly define predicted messages  $J_{t+1|f}, h_{t+1|f}$  as the natural parameters of  $m_{\omega_{t+1},z_{t+1}}(z_{t+1})$ , which corresponds to a distribution over  $z_{t+1}$  given the first  $t$  "observations". Finally we describe the smoothed messages  $J_{t|T}, h_{t|T}$  to be the natural parameters of  $q(z_t)$  (note that for simplicity the natural parameters for each distribution are actually  $-\frac{1}{2}J_{\cdot|.}, h_{\cdot|.}$  and the negative one half is assumed to be factored out).
The message passing algorithm is outlined below:
Input:  $\omega_{t},\lambda_{t}$
Initialize:
$$
F _ {1 \mid 1} = J _ {0} + R _ {1} \tag {71}
$$
$$
f _ {1 \mid 1} = h _ {0} + r _ {1} \tag {72}
$$
$$
\log Z = 0 \tag {73}
$$
Filter: for  $t = 1, \dots, T - 1$
Predict:
$$
P _ {t} = F _ {t \mid t} + J _ {1 1, t} \tag {74}
$$
$$
F _ {t + 1 \mid t} = J _ {2 2, t} - J _ {1 2, t} ^ {T} \cdot P _ {t} ^ {- 1} \cdot J _ {1 2, t} \tag {75}
$$
$$
f _ {t + 1 \mid t} = h _ {2, t} + J _ {1 2, t} ^ {T} \cdot P _ {t} ^ {- 1} \cdot \left(f _ {t \mid t} - h _ {1, t}\right) \tag {76}
$$
$$
\log Z = \log Z + \frac {1}{2} \left(\left(f _ {t | t} - h _ {1, t}\right) ^ {T} P _ {t} ^ {- 1} \left(f _ {t | t} - h _ {1, t}\right) - \log | P _ {t} |\right) \tag {77}
$$
Measurement:
$$
F _ {t + 1 \mid t + 1} = F _ {t + 1 \mid t} + R _ {t + 1} \tag {78}
$$
$$
f _ {t + 1 \mid t + 1} = f _ {t + 1 \mid t} + r _ {t + 1} \tag {79}
$$
Finalize:
$$
\log Z = \log Z + \frac {1}{2} \left(f _ {T | T} ^ {T} F _ {T | T} ^ {- 1} F _ {T | T} - \log | F _ {T | T} |\right) \tag {80}
$$
Smother: for  $t = T - 1,\ldots ,1$
$$
C _ {t} = F _ {t + 1 \mid T} - F _ {t + 1 \mid t} + J _ {2 2, t} \tag {81}
$$
$$
F _ {t \mid T} = F _ {t \mid t} + J _ {1 1, t} - J _ {1 2, t} \cdot C _ {t} ^ {- 1} \cdot J _ {1 2, t} ^ {T} \tag {82}
$$
$$
f _ {t \mid T} = f _ {t \mid t} - h _ {1, t} + J _ {1 2, t} C _ {t} ^ {- 1} \left(f _ {t + 1 \mid T} - f _ {t + 1 \mid t} + h _ {2, t}\right) \tag {83}
$$
Sample: (the  $\mathcal{N}$  below are taking natural parameters as input)
$$
z _ {T} \sim \mathcal {N} \left(f _ {T \mid T}, - \frac {1}{2} F _ {T \mid T}\right) \tag {84}
$$
$$
\text {f o r} t = T - 1, \dots , 1 \quad z _ {t} \sim \mathcal {N} \left(f _ {t | t} + J _ {1 2, t} z _ {t + 1} - h _ {1, t}, - \frac {1}{2} \left(F _ {t | t} + J _ {1 1, t}\right)\right) \tag {85}
$$
Return: marginal params  $f_{t|T}, F_{t|T}; \log Z$ ; and samples  $x_{1:T}$ .
From the log marginals, we can recover the expected statistics:
$$
\Sigma_ {i} = F _ {t \mid T} ^ {- 1} \tag {86}
$$
$$
\mu_ {i} = \Sigma_ {i} f _ {t | T} \tag {87}
$$
$$
E \left[ z _ {i} \right] = \mu_ {i} \tag {88}
$$
$$
E \left[ z _ {i} z _ {i} ^ {T} \right] = \Sigma_ {i} + \mu_ {i} \mu_ {i} ^ {T} \tag {89}
$$
$$
E \left[ z _ {i} z _ {i + 1} ^ {T} \right] = \Sigma_ {i} J _ {1 2, t} C _ {t} ^ {- 1} + \mu_ {i} \mu_ {i + 1} ^ {T} \tag {90}
$$
Derivation of filter: The measurement step trivially follows from Eq. (69). The predict step, on the other hand, attempts to compute Eq. (70), which can be written out:
$$
\int_ {z _ {t}} \exp \left\{\left\langle \left[ \begin{array}{c} f _ {t | t} \\ - \frac {1}{2} F _ {t | t} \end{array} \right], \left[ \begin{array}{c} z _ {t} \\ z _ {t} z _ {t} ^ {T} \end{array} \right] \right\rangle + \left\langle \left[ \begin{array}{c} - \frac {1}{2} J _ {1 1, t} \\ J _ {1 2, t} \\ - \frac {1}{2} J _ {2 2, t} \\ - h _ {1, t} \\ h _ {2, t} \end{array} \right], \left[ \begin{array}{c} z _ {t} z _ {t} ^ {T} \\ z _ {t} z _ {t + 1} ^ {T} \\ z _ {t + 1} z _ {t + 1} ^ {T} \\ z _ {t} \\ z _ {t + 1} \end{array} \right] \right\rangle \right\} d z _ {t} \tag {91}
$$
$$
\begin{array}{l} = \exp \left\{\left\langle \left[ \begin{array}{c} - \frac {1}{2} J _ {2 2, t} \\ h _ {2, t} \end{array} \right], \left[ \begin{array}{c} z _ {t + 1} z _ {t + 1} ^ {T} \\ z _ {t + 1} \end{array} \right] \right\rangle \right\} \int_ {z _ {t}} \exp \left\{\left\langle \left[ \begin{array}{c} f _ {t | t} + J _ {1 2, t} z _ {t + 1} - h _ {1, t} \\ - \frac {1}{2} (F _ {t | t} + J _ {1 1, t}) \end{array} \right], \left[ \begin{array}{c} z _ {t} \\ z _ {t} z _ {t} ^ {T} \end{array} \right] \right\rangle \right\} d z _ {t} (92) \\ = \exp \left\{\left\langle \left[ \begin{array}{c} - \frac {1}{2} J _ {2 2, t} \\ h _ {2, t} \end{array} \right], \left[ \begin{array}{c} z _ {t + 1} z _ {t + 1} ^ {T} \\ z _ {t + 1} \end{array} \right] \right\rangle + \log Z \left(f _ {t | t} - h _ {1, t} + J _ {1 2, t} z _ {t + 1}, F _ {t | t} + J _ {1 1, t}\right) \right\} (93) \\ \end{array}
$$
What is that log partition function term at the end?
$$
\frac {1}{2} \left(f _ {t | t} - h _ {1, t} + J _ {1 2, t} z _ {t + 1}\right) ^ {T} \left(F _ {t | t} + J _ {1 1, t}\right) ^ {- 1} \left(f _ {t | t} - h _ {1, t} + J _ {1 2, t} z _ {t + 1}\right) - \frac {1}{2} \log \left| F _ {t | t} + J _ {1 1, t} \right| \tag {94}
$$
Simplifying  $P_{t} = F_{t|t} + J_{11,t}$ , we recover normalization term
$$
\frac {1}{2} \left(f _ {t | t} - h _ {1, t}\right) ^ {T} P _ {t} ^ {- 1} \left(f _ {t | t} - h _ {1, t}\right) - \frac {1}{2} \log | P _ {t} | \tag {95}
$$
Linear term
$$
z _ {t + 1} ^ {T} J _ {1 2, t} ^ {T} P _ {t} ^ {- 1} \left(f _ {t | t} - h _ {1, t}\right) \tag {96}
$$
And quadratic term
$$
\frac {1}{2} z _ {t + 1} ^ {T} J _ {1 2, t} ^ {T} P _ {t} ^ {- 1} J _ {1 2, t} z _ {t + 1} \tag {97}
$$
Thus the entire message is:
$$
\exp \left\{\left\langle \left[ h _ {2, t} + J _ {1 2, t} ^ {T} P _ {t} ^ {- 1} \left(f _ {t | t} - h _ {1, t}\right) \right], \left[ \begin{array}{c} z _ {t + 1} \\ z _ {t + 1} z _ {t + 1} ^ {T} \end{array} \right] \right\rangle + \frac {1}{2} \left(f _ {t | t} - h _ {1, t}\right) ^ {T} P _ {t} ^ {- 1} \left(f _ {t | t} - h _ {1, t}\right) - \frac {1}{2} \log | P _ {t} | \right\} \tag {98}
$$
To calculate  $\log Z$ , instead of normalizing each message to be a valid distribution, we can think of normalizing each message so it's of the form  $\exp \{\langle \eta ,t(z)\rangle \}$ , i.e. remove all normalizers. Then to calculate the entire model's  $\log Z$ , we must accrue all normalizers that appear in predict steps (none appear in measurement steps) and simply add the normalizer for the last message.
Derivation of Smoother: Note that
$$
q \left(z _ {t}\right) \propto m _ {\omega_ {t + 1}, z _ {t}} \left(z _ {t}\right) \cdot m _ {z _ {t}, \omega_ {t + 1}} \left(z _ {t}\right) \tag {99}
$$
The second term is the filtered message computed in the filtering step. Thus we must simply compute the backwards messages. The message becomes:
$$
\begin{array}{l} q \left(z _ {t}\right) \propto m _ {z _ {t}, \omega_ {t + 1}} \left(z _ {t}\right) \cdot m _ {\omega_ {t + 1}, z _ {t}} \left(z _ {t}\right) (100) \\ \propto m _ {z _ {t}, \omega_ {t + 1}} (z _ {t}) \cdot \int m _ {z _ {t + 1}, \omega_ {t + 1}} (z _ {t + 1}) \exp \left\{\langle \omega_ {t + 1}, t (z _ {t}, z _ {t + 1}) \rangle \right\} d z _ {t + 1} (101) \\ = m _ {z _ {t}, \omega_ {t + 1}} (z _ {t}) \cdot \int \left(m _ {\omega_ {t + 2}, z _ {t + 1}} \left(z _ {t + 1}\right) \cdot m _ {\lambda_ {t + 1}, z _ {t + 1}} \left(z _ {t + 1}\right)\right) \cdot \exp \left\{\langle \omega_ {t + 1}, t \left(z _ {t}, z _ {t + 1}\right) \rangle \right\} d z _ {t + 1} (102) \\ \propto m _ {z _ {t}, \omega_ {t + 1}} (z _ {t}) \cdot \int \frac {q (z _ {t + 1})}{m _ {\omega_ {t + 1} , z _ {t + 1}} (z _ {t + 1})} \cdot \exp \left\{\langle \omega_ {t + 1}, t (z _ {t}, z _ {t + 1}) \rangle \right\} d z _ {t + 1} (103) \\ \end{array}
$$
Now let's evaluate the integral:
$$
\begin{array}{l} \int \exp \left\{\left\langle \left[ \begin{array}{c} - \frac {1}{2} J _ {1 1, t} \\ J _ {1 2, t} \\ - \frac {1}{2} J _ {2 2, t} \\ - h _ {1, t} \\ h _ {2, t} \end{array} \right], \left[ \begin{array}{c} z _ {t} z _ {t} ^ {T} \\ z _ {t} z _ {t + 1} ^ {T} \\ z _ {t + 1} z _ {t + 1} ^ {T} \\ z _ {t} \\ z _ {t + 1} \end{array} \right] \right\rangle + \left\langle \left[ \begin{array}{c} f _ {t + 1 | T} - f _ {t + 1 | t} \\ - \frac {1}{2} (F _ {t + 1 | T} - F _ {t + 1 | t}) \end{array} \right], \left[ \begin{array}{c} z _ {t + 1} \\ z _ {t + 1} z _ {t + 1} ^ {T} \end{array} \right] \right\rangle \right\} d z _ {t + 1} (104) \\ = \exp \left\{\left\langle \left[ \begin{array}{c} - \frac {1}{2} J _ {1 1, t} \\ - h _ {1, t} \end{array} \right], \left[ \begin{array}{l} z _ {t} z _ {t} ^ {T} \\ z _ {t} \end{array} \right] \right\rangle \right\} \int_ {z _ {t + 1}} \exp \left\{\left\langle \left[ \begin{array}{c} f _ {t + 1 | T} - f _ {t + 1 | t} + h _ {2, t} + J _ {1 2, t} ^ {T} z _ {t} \\ - \frac {1}{2} \left(F _ {t + 1 | T} - F _ {t + 1 | t} + J _ {2 2, t}\right) \end{array} \right], \left[ \begin{array}{c} z _ {t + 1} \\ z _ {t + 1} z _ {t + 1} ^ {T} \end{array} \right] \right\rangle \right\} d z _ {t + 1} (105) \\ = \exp \left\{\left\langle \left[ \begin{array}{c} - \frac {1}{2} J _ {1 1, t} \\ - h _ {1, t} \end{array} \right], \left[ \begin{array}{c} z _ {t} z _ {t} ^ {T} \\ z _ {t} \end{array} \right] \right\rangle + \log Z \left(f _ {t + 1 | T} - f _ {t + 1 | t} + h _ {2, t} + J _ {1 2, t} ^ {T} z _ {t}, F _ {t + 1 | T} - F _ {t + 1 | t} + J _ {2 2, t}\right) \right\} (106) \\ \end{array}
$$
Again, let's evaluate the log partition function at the end, simplifying  $C_t = F_{t+1|T} - F_{t+1|t} + J_{22,t}$ :
$$
\frac {1}{2} \left(f _ {t + 1 | T} - f _ {t + 1 | t} + h _ {2, t} + J _ {1 2, t} ^ {T} z _ {t}\right) ^ {T} C _ {t} ^ {- 1} \left(f _ {t + 1 | T} - f _ {t + 1 | t} + h _ {2, t} + J _ {1 2, t} ^ {T} z _ {t}\right) - \frac {1}{2} \log | C _ {t} | \tag {107}
$$
We recover linear term
$$
z _ {t} ^ {T} J _ {1 2, t} C _ {t} ^ {- 1} \left(f _ {t + 1 | T} - f _ {t + 1 | t} + h _ {2, t}\right) \tag {108}
$$
$$
\frac {1}{2} z _ {t} ^ {T} J _ {1 2, t} C _ {t} ^ {- 1} J _ {1 2, t} ^ {T} z _ {t} \tag {109}
$$
Combining these with the  $J_{11,t}$  and  $h_{1,t}$  in Eq. (106) and the filtered distributions outside the integral in Eq. (103), we arrive at the update equations above.
Derivation of Sampler: Moving backwards, we want to sample from
$$
q \left(z _ {t} \mid z _ {t + 1}\right) \propto m _ {z _ {t}, \omega_ {t + 1}} \left(z _ {t}\right) \cdot \exp \left\{\left\langle \omega_ {t + 1}, t \left(z _ {t}, z _ {t + 1}\right) \right\rangle \right\} \tag {110}
$$
The rest of the derivation is easy.
D.2 Parallel LDS Inference
This adaptation of the Kalman smoother algorithm is highly sequential and therefore inefficient on GPU architectures. We adapt the technique of Särkkä & García-Fernández [56] to parallelize our algorithm across time steps.
Let us define the kernel of a (possibly un-normalized) exponential family distribution as the inner product portion of the distribution:
$$
\ker \left(\exp \left\{\langle \eta , t (z) \rangle - c \right\}\right) = \exp \left\{\langle \eta , t (z) \rangle \right\} \tag {111}
$$
Note that  $\ker(f(z)) \propto f(z)$ . We define two sequences of functions:
$$
\begin{array}{l} g _ {t} \left(z _ {t - 1}\right) = \ker \left(\int_ {z _ {t}} \exp \left\{\langle t \left(\theta_ {t}\right), t \left(z _ {t - 1}, z _ {t}\right) \rangle \right\} \cdot \exp \left\{\langle \lambda_ {t}, t \left(z _ {t}\right) \rangle \right\} d z _ {t}\right) (112) \\ = \exp \left\{\left\langle \left[ \begin{array}{c} - \frac {1}{2} \left(J _ {1 1, t} - J _ {1 2, t} \left(R _ {t} + J _ {2 2, t}\right) ^ {- 1} J _ {1 2, t} ^ {T}\right) \\ - h _ {1, t} + J _ {1 2, t} \left(R _ {t} + J _ {2 2, t}\right) ^ {- 1} \left(r _ {t} + h _ {2}\right) \end{array} \right], \left[ \begin{array}{c} z _ {t - 1} z _ {t - 1} ^ {T} \\ z _ {t - 1} \end{array} \right] \right\rangle \right\} (113) \\ = \exp \left\{\left\langle \left[ \begin{array}{c} - \frac {1}{2} \Gamma_ {t} \\ \gamma_ {t} \end{array} \right], \left[ \begin{array}{c} z _ {t - 1} z _ {t - 1} ^ {T} \\ z _ {t - 1} \end{array} \right] \right\rangle \right\} (114) \\ \end{array}
$$
Thus  $g_{t}(z_{t - 1})$  is defined by natural parameters  $\Gamma_t, \gamma_t$ . For  $t = 1$ , we set  $\Gamma_1 = 0$ ,  $\gamma_1 = 0$  as there is no time-step 0. Thus we claim  $g_{1}(z_{0}) = 1$  is a uniform function.
Further define  $f_{t}(z_{t - 1},z_{t})$
$$
\begin{array}{l} f _ {t} \left(z _ {t - 1}, z _ {t}\right) = \exp \left\{\langle t \left(\theta_ {t}\right), t \left(z _ {t - 1}, z _ {t}\right) \rangle \right\} \cdot \exp \left\{\langle \lambda_ {t}, t \left(z _ {t}\right) \rangle \right\} / g _ {t} \left(z _ {t - 1}\right) (115) \\ = \exp \left\{\left\langle \left[ \begin{array}{c} - \frac {1}{2} \left(J _ {1 1, t} - \Gamma_ {t}\right) \\ - \left(h _ {1, t} - \gamma_ {t}\right) \\ J _ {1 2, t} \\ h _ {2, t} + r _ {t} \\ - \frac {1}{2} \left(J _ {2 2, t} + R _ {t}\right) \end{array} \right], \left[ \begin{array}{c} z _ {t - 1} z _ {t - 1} ^ {T} \\ z _ {t - 1} \\ z _ {t - 1} z _ {t} ^ {T} \\ z _ {t} \\ z _ {t} z _ {t} ^ {T} \end{array} \right] \right\rangle \right\} (116) \\ = \exp \left\{\left\langle \left[ \begin{array}{c} - \frac {1}{2} \Phi_ {1 1, t} \\ - \phi_ {1, t} \\ \Phi_ {1 2, t} \\ \phi_ {2, t} \\ - \frac {1}{2} \Phi_ {2 2, t} \end{array} \right], \left[ \begin{array}{c} z _ {t - 1} z _ {t - 1} ^ {T} \\ z _ {t - 1} \\ z _ {t - 1} z _ {t} ^ {T} \\ z _ {t} \\ z _ {t} z _ {t} ^ {T} \end{array} \right] \right\rangle \right\} (117) \\ \end{array}
$$
We similarly define  $f_{1}(z_{0},z_{1}) = f_{1}(z_{1})$ , or equivalently  $\Phi_{11,1} = \Phi_{12,1} = 0$  and  $\phi_{1,1} = 0$  as there is no  $z_{0}$ .  $\phi_{2,t} = h_0 + r_1$  and  $\Phi_{22,t} = J_0 + R_1$  to include the factors  $h_0, J_0$  from the initial state distribution  $p(z_{1})$ .
We define the associative operation:
$$
\left(f _ {i}, g _ {i}\right) \otimes \left(f _ {j}, g _ {j}\right) = \left(f _ {i j}, g _ {i j}\right) \tag {118}
$$
$$
f _ {i j} (a, c) = \ker \left(\frac {\int g _ {j} (b) f _ {j} (b , c) f _ {i} (a , b) d b}{\int g _ {j} (b) f _ {i} (a , b) d b}\right) \tag {119}
$$
$$
g _ {i j} (a) = \ker \left(g _ {i} (a) \int g _ {j} (b) f _ {i} (a, b) d b\right) \tag {120}
$$
Proposition D.1. The operation  $\otimes$  is associative.
Proof. Särkkä & García-Fernández [56] define an operator
$$
\left(f _ {i}, g _ {i}\right) \times \left(f _ {j}, g _ {j}\right) = \left(\hat {f} _ {i j}, \hat {g} _ {i j}\right) \tag {121}
$$
$$
f _ {i j} (a, c) = \frac {\int g _ {j} (b) f _ {j} (b , c) f _ {i} (a , b) d b}{\int g _ {j} (b) f _ {i} (a , b) d b} \tag {122}
$$
$$
g _ {i j} (a) = g _ {i} (a) \int g _ {j} (b) f _ {i} (b | a) d b \tag {123}
$$
and prove that this operator is associative. Further, any rescaling of inputs leads to a simple rescaling of outputs due to the multi-linear nature of these functions. Thus for constants  $c_{1}, c_{2}, c_{3}, c_{4}$  there exist constants  $c_{5}, c_{6}$  such that
$$
\left(c _ {1} f _ {i}, c _ {2} g _ {i}\right) \times \left(c _ {3} f _ {j}, c _ {4} g _ {j}\right) = \left(c _ {5} \hat {f} _ {i j}, c _ {6} \hat {g} _ {i j}\right) \tag {124}
$$
This gives us that
$$
\begin{array}{l} \left(\left(f _ {i}, g _ {i}\right) \otimes \left(f _ {j}, g _ {j}\right)\right) \otimes \left(f _ {k}, g _ {k}\right) = \left(\ker \left(\hat {f} _ {i j}\right), \ker \left(\hat {g} _ {i j}\right)\right) \otimes \left(f _ {k}, g _ {k}\right) (125) \\ = \left(c _ {1} \hat {f} _ {i j}, c _ {2} \hat {g} _ {i j}\right) \otimes \left(f _ {k}, g _ {k}\right) (126) \\ = \ker \left(\left(c _ {1} \hat {f} _ {i j}, c _ {2} \hat {g} _ {i j}\right) \times \left(f _ {k}, g _ {k}\right)\right) (127) \\ = \ker \left(\left(\hat {f} _ {i j}, \hat {g} _ {i j}\right) \times \left(f _ {k}, g _ {k}\right)\right) (128) \\ \end{array}
$$
A similar decomposition shows that
$$
\begin{array}{l} \left(f _ {i}, g _ {i}\right) \otimes \left(\left(f _ {j}, g _ {j}\right) \otimes \left(f _ {k}, g _ {k}\right)\right) = \ker \left(\left(f _ {i}, g _ {i}\right) \times \left(\hat {f} _ {j k}, \hat {g} _ {j k}\right)\right) (129) \\ = \ker \left(\left(\hat {f} _ {i j}, \hat {g} _ {i j}\right) \times \left(f _ {k}, g _ {k}\right)\right) (130) \\ = \left(\left(f _ {i}, g _ {i}\right) \otimes \left(\left(f _ {j}, g _ {j}\right)\right) \otimes \left(f _ {k}, g _ {k}\right) \right. (131) \\ \end{array}
$$
Where the second equality comes from the associativity of the  $\times$  operator.
Finally, we need to show that this operation gives us the right results:
Proposition D.2. Let  $a_{i} = (f_{i},g_{i})$ . Then
$$
a _ {1} \otimes a _ {2} \otimes \dots \otimes a _ {t} = \left(f _ {1: t}, g _ {1: t}\right) \tag {132}
$$
$$
f _ {1: t} \left(z _ {t}\right) = \exp \left\{\left\langle \left[ - \frac {1}{2} F _ {t | t} \right], \left[ z _ {t} z _ {t} ^ {T} \right] \right\rangle \right\} \tag {133}
$$
is parameterized by the filtered messages  $f_{t|t}, F_{t|t}$ .
Proof. We prove by induction that this holds when the operations are performed sequentially, namely
$$
\left(\left(\left(\left(a _ {1} \otimes a _ {2}\right) \otimes a _ {3}\right) \otimes \dots\right) \otimes a _ {t} \right. \tag {134}
$$
These "left-justified" operations are simpler, and thus allow us to easily demonstrate correctness. We then can make use of the associative property of  $\otimes$  to show that no matter the order of operations we get the correct result.
Recall that  $g_{1}(z_{0}) = 1$  is the uniform function (as there is no  $z_{0}$ ) and thus has  $\Gamma_{1} = 0, \gamma_{1} = 0$ . By induction, this must be true of all the cumulative kernels  $g_{ij}(z_0)$ , as if  $g_{i}(a)$  in Eq. (120) is independent of  $a$ , then  $g_{ij}(a)$  must be, too. Similarly, we recall that  $f_{1}(z_{0},z_{1}) = f_{1}(z_{1})$  is independent of its first argument and thus has  $\Phi_{11,1} = \Phi_{12,1} = 0$  and  $\phi_{1,1} = 0$ . Again, by induction all of the cumulative kernels  $f_{1:t}(z_0,z_t)$  must have their first three parameters equal to 0; in Eq. (119) if  $f_{i}(a,b)$  is independent of  $a$  then  $f_{ij}(a,c)$  will be, too. Thus  $f_{1:t}(z_0,z_t) = f_{1:t}(z_t)$ .
Next we show that  $f_{1:t}(z_t) = m_{z_t,\omega}(z_t)$ , the filtered messages from the sequential filter. For  $t = 1$ ,  $f_{1:1}(z_1) = m_{z_1,\omega}(z_1)$  by construction. We then see:
$$
\begin{array}{l} f _ {1: t + 1} \left(z _ {t + 1}\right) = \ker \left(\frac {\int_ {z _ {t}} g _ {t + 1} \left(z _ {t}\right) f _ {t + 1} \left(z _ {t + 1}\right) m _ {z _ {t} , \omega} \left(z _ {t}\right) d z _ {t}}{\int g _ {t + 1} \left(z _ {t}\right) f _ {1 : t} \left(z _ {t}\right) d z _ {t}}\right) (135) \\ = \ker \left(\int_ {z _ {t}} g _ {t + 1} \left(z _ {t}\right) f _ {t + 1} \left(z _ {t + 1}\right) m _ {z _ {t}, \omega} \left(z _ {t}\right) d z _ {t}\right) (136) \\ = \ker \left(\int_ {z _ {t}} \exp \left\{\langle \omega_ {t + 1}, t \left(z _ {t}, z _ {t + 1}\right) \rangle \right\} \cdot \exp \left\{\langle \lambda_ {t + 1}, t \left(z _ {t + 1}\right) \rangle \right\} \cdot m _ {z _ {t}, \omega} \left(z _ {t}\right) d z _ {t}\right) (137) \\ = \ker \left(\exp \left\{\left\langle \lambda_ {t + 1}, t \left(z _ {t + 1}\right) \right\rangle \right\} \cdot \int_ {z _ {t}} \exp \left\{\left\langle t \left(\theta_ {t + 1}\right), t \left(z _ {t}, z _ {t + 1}\right) \right\rangle \right\} \cdot m _ {z _ {t}, \omega} \left(z _ {t}\right) d z _ {t}\right) (138) \\ = \ker \left(\exp \left\{\left\langle \lambda_ {t + 1}, t \left(z _ {t + 1}\right) \right\rangle \right\} \cdot m _ {z _ {t + 1}, \omega} \left(z _ {t + 1}\right)\right) = m _ {z _ {t + 1}, \omega_ {t + 1}} \left(z _ {t + 1}\right) (139) \\ \end{array}
$$
Completing our proof!