When predict can also explain: few-shot prediction to select better neural latents

## Abstract

Abstract Latent variable models serve as powerful tools to infer underlying dynamics from observed neural activity. However, due to the absence of ground truth data, prediction benchmarks are often employed as proxies. In this study, we reveal the limitations of the widely-used ’co-smoothing’ prediction framework and propose an improved few-shot prediction approach that encourages more accurate latent dynamics. Utilizing a student-teacher setup with Hidden Markov Models, we demonstrate that the high co-smoothing model space can encompass models with arbitrary extraneous dynamics within their latent representations. To address this, we introduce a secondary metric – a few-shot version of co-smoothing. This involves performing regression from the latent variables to held-out channels in the data using fewer trials. Our results indicate that among models with near-optimal co-smoothing, those with extraneous dynamics underperform in the few-shot co-smoothing compared to ’minimal’ models devoid of such dynamics. We also provide analytical insights into the origin of this phenomenon. We further validate our findings on real neural data using two state-of-the-art methods: LFADS and STNDT. In the absence of ground truth, we suggest a proxy measure to quantify extraneous dynamics. By cross-decoding the latent variables of all model pairs with high co-smoothing, we identify models with minimal extraneous dynamics. We find a correlation between few-shot co-smoothing performance and this new measure. In summary, we present a novel prediction metric designed to yield latent variables that more accurately reflect the ground truth, offering a significant improvement for latent dynamics inference.

## 1 Introduction

In neuroscience, we often have access to simultaneously recorded neurons during certain behaviors. These observations, denoted $x$, are believed to be a window onto the actual hidden (or latent) dynamics of the relevant brain circuit, denoted $z$ []. Although in general these dynamics can be complex and high-dimensional, capturing them in a concrete mathematical model opens doors to reverse-engineering, revealing simpler explanations and insights [, ]. Inferring a model of the $z$ variables, also known as latent variable modeling, is part of the larger field of system identification with applications in many areas outside of neuroscience, such as fluid dynamics [] and finance [].

Because we don’t have ground truth for $z$, prediction metrics on held-out parts of $x$ are commonly used as a proxy []. However, it has been noted that prediction and explanation are often distinct endeavors []. For instance, [] use an example where ground truth is available to show how different models that all achieve good prediction nevertheless have varied latents that can differ from the ground truth. Such behavior might be expected when using highly expressive models with large latent spaces. Bad prediction with good latents is demonstrated by [] for the case of chaotic dynamics.

Various regularization methods on the latents have been suggested to improve the similarity of $z$ to the ground truth, such as recurrence and priors on external inputs [], low-dimensionality of trajectories [], low-rank connectivity [], injectivity constraints from latent to predictions [], low-tangling [], piecewise-linear dynamics []. However, the field lacks a quantitative, prediction-based metric that credits the simplicity of the latent representation—an aspect essential for interpretability and ultimately scientific discovery, while still enabling comparisons across a wide range of LVM architectures.

Here, we characterize the diversity of model latents achieving high co-smoothing, a standard prediction-based framework for Neural LVMs, and demonstrate potential pitfalls of this framework. We propose a few-shot variant of co-smoothing which, when used in conjunction with co-smoothing, differentiates varying latents. We verify this approach both on synthetic toy problems and state-of-the-art methods on neural data, providing an analytical explanation of why it works in a simple setting.

## 2 Related Work

Our work builds on recent developments in Neural LVMs for the discovery of latent structure in noisy neural data on single trials. We refer the reader to [] supplementary table 3 for a comprehensive list of Neural LVMs published from 2008-2021. Central to our work is the co-smoothing procedure for evaluating models based on the prediction of activity from held-out neurons provided held-in neuron activity from the same trial. Co-smoothing was first introduced in [] and [] for the validation of GPFA as a Neural LVM.

Pei et al. [] curated four datasets of neural activity recorded from behaving monkeys and established a framework to evaluate co-smoothing among other prediction-based metrics on several models in the form of a standardized benchmark and competition. This led to significant advances in prediction methods. Some advances came in the form of harnessing ideas and tools from machine learning such as ensembling populations of models [] and Bayesian hyperparameter optimization. Other advances involved incorporating statistical priors on the geometry of neural trajectories, yielding simpler and more interpretable methods [].

In contrast to prediction approaches, a parallel line of work focuses on explaining and validating Neural LVMs on synthetic data, enabling direct comparison with the ground truth [, , ]. Versteeg et al. [] validated their method with both ground truth and neural data, demonstrating high predictive performance with low-dimensional latents.

Our work straddles the interface of prediction and explanation approaches. We propose a predictive framework that attempts to evaluate using unsupervised prediction metrics that can also reveal the similarity of the LVM to the ground truth without direct access to it. To verify our predictive method, we map ground truth states and latents with synthetic data to correlate with the predictive framework.

Central to our work is the concept of few-shot learning a decoder from a frozen intermediate representation. Sorscher et al. [] developed a theory of geometric properties of representations that enables few-shot generalization to novel classes. They identified the geometric properties that determine a signal-to-noise ratio for classification, which dictates few-shot performance. While this setting differs from ours, links between our works are a topic for future research.

Another concept we introduce is cross-decoding across a population of models to find the most parsimonious representation. Several works compare representations of large model populations [, ]. They apply Canonical Correlation Analysis (CCA), a symmetric measure of representational similarity, whereas we use regression, which is not symmetric. The application to Neural LVMs may be novel.

One related approach comparing representations in deep neural networks is stitching components of separately trained (and subsequently frozen) models into a composite model using a simple linear layer [, ]. In contrast, here we focus on representation in the last layer. We reveal it’s simplicity by ‘weakening’ the final layer by re-training it with less target data. Cross-decoding across highly-performing models serves as a secondary measure of representation simplicity. To our knowledge, the use of few-shot generalization as a means to identify interpretable latent representations, particularly for Neural LVMs, is a novel idea.

## 3 Co-smoothing: a cross-validation framework

Let $\boldsymbol{x}\in\mathbb{Z}_{\geq 0}^{T\times N}$ be spiking neural activity of $N$ channels recorded over a finite window of time, i.e a trial, and subsequently quantised into $T$ time-bins. $x_{t,n}$ representing the number of spikes in channel $n$ during time-bin $t$. The data set $X:=\{\boldsymbol{x}^{(i)}\}_{i=1}^{S}$ consists of $S$ trials of the experiment.

The latent-variable model (LVM) approach posits that each time-point in the data $\boldsymbol{x}^{(i)}_{t}$ is a noisy measurement of a latent state $\boldsymbol{z}_{t}^{(i)}$.

To infer the latent trajectory $\boldsymbol{z}$ is to learn a mapping $f:\boldsymbol{x}\mapsto\boldsymbol{z}$. On what basis do we decide the inferred $\boldsymbol{z}$? We have no ground-truth on $z$, so instead we test the ability of $z$ to predict unseen or held-out data. Data may be held-out in time, e.g forward-prediction or in space, co-smoothing.[] We focus here on co-smoothing.

The set of $N$ available channels is partitioned into two: $N^{\text{in}}$ held-in channels and $N^{\text{out}}$ held-out channels. The $S$ trials are partitioned into train and test. During training, both channel partitions are available to the model and during test, only the held-in partition is available. During evaluation the model must generate rate-predictions $r^{\text{out}}$ for the held-out partition. This framework is visualised in figure 1A.

Importantly, the encoding-step or inference of the latents is done using a full time-window i.e analogous to smoothing in control-theoretic literature, whereas the decoding step, mapping the latents to predictions of the data is done on individual time-steps:

$$ $\displaystyle z_{t}$ $\displaystyle=f(x^{\text{in}}_{1:T};t)$ (1) $\displaystyle r_{t}^{\text{out}}$ $\displaystyle=g(z_{t}),$ (2) $$

where the superscript of $x$ denotes subsets of neurons, and the subscript denotes subsets of time bins. During evaluation the held-out data from test trials $\boldsymbol{x}^{\text{out}}$ is compared to the rate-predictions $r^{\text{out}}$ from the model using the co-smoothing metric $Q$ defined as the normalised log-likelihood given by:

$$ $\displaystyle Q(r_{t,n},x_{t,n})=\frac{1}{\mu_{n}\log 2}\bigg{(}\mathcal{L}(r_ {t,n};x_{t,n})-\mathcal{L}(\bar{r}_{n};x_{t,n})\bigg{)}$ (3) $\displaystyle\mathcal{Q}(R^{\text{out}}_{\text{test}},X^{\text{out}}_{\text{ test}})=\sum_{i\in\text{test}}\sum_{t=1}^{T}\sum_{n\in\text{held-out}}Q( \boldsymbol{r}^{(i)}_{t,n},\boldsymbol{x}^{(i)}_{t,n})$ (4) $$

where $\mathcal{L}$ is poisson log-likelihood $\bar{r}_{n}=\frac{1}{TS}\sum_{i}\sum_{t}x^{(i)}_{t,n}$ is a the mean rate for channel $n$ and $\mu_{n}:=\sum_{i}\sum_{t}x^{(i)}_{t,n}$ is the total number of spikes, following []. We use the notation $X^{\circ}_{\star}:=\{x^{(i)}_{1:T,n}\}_{i\in\star,n\in\circ}$, similarly for $R$.

Thus, inference of $z$ is done by the optimization:

$$ $\displaystyle f^{*},g^{*}=\text{argmax}_{f,g}\mathcal{Q}(R^{\text{out}}_{\text {test}},X^{\text{out}}_{\text{test}})$ (5) $$

without direct access to $X^{\text{out}}_{\text{test}}$.

Figure: Figure 1: A Overview of the co-smoothing cross-validation framework for Neural LVMs. Part of the data is used to infer latents, while the rest serves as a target for evaluation. The functions $f,g$ are optimized to improve co-smoothing. The lower arm is our few-shot proposal. where a function $g^{\prime}$ is optimized using the existing latents and a few observations of new neurons (see also Fig. 3).B Hypothesis: high co-smoothing ensures the model representation contains the ground truth, but not vice-versa. This may be revealed by the unequal performance of regression on the states in the two directions. C Several student HMMs are trained on a dataset generated by a single teacher HMM. The student$\rightarrow$teacher decoding error is low and tightly related to the co-smoothing score. D The teacher$\rightarrow$student decoding error is more varied and uncorrelated to co-smoothing. Dashed lines represent the ground truth, evaluating the teacher as a candidate model. Co-smoothing is $\mathcal{Q}(R^{\text{out}}_{\text{test}},X^{\text{out}}_{\text{test}})$, where a value of zero corresponds to predicting the mean activation for each neuron. Green and red arrows represent “Good” and “Bad” models respectively, presented in Fig. 2.
Refer to caption: x1.png

## 4 Good co-smoothing does not guarantee correct latents

It is reasonable to assume that being able to predict $x$ on held-out data will guarantee that the inferred latent is the true one ([, , , , , , , , , , , , , , ]). We hypothesize, however, that good prediction guarantees that the true latents are contained within the inferred ones (assuming observability of the true latents), but not vice versa (Figure 1B).

To verify this hypothesis, we simulate a scenario where we know the ground truth. Specifically, we use a student-teacher setting, where both student and teacher are described by a discrete-space, discrete-time Hidden Markov Model (HMM). The HMM has a state space $z\in\{1,2,\dots M\}$, and produces observations (emissions in HMM notation) along neurons $\boldsymbol{x}$, with a state transition matrix $A$, emission model $B$ and initial state distribution $\pi$. More explicitly:

$$ $\displaystyle A_{m,l}$ $\displaystyle=p(z_{t+1}=l|z_{t}=m)\$ $\displaystyle\forall\ m,l$ (6) $\displaystyle B_{m,n}$ $\displaystyle=p(x_{n,t}=1|z_{t}=m)\$ $\displaystyle\forall\ i,n$ (7) $\displaystyle\pi_{m}$ $\displaystyle=p(z_{0}=m)\$ $\displaystyle\forall\ m$ (8) $$

The same HMM can serve two roles: a) data-generation by sampling from (6), (7), (8) and b) inference of the latents from data on a trial-by-trial basis:

$$ $\displaystyle\xi_{t,m}^{(i)}$ $\displaystyle=f_{m}((x^{\text{in}}_{1:T})^{(i)})=p(z_{t}^{(i)}=m|(x^{\text{in} }_{1:T})^{(i)}),$ (9) $$

i.e smoothing, computed exactly with the well known forward-backward algorithm []. Note that although $z$ is the latent state of the HMM, we find it convenient to use its posterior probability mass function $\boldsymbol{\xi}_{t}$ as the relevant intermediate representation. To make predictions of the rates of held-out neurons for co-smoothing we compute:

$$ $\displaystyle r_{n,t}^{(i)}$ $\displaystyle=g_{n}(\boldsymbol{\xi}^{(i)}_{t})=\sum_{m}B_{m,n}\xi^{(i)}_{t,m}$ $\displaystyle\forall\ n\in\text{out},1\leq t\leq T,i\in\text{test}$ (10) $$

As a teacher, we constructed a 4-state model of a noisy chain $A_{m,l}\propto\mathbb{I}[l=(m+1)\mod M]+\epsilon$, with $\epsilon=1e-2$, $\pi=\frac{1}{M}$, and $B_{m,n}\sim\text{Unif}(0,1)$ sampled once and frozen (fig. 2). We generated a dataset of observations from this teacher (see supp. E). We trained $400$ students with $6-15$ states on the same teacher data using gradient-based methods (see supplement A). All students had high co-smoothing scores, with some variance. Consistent with our hypothesis, the ability to decode the teacher from the student varied little, and was highly correlated to the co-smoothing score (FIG 1C). In contrast, the ability to decode the student from the teacher displayed a large variability, and little correlation to the co-smoothing score (FIG 1D). See supplement B for details of the regression.

What is it about a student model, that produces good co-smoothing with the wrong latents? We consider the HMM transition matrix for the teacher and two exemplar students – named "Good" and "bad" – and visualise their states and transition probabilities using graphs in figure 2 (and marked in FIG 1CD). The teacher is a cycle of 4 steps. The good student has such a cycle ($1\rightarrow 7\rightarrow 0\rightarrow 4$), and the initial distribution $\pi$ is only on that cycle, rendering the other states irrelevant. In contrast, the bad student also has this cycle ($6\rightarrow 0\rightarrow 1\rightarrow 7$), but the $\pi$ distribution is not consistent with the cycle, and there is an outgoing edge from the cycle ($6\rightarrow 2$). Note that this does not interfere with co-smoothing, because the teacher itself is noisy. Thus, occasionally, there will be trials where the teacher will not have an exact period of 4 states. In such trials, the bad model will infer the irrelevant states instead of jumping to another relevant state, as in the teacher model.

Figure: Figure 2: Graph visualisations of HMMs: the ground-truth or teacher model along with two representative extreme student models. All three models have high co-smoothing (low student$\rightarrow$teacher decoding error). The students differ in their teacher$\rightarrow$student decoding error (Fig. 1C,D). Node-colors represent initial state probabilities $\pi_{m}$ (bright is high probability) and edge-width and opacity denote transition probabilities $A_{m,l}$. Edges with values below $0.02$ are removed for visualisation. Note the ($1\rightarrow 7\rightarrow 0\rightarrow 4$) cycle of the good student, and the ($6\rightarrow 0\rightarrow 1\rightarrow 7$) cycle in the bad student. They differ in $\pi$, and the latter has an outgoing edge ($6\rightarrow 2$), with $A_{6,2}=0.08$, $A_{6,0}=0.89$.
Refer to caption: x2.png

## 5 Few-shot prediction selects better models

Because our objective is to obtain latent models that are close to the ground truth, the co-smoothing prediction scores described above are not satisfactory. Can we devise a new prediction score that will be correlated with ground truth similarity? The advantage of prediction benchmarks is that they can be optimized, and serve as a common language for the community as a whole to produce better algorithms [].

We suggest few-shot co-smoothing as a prediction score. Few-shot co-smoothing is a modification of the standard co-smoothing procedure (Fig. 1A, bottom arm): the data for held-out neurons is only made available for a limited number of $k$ trials (see Fig. 3, right). Given these $k$ trials, a function $g^{\prime}$ is estimated to map the latents to the held-out neurons. We highlight the following conceptual difference: standard co-smoothing does not require a defined encoding ($f$), latent ($z$), and decoding stage ($g$). It evaluates $g\circ f$ in an end-to-end manner. In contrast, our few-shot variation demands this distinction and explores the effects of estimating $g^{\prime}$ with lower statistical power than $g$, to uncover other properties of the latent $z$. After the estimation ‘training’ of $g^{\prime}$, we make rate predictions $R^{\text{out}}_{\text{test}}$ for held-out neurons on the test set inputs $X^{\text{in}}_{\text{test}}$ using $g^{\prime}\circ f$ instead of $g\circ f$ and compute the few-shot co-smoothing score $\mathcal{Q}(R^{\text{out}}_{\text{test}},X^{\text{out}}_{\text{test}})$ with (3).

Figure: Figure 3: Neural data $X$ is partitioned for cross-validating Latent Variable Models. Data accessible during training is in Orange, and data used for prediction is in green. Co-smoothing (center) and few-shot co-smoothing (right).
Refer to caption: x3.png

In order to evaluate both co-smoothing and few-shot co-smoothing simultaneously for each model we partition the set of neurons into three sets of size $N^{\text{in}}$, $N^{\text{out}}$, $N^{\text{$k$-out}}$ each. The $k$-out neurons kept aside until after the training of $f$ is complete, and used instead of the held-out neurons for training and testing $g^{\prime}$.

This procedure may be repeated several times independently on i.i.d sets of $k$ trials, giving several i.i.d estimates of $g^{\prime}$. For small $k$, $\mathcal{Q}$ tends to be highly variable across instances of $k$-shot estimates. Thus we compute and report the average score across several instances.

In the case of bernoulli HMMs the maximum likelihood estimate of $g^{\prime}$ given a fixed $f$ and $k$ trials has a closed form:

$$ $\displaystyle\hat{B}_{m,n}$ $\displaystyle=\frac{\sum_{i\in\text{$k$-shot trials}}\sum_{t=1}^{T}\mathbb{I}[ x^{(i)}_{t,n}=1]\xi^{(i)}_{t,m}}{\sum_{i^{\prime}\in\text{$k$-shot trials}} \sum_{t^{\prime}=1}^{T}\xi^{(i^{\prime})}_{t^{\prime},m}}$ $\displaystyle\forall\ 1\leq m\leq M\text{ and }n\in\text{$k$-out neurons}$ (11) $$

To show the utility of the newly-proposed prediction score, we return to the same HMM students from figure 1. For each student, we evaluate the average $k$-shot smoothing. First, we see that it provides new information on the models, as it is not correlated with co-smoothing (Fig. 4A). We are only interested in models that have good co-smoothing, and thus select the students with the highest co-smoothing, those satisfying $Q_{\text{student}}>Q_{\text{teacher}}-\epsilon$, choosing $\epsilon=10^{-3}$. For these students, we see that despite having very similar co-smoothing scores, their k-shot scores are highly correlated with their distance from the ground truth (as measured by decoding of student states from teacher) Fig. 4B.

Figure: Figure 4: Few-shot prediction selects better models. A. Models with high co-smoothing have highly variable $6$-shot co-smoothing and uncorrelated to co-smoothing. B. For the set of student with high co-smoothing $Q_{\text{student}}>Q_{\text{teacher}}-10^{-3}$, $6$-shot co-smoothing to held-out neurons is negatively correlated with decoding error from teacher-to-student. Following fig. 4C,D dashed lines represent the ground truth, green and red arrows represent “Good” and “Bad” models (Fig. 2).
Refer to caption: x4.png

## 6 Why does few-shot work?

The example HMM students of fig. 2 can help us understand why few-shot prediction identifies good models. The students differ in that the bad student has more than one state corresponding to the same teacher state. Because these states provide the same output, this feature does not hurt co-smoothing. In the few-shot setting, however, the output of all states needs to be estimated using a limited amount of data. Thus the information from the same amount of observations has to be distributed across more states. This data efficiency argument can be made more precise.

Consider a student-teacher scenario as in section 4. We let $T=2$ and a stationary teacher $z^{(i)}_{1}=z^{(i)}_{2}=m$. Now consider two examples of inferred students. To ensure a fair comparison, both must have two latent states. In the good student, $\xi$, these two states statistically do not depend on time, and therefore it does not have extraneous dynamics. In contrast, the bad student, $\mu$, uses one state for the first time step, and the other for the second time step. A particular example of such students is:

$$ $\displaystyle\xi_{t}=\begin{bmatrix}0.5&0.5\end{bmatrix}^{T}\ t\in\{1,2\}$ (12) $\displaystyle\begin{array}[]{ccc}\mu_{t=1}=\begin{bmatrix}1&0\end{bmatrix}^{T} &&\mu_{t=2}=\begin{bmatrix}0&1\end{bmatrix}^{T}\end{array}$ (14) $$

where each vector corresponds to the two states, and we only consider two time steps $t=1,2$.

We can now evaluate the maximum likelihood estimator of the emission matrix from $K$ trials for both students, using equation (11). We consider a single neuron, and thus omit $n$. Because both states play the same role, we write the $m=1$ case:

$$ $\begin{array}[]{ccc}\hat{B}_{1}(\xi)=\frac{0.5(C_{1}+C_{2})}{0.5KT}&&\hat{B}_{ 1}(\mu)=\frac{C_{1}}{KT}\end{array}$ (15) $$

where $C_{t}$ is the number of times $x$ occurs at time $t$ in $K$ trials.

The average of both quantities is the same, but the good student averages over more Bernoulli variables, and hence has a smaller variance. We show in the supplement that this larger variability translates to lower performance. Overall we that every time that a student has an extra state instead of reusing existing states, this costs the estimator more variance. In the appendix, we show how a more general setting can be approximated.

## 7 SOTA LVMs on neural data

In section 4 we showed that models with near perfect co-smoothing may possess latents with extraneous dynamics. We established this in a synthetic student-teacher setting with simple HMM models.

To show the applicability in more realistic scenarios, we trained several models from two SOTA architectures on mc_maze_20 consisting of neural activity recorded from monkeys performing a maze solving task [], curated by []. The 20 indicates that spikes were binned into $20ms$ time bins. We evaluate co-smoothing on a test set of trials and define the set of models with the best co-smoothing (supp. D and E).

An integral part of lfads-torch and STNDT frameworks is the random hyperparameter sweep which generates several candidate solutions to the optimization problem (5).

With each model $f_{u}$, we infer latents evaluated over a fixed set of test trials $X^{\text{test}}=\{\boldsymbol{x}^{(i)}\}_{i\in\text{test}}$ $Z_{u}:=\{(z_{t}^{(i)})_{u}\}_{i\in\text{test},1\leq t\leq T}$ by (1).

In the HMM case, we had ground truth that enabled us to directly compare the student latent to that of the teacher. With real neural data we do not have this privilege. To nevertheless reveal the presence or absence of extraneous dynamics, we instead compare the models to each other. The key idea is that all models contain the teacher latent, because they have good co-smoothing. One can then imagine that each student contains a selection of several extraneous features. The best student is the one containing the least such features, which would imply that all other students can decode its latents, while it cannot decode theirs. We therefore use cross-decoding among student models as a proxy to the ground truth.

Instead of decoding from each student to the teacher as in section 4 we perform a cross-decoding from latents of model $u$ to model $v$ for every pair of models $u$ and $v$ using linear regression (supp. D). We then evaluate a $R^{2}$ score for each mapping. In Fig 5 the results are visualised by a $U\times U$ matrix with entries $(R^{2})_{u,v}$.

We hypothesize that the latents $z_{u}$ contain the information necessary to output good rate predictions $r$ that match the outputs plus the arbitrary extraneous dynamics. This former components must be shared across all models, whereas the latter could be unique in each model – or less likely to be consistent in the population. The ideal model $v^{*}$ would have no extraneous dynamics therefore, all the other models should be able to decode to it, i.e $(R^{2})_{u,v^{*}}=1\ \forall\ u$. Provided a large and diverse population $\mathcal{F}$ only the ‘pure’ ground truth would satisfy this condition. To evaluate how close is a model $v$ to the ideal $v^{*}$ we propose a simple metric: the column average $\mu_{v}=\frac{1}{U}\sum_{u=1}^{U}1-(R^{2})_{u,v}$. $\mu_{v}$ will serve as proxy for the distance to ground-truth, analogous to the teacher$\rightarrow$student decoding error in figure 4.

Figure: Figure 5: Cross-decoding as a proxy for distance to the ground truth in near-SOTA models. 200 LFADS models (left) and 120 STNDT models (right) were trained on the mc-maze data set then selected for high-cosmoothing (supp. D). The latents of each pair of models were decoded from one another, and the decoding error is shown in the matrices. Good models are expected to be decoded from all other models, and hence have low values in their corresponding columns. Bottom left: Trajectories of two LFADS models, with the lowest (left, best model) and highest (right) column averaged $1-R^{2}$ projected onto their leading two principal components. Each trace is the trajectory for a single trial, starting at a green dot and ending at a red dot. Bottom right Same for STNDT.
Refer to caption: x5.png

Having developed a proxy for the ground-truth we can now correlate it with the few-shot regression to held-out neurons. Fig 6 shows a negative correlation for both architectures, similar to the HMM examples above. As an illustration of the latents of different models, fig 6 shows the PCA projection of several trials from two different models. Both have high co-smoothing scores (SCORE1SCORE2), but differ in their cross-decoding column average $\mu$. Note the somewhat smoother trajectories in the model with higher few-shot score.

Figure: Figure 6: Few-shot scores correlate with the proxy of distance to the ground truth. Two models (LFADS top, STNDT bottom) were trained on neural recordings from monkeys performing a maze task, the mc_maze_20 benchmark[, ]. Distance to ground truth was approximated by the cross-decoding column average $\mu$ (Fig. 5). Few-shot ($K=128$) co-smoothing scores (left) negatively correlate with $\mu$, while regular co-smoothing (right) does not.
Refer to caption: x6.png

## 8 Discussion

Latent variable models aim to infer the underlying latents using observations of a target system. We showed that co-smoothing, a common prediction measure of the goodness of such models cannot discriminate between certain classes of latents. In particular, extraneous dynamics can be invisible to such a measure.

We suggest a new prediction measure: few-shot co-smoothing. Instead of directly regressing from held-in to held-out neurons as is done to evaluate co-smoothing, we distinguish the encoder and the decoder. To evaluate the trained model we substitute the decoder with a new decoder estimated using only a ‘few’ ($K$) number of trials. The rate predictions provided by the few-shot decoder are evaluated the same as in standard co-smoothing. Using synthetic datasets and HMM models, we show numerically and analytically that this measure correlated with the distance of model latents to the ground truth.

We demonstrate the applicability of this measure to real world neural data sets, with SOTA architectures. This required developing a new proxy to ground truth – cross decoding. For each pair of SOTA models we obtained, we performed a linear regression across model latents, provided identical input data. Models with extraneous dynamics showed up as a bad target latent on average, and vice versa. Finally we show that these two characterisations of extraneous dynamics are correlated.

While we believe the combination of results in the toy setting and SOTA put forth a strong arguement, we address here a few limitations of our work. Firstly, our SOTA results use only one of the datasets in the benchmark suite []. With regard to the few-shot regression, while the bernoulli HMM scenario has a closed form solution: the maximum likelihood estimate, the poisson GLM regression is optimised iteratively and is sensitive to the l2 hyperparameter alpha. In our results we select $K$ and $\alpha$ that distinguish models in our candidate model sets giving moderate/high few-shot scores for some models and low scores to others. This is an empirical choice that must be made for each dataset and model-set. The few-shot training of $g^{\prime}$ is computationally inexpensive and may be thus can evaluated over a range of values to find the ideal ones.

Overall, our work advances latent dynamics inference in general and prediction frameworks in particular. By exposing a failure mode of standard prediction metrics, we can guide the design of inference algorithms that take this into account. Furthermore, the few-shot prediction can be incorporated into existing benchmarks and help guide the community to build models that are closer to the desired goal of uncovering latent dynamics in the brain.

## Appendix A Hidden Markov Model training

HMMs are traditionally trained with expectation maximisation (EM), but they can also be trained using gradient-based methods. We focus here on the latter, using an implementation of HMMs in dynamax – a library of differentiable state-space models built with jax.

We seek HMM parameters $\theta:=(A,B^{[\text{in},\text{out}]},\pi)$ that minimise the negative log-likelihood loss $L$ of the held-in and held-out neurons in the train trials:

$$ $\displaystyle L(\theta;X^{[\text{in},\text{out}]}_{\text{train}})$ $\displaystyle=-\log p(X^{[\text{in},\text{out}]}_{\text{train}};\theta)$ (16) $\displaystyle=\sum_{i\in\text{train}}-\log p\left(\left(x_{1:T}^{[\text{in}, \text{out}]}\right)^{(i)};\theta\right)$ (17) $$

To find the minimum we do full-batch gradient descent on $L$, using dynamax together with the Adam optimiser.[]

## Appendix B Decoding across HMM latents: fitting and evaluation

Consider $\boldsymbol{\xi}^{(i)}$ and $\boldsymbol{\hat{\xi}}^{(i)}$ the latents inferred by the $M$-state teacher and $\hat{M}$-state student HMMs respectively from the data $(x^{\text{in}}_{1:T})^{(i)}$ using (9).

We now perform a multinomial regression from individual time-steps $\boldsymbol{\xi}^{(i)}_{t}$ to $\boldsymbol{\hat{\xi}}^{(i)}_{t}$ (and vice versa by reversing the hat notation) using sklearn.linear_model.LogisticRegression.

$$ $\displaystyle\boldsymbol{\hat{p}^{(i)}_{t}}=\sigma\left(h\left(\boldsymbol{\xi }^{(i)}_{t}\right)W\boldsymbol{+}\boldsymbol{b}\right)$ (18) $$

where $W\in\mathbb{R}^{\hat{M}\times M}$, $\boldsymbol{b}\in\mathbb{R}^{\hat{M}}$ $\sigma$ is the softmax. During training we sample states from the target PMFs $\hat{z}_{t}^{(i)}\sim\boldsymbol{\hat{\xi}}^{(i)}_{t}$ to a more well know problem scenario: classification. We optimize $W$ and $b$ to minimise a cross-entropy loss to the target $\hat{z}_{t}^{(i)}$.

To evaluate decoding error, we evaluate the average Kullback-Leibler divergence $D_{KL}$ between target and predicted distributions:

$$ $\displaystyle\frac{1}{S^{\text{test}}T}\sum_{i\in\text{test}}\sum_{t=1}^{T}D_{ KL}(\boldsymbol{\hat{p}}^{(i)}_{t},\boldsymbol{\hat{\xi}}^{(i)}_{t})$ (19) $$

where $D_{KL}$ computed with a scipy.special.rel_entr.

## Appendix C Cramer Rao approximation for score

In the main text, we showed how a good student outperforms a bad one on few-shot learning. This was done in a specific case of two time points, a single teacher state and two student states. To link few-shot score and representations in more general settings, we look at the drop in average few-shot score for finite $k$: $\mathbb{E}\mathcal{L}(\hat{B}^{k})-\mathcal{L}(\hat{B}^{\infty})$. We use $\hat{B}^{\infty}$, the limit when$k\to\infty$, as a proxy for regular co-smoothing. $\hat{B}^{k}$ is the maximum likelihood estimator (MLE) of:

$$ $\displaystyle L_{K}(B)$ $\displaystyle=\sum_{t=1}^{T}\sum_{i=1}^{k}\log f(\tilde{x}_{t}^{(i)};r_{t}^{(i )}(B))$ (20) $\displaystyle=\sum_{t=1}^{T}\sum_{i=1}^{k}\log\begin{cases}r_{t}^{(i)}(B)& \text{ if }\tilde{x}_{t}^{(i)}=1\\ 1-r_{t}^{(i)}(B)&\text{ if }\tilde{x}_{t}^{(i)}=0\end{cases}$ (21) $\displaystyle=\sum_{t=1}^{T}\sum_{i=1}^{k}\log\begin{cases}\sum_{m}B_{m}\xi^{( i)}_{t,m}&\text{ if }\tilde{x}_{t}^{(i)}=1\\ 1-\sum_{m}B_{m}\xi^{(i)}_{t,m}&\text{ if }\tilde{x}_{t}^{(i)}=0\end{cases}$ using ( 10 ) (22) $$

In the main text, we showed that the good student has a less variable $\hat{B}$. To link this with few-shot performance, we compute the sensitivity of the log-likelihood score to variations in $B^{k}$.

$$ $\displaystyle\mathbb{E}_{\hat{B}_{K}}L(\hat{B}_{K})$ $\displaystyle=\mathbb{E}_{\hat{B}_{K}}\left[L(B_{\infty})+\frac{1}{2}(\hat{B}_ {K}-B_{\infty})^{T}\frac{\partial^{2}L}{\partial B^{2}}\bigg{|}_{B_{\infty}}( \hat{B}_{K}-B_{\infty})+\dots\right]$ (23) $\displaystyle=L(B_{\infty})+\mathbb{E}_{\hat{B}_{K}}\frac{1}{2}(\hat{B}_{K}-B_ {\infty})^{T}H(B_{\infty})(\hat{B}_{K}-B_{\infty})+\dots$ (24) $\displaystyle=L(B_{\infty})+\frac{1}{2}\text{Tr}[\Sigma H]+\dots$ (25) $$

where $\Sigma:=\text{Cov}[\hat{B}_{k}]$. Using the Cramer-Rao bound, we can estimate this covariance:

$$ $\displaystyle\text{Cov}(\hat{B}^{k})$ $\displaystyle\geq I^{-1}(\hat{B}^{k})|_{\hat{B}^{k}=B^{*}}$ (26) $$

where $I$ is the Fisher Information matrix:

$$ $\displaystyle I_{mm}$ $\displaystyle:=\mathbb{E}\left[\frac{\partial L_{K}}{\partial B_{m}}\frac{ \partial L_{K}}{\partial B_{m}}\right]$ (27) $\displaystyle=\mathbb{E}\sum_{i=1}^{k}\sum_{j=1}^{k}\sum_{s=1}^{T}\sum_{t=1}^{ T}\frac{(2\tilde{x}^{(i)}_{s}-1)(2\tilde{x}^{(j)}_{t}-1)}{f(\tilde{x}_{s}^{(i) };r_{s}^{(i)})f(\tilde{x}_{t}^{(j)};r_{t}^{(j)})}\xi^{(i)}_{s,m}\xi^{(j)}_{t,m}$ (28) $$

using the tilde on $\tilde{x}$ to denote that these are the target of estimate.

From this, we obtain

$$ $\displaystyle\mathbb{E}_{\hat{B}_{K}}L(\hat{B}_{K})$ $\displaystyle\geq L(B_{\infty})+\frac{1}{2}\text{Tr}[I(B_{\infty})^{-1}H]+\dots$ (29) $$

Finally we have

$$ $\displaystyle\mathbb{E}\mathcal{L}(\hat{B}^{k})-\mathcal{L}(\hat{B}^{\infty}) \approx\frac{1}{2}\sum_{m}\frac{H_{mm}}{I_{mm}}$ (30) $$

where

$$ $\displaystyle H_{mm}:=\mathbb{E}\frac{\partial^{2}L}{\partial B_{m}\partial B_ {m}}\bigg{|}_{B=\hat{B}^{\infty}}=\mathbb{-}\sum_{i}\sum_{t=1}^{T}\frac{\left( \hat{\xi}^{(i)}_{t,m}\right)^{2}}{f(x_{t}^{(i)};r_{t}^{(i)})^{2}}$ (31) $$

are the diagonal terms of the Hessian of the log-likelihood with respect to $\hat{B}$.

## Appendix D Analysis of SOTA models

We denote the set of high co-smoothing models as those satisfying $Q_{\text{model}}>Q_{\text{best model}}-\epsilon$, choosing $\epsilon=5\times 10^{-3}$ for lfads-torch and $\epsilon=1.3\times 10^{-2}$ for STNDT. $\mathcal{F}:=\{(f_{u},g_{u})\}_{u=1}^{U}$, the encoders and decoders respectively. Note that both architectures are deep neural networks given by the composition $g\circ f$, and the choice of intermediate layer whose activity is deemed the ‘latent’ $z$ is arbitrary. Here we consider $g$ the last ’read-out’ layer and $f$ to represent all the layers up-to $g$. $g$ takees the form of Poisson Generalised Linear Model, a natural and simple choice for the few-shot version $g^{\prime}$.

This choice of $f$, suggests a natural choice $g^{\prime}$ to be a Poisson Generalised Linear Model (GLM). We use sklearn.linear_model.PoissonRegressor. The poisson regressor has a hyperparameter alpha, the amount of l2 regularisation. For the results in the main text we select $\alpha=10^{-3}$.

To perform few-shot co-smoothing, we partition the train data into several subsets of $k$ trials. To implement this in a standarised way, we build upon the nlb_tools library (supp. F). This way we ensure that all models are trained and tested on identitical partitions.

We perform a cross-decoding from $Z_{u}$ to $Z_{v}$ for every pair of models $u$ and $v$ using a linear mapping $h(\boldsymbol{z}):=W\boldsymbol{z}+\boldsymbol{b}$ implemented with sklearn.linear_model.LinearRegression:

$$ $\displaystyle\left(\hat{z}^{(i)}_{t}\right)_{v}=h_{v\leftarrow u}\left(\left(z ^{(i)}_{t}\right)_{u}\right)$ (32) $$

minimising a mean squared error loss. We then evaluate a $R^{2}$ score (sklearn.metrics.r2_score) of $z_{v}$ and $\hat{z}_{v}$ for each mapping. Results are accumulated into a matrix $(R^{2})_{u,v}$.

## Appendix E Dimensions of datasets

We analyse two datasets in this work. A synthetic dataset generated by with an HMM (ground-truth in figure 2) and the mc_maze_20 data set from the NLB suite [, ]. To evaluate $k$-shot on the existing SOTA methods while maintaining the NLB evaluations, we conserved the forward-prediction aspect. During model training, models make rate predictions for $T^{\text{fp}}$ future time bins in each trial, i.e (1) and (2) are evaluated for $1\leq t\leq T^{\text{fp}}$ while input remains as $\boldsymbol{x}^{\text{in}}_{1:T}$. Although we do not discuss the forward-prediction metric in our work, we note that the SOTA models receive gradients from this portion of the data.

In mc_maze_20 we reuse held-out neurons as $k$-out neurons. We do this to preserve NLB evaluation metrics on the SOTA models, as opposed to re-partitioning the dataset resulting in different scores from previous works. This way existing co-smoothing scores are preserved and $k$-shot co-smoothing scores can be directly compared to the original co-smoothing scores. The downside is that we are not testing the few-shot on ‘novel’ neurons. Our numerical results (figure 6) show that our concept still applies.

**Table 1: Dimensions of real and synthetic datasets. Number of train and test trials $S^{\text{train}}$, $S^{\text{test}}$, time-bins per trial for co-smoothing $T$, and forward-prediction $T^{\text{fp}}$, held-in, held-out and $k$-out neurons $N^{\text{in}}$, $N^{\text{out}}$, $N^{\text{$k$-out}}$. *In mc_maze_20 we use the same set of neurons for $N^{\text{out}}$ and $N^{\text{$k$-out}}$.**
| Dataset | $S^{\text{train}}$ | $S^{\text{test}}$ | $T$ | $T^{\text{fp}}$ | $N^{\text{in}}$ | $N^{\text{out}}$ | $N^{\text{$k$-out}}$ |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Synthetic HMM | $2000$ | $100$ | $10$ | – | $20$ | $50$ | $50$ |
| NLB mc_maze_20 [, ] | $1721$ | $574$ | $35$ | $10$ | $127$ | $55$ | $55^{*}$ |

## Appendix F Code repositories

The experiments done in this work are largely based on code repositories from previous works. Table 2 provides links to the code repositories used or developed in this work.

**Table 2: Summary of key repositories used in this paper**
| Repository | Forked from | Citations |
| --- | --- | --- |
| [https://github.com/KabirDabholkar/nlb_tools_fewshot](https://github.com/KabirDabholkar/nlb_tools_fewshot) | [https://github.com/neurallatents/nlb_tools](https://github.com/neurallatents/nlb_tools) | [] |
| [https://github.com/KabirDabholkar/STNDT_fewshot](https://github.com/KabirDabholkar/STNDT_fewshot) | [https://github.com/trungle93/STNDT](https://github.com/trungle93/STNDT) | [, , , , ] |
| [https://github.com/KabirDabholkar/lfads-torch](https://github.com/KabirDabholkar/lfads-torch) | [https://github.com/arsedler9/lfads-torch](https://github.com/arsedler9/lfads-torch) | [, , ] |
| [https://github.com/KabirDabholkar/hmm_analysis](https://github.com/KabirDabholkar/hmm_analysis) |  |  |
| [https://github.com/probml/dynamax](https://github.com/probml/dynamax) |  |  |

## Peer Reviews

confidence: 2
contribution: 2
initial_rating: 3
presentation: 2
questions: 1. L58: LVM is introduced without explanation, and the explanation is finally given in L113.

2. L118: "Data may be held-out in time, e.g forward-prediction or in space, co-smoothing (Pei et al., 2021)." Here, is "held-out in time" the same as "forward-prediction" of signal in the latent space or signal space? Besides, is "Data held-out in space" the same as "co-smoothing"? But what does "Data held-out in space" exactly mean?

3. L149: why is equation (5) "without direct access of $X_{\text {test }}^{\text {out }}$"? From the definition in (3) and (4), I feel $X_{\text {test }}^{\text {out }}$ is required.

4. In section 4, how to validate that the specific setting of HMM is a suitable synthetic data model that can reveal features of real neural signals?

5. In section 5, how to choose "k"? What about selecting "k" for real data, is there a principled rule?

I am willing to hear from authors and other reviewers in addressing my questions and concerns. I will change the score if others' comments solve my questions and convince me about the quality and importance of this work.
soundness: 2
strengths: 1. The paper has a high-level storyline that is easy to follow: from the limitation of co-smooth, proposing new few-shot co-smoothing, reasons for few-shot co-smoothing to work, and how to adapt few-shot co-smoothing to real neural signals.
2. The paper covers both empirical analysis of existing methods, practical improvements based on the existing method, and mathematical interpretations of the proposed improvement.
3. The paper provides an honest discussion of limitations.
summary: The paper focuses on the evaluation method of neural latent-variable models (LVM). Neural LVM aims to learn representations of neural dynamics in a latent space. This space should be able to represent the ground truth hidden representation space. However, due to the lack of ground truth latent in neural signal data, people propose to evaluate the quality of learned latent via co-smoothing, which jointly estimates latent variables and predicts observations along held-out channels as evaluation metrics. With HMM synthetic data, the paper empirically shows that good co-smoothing performance does not ensure good latent, since the model can learn redundant information to have good prediction results. Therefore, the paper instead proposes the few-shot co-smoothing, which better separates good LVM from bad LVM. The paper further provides mathematical insights based on the prediction variance of co-smoothing and few-shot co-smoothing methods. Lastly, the paper adapts co-smoothing with cross-decoding to real data and shows good "accuracy".
weaknesses: 1. Writing of limited works is a bit messy: too many paragraphs, mixing related works with the introduction of the work's methodology and contribution. It would be clearer to provide clean related works as a background and only include a short emphasis on the uniqueness of this paper.

2. The writing in other parts is also a bit confusing and the paper seems to be finished in a rush, missing ","s in several positions: L118 after "forward-prediction", L122 after "During evaluation", L125 after "full time-window", L142 after "log-likelihood", etc, ... Let me know if I misunderstood these parts.

3. The paper lacks an explanation for choosing HHM as a synthetic data model to approximate real neural signals.

4. The paper lacks clarity and technical details, see Questions parts.

---

confidence: 3
contribution: 3
initial_rating: 5
presentation: 4
questions: * My main questions revolve around evaluations in section 7 and the cross-decoding metric used as ground truth:
   * Can the authors provide the performance of the cross-decoding score in the synthetic setup where the latent variables are known? can one use it to choose a model with latent representation closest to the ground truth?
   * In the general case and assuming no computational constraints, is this metric supposed to be superior to scores from few-shot cosmoothing and can this be verified empirically?
   * Even if the assumption on cross-decoding as a proxy to the ground truth is correct - selecting from a pool of models that differ by hyperparameters can be highly dependent. Is it possible to select jointly from a pool of different architectures? .e.g, LFADS, STDNT.

* In section 3 can you clarify which notation (bold/regular) refers to vectors/tensors/scalars? I could not understand from the text.
soundness: 3
strengths: * The paper is well organized and each section has a clear and consice statement.

* The teacher-student setup provides a convincing example of the problem with co-smoothing scores.

* The suggested solution of few-shot co-smoothing is simple and seems to be reasonably effective in the suggested setups.
summary: The paper studies the limitations of the commonly used co-smoothing prediction framework applied to latent variable models predicting underlying dynamics, specifically in the context of neural activity modeling. The authors provide synthetic teacher-student setup demonstrating how good performance in terms of co-smoothing does not imply learning the ground-truth latents. In the same setup, an additional prediction score termed few-shot co-smoothing is suggested leading to better correlation with true latents for models with high co-smoothing. In a real-world setup, the authors suggest a proxy metric assessing the similarity of a model’s latents to an unknown teacher, and show that their suggested score correlates with it.

I am relatively unfamiliar with the subject of the paper and prior work but found the presentation and ideas easy to follow.
weaknesses: * Evaluations in section 7 rely on a metric, cross-decoding error, defined by the authors themselves and argued for in the text. No prior work pointing to the validity of the score is provided, see further questions below.

* Since correlation in section 7 is not perfect, and due to the above, further empirical verification of the method, with either additional datasets and/or models, is missing.

---

confidence: 2
contribution: 2
initial_rating: 3
presentation: 3
questions: See above.
soundness: 3
strengths: 1. The writing of this paper is good, and the illustration figures are clear. 

2. The example provided in Section 4 and corresponding observations are interesting.
summary: This paper focuses on latent variable models (LVMs) for inferring neural activities (neural spikings). Conventional approaches in this field rely on co-smoothing, which jointly estimates the latent variables and predicts observations. This work hypothesizes that, accurate prediction via co-smoothing ensures that LVMs captures the true latent structure, while the reverse may not be true. This claim is justified using an example of discrete-time Hidden Markov Model (HMM). To overcome the aforementioned limitation, this paper proposes an alternative metric called few-shot co-smoothing, which reduces the number of data (trials) used for regression. Experiments on real neural data are conducted with two SOTA methods to validate the effectiveness of the advocated metric.
weaknesses: Major concerns:

1. The proposed few-shot co-smoothing is based on empirical observations on a simple HMM model. It would be highly beneficial if theoretical results regarding extending this analysis to more generic cases can be provided. In the current manuscript, it is hard to conclude for which family of problems and under what conditions the proposed metric can be effective. It is unknown whether and how few-shot co-smoothing will also help when the relationship between the latent variables and observations becomes complicated. 

2. The relationship between the claim, "good prediction guarantees that the true latents are contained within the inferred ones, but not vice versa", and the example provided in Figure 1.DE is unclear. In my understanding, both $\mathcal{D}_{\rm S\rightarrow T}$ and $\mathcal{D} _{\rm T\rightarrow S}$ are utilized to assess whether "the true latents ($z_T$) are contained within the inferred ones ($z_S$)". The paper has not clearly explained their differences, and what each metric indicates. Furthermore, Appendix B leverages multinomial regression to "align" two multinomial distributions and then measure their KL distance. Why is multinomial regression performed with a logistic model rather than other models? It is also possible that two distributions are close under certain mapping but logistic model fail to identify this mapping. To measure the distance between two distributions, there can be some other alternative approaches such as optimal transport. The rationale behind this design choice needs further clarification. 

3. Experiments are not sufficiently convincing due to the lack of quantitative comparisons. Section 7 showcases $\mathcal{D}_{\rm S\rightarrow T}$ and $\mathcal{D} _{\rm T\rightarrow S}$ when applying few-shot co-smoothing to SOTA LVMs. However, it remains unknown how mismatched the latents can be with co-smoothing, and how this will affect the prediction performance of the model. 

4. No supplementary files or codes are provided, making it hard to verify the reported results. 

Minor comments:
1. In line 111, "representing" should be "represents". 

2. In line 142, add a comma after "log-likelihood". 

3. "Figure 1B" in line 159 should be corrected to "Figure 1C". 

4. The cross references for equations are not in correct formats; "\eqref" should be used instead of "\ref". 

5. The abbreviations for "Figure" are not consistent throughout the paper. For instance, lines 234, 241, and 244 respectively adopts "fig.", "Fig.", and "FIG".

Meta-review: The paper critiques the commonly used co-smoothing evaluation metric for latent variable models (LVMs), demonstrating its failure to reliably infer true latent dynamics due to the inclusion of extraneous dynamics. The authors propose a new metric, few-shot co-smoothing, which evaluates performance over limited data trials, improving the selection of models with minimal extraneous dynamics. They validate this metric using a teacher-student setup with Hidden Markov Models (HMMs) and extend it to real-world neural data using cross-decoding to assess latent space similarity. Strengths include the clear identification of a critical issue in LVM evaluation, the introduction of an effective and simple metric, and validation through both theoretical analysis and empirical data. However, the work lacks sufficient theoretical generalization beyond HMMs, robust real-world validation, and comparisons to alternative evaluation methods. Additionally, the limited scope and unclear applicability of the few-shot metric in broader LVM settings raise concerns about its utility. While the paper provides interesting insights, these limitations undermine its contributions, leading to a borderline rejection recommendation with encouragement for further development and more comprehensive validation.

Decision: Reject