Words in Motion: Extracting Interpretable Control Vectors for Motion Transformers

## Abstract

Abstract Motion forecasting transforms sequences of past movements and environment context into future motion.
Recent methods rely on learned representations, resulting in hidden states that are difficult to interpret.
In this work, we use natural language to quantize motion features in a human-interpretable way, and measure the degree to which they are embedded in hidden states.
Our experiments reveal that hidden states of motion sequences are arranged with respect to our discrete sets of motion features.
Following these insights, we fit control vectors to motion features, which allow for controlling motion forecasts at inference.
Consequently, our method enables controlling transformer-based motion forecasting models with textual inputs, providing a unique interface to interact with and understand these models. 1 1 1 Our implementation is available at https://github.com/kit-mrt/future-motion .

## 1 Introduction

Accurate and interpretable motion forecasting of road users is essential for autonomous driving.
These objectives are often in conflict, since high accuracy frequently comes at the cost of lower interpretability.
This trade-off is primarily linked to the representational capacity of the underlying model: methods achieving higher accuracy tend to rely on the increased complexity of their underlying models.
This, in turn, renders them difficult to understand and interpret in terms of semantically meaningful patterns.

Deep learning models employ loss functions that encourage the clustering of data samples based on their patterns [].
Together with regularizers that prevent overfitting, clusters become more distinct over the course of training [, ].
Recent motion forecasting methods use transformer models [, , ].
They process features of past motion sequences (i.e., past positions, orientation, acceleration, speed) and environment context (i.e., map data and traffic light states), and transform them into future motion sequences.
Recent methods rely on learned representations of these features, resulting in hidden states that are difficult to interpret.

Natural language is a medium to describe semantically meaningful patterns.
Previous works have leveraged language models to enable text conditioned scenario generation and to decode essential textual information about the underlying model’s actions and reasoning (e.g., [, , ]).
However, analyzing learned representations with respect to human-interpretable classes of motion features remains an open challenge.

We use natural language to quantize motion features into discrete and interpretable sets.
During training, we use linear probing to measure the degree to which these features are embedded in hidden states.
Our experiments demonstrate that hidden states of motion sequences are arranged with respect to our discrete sets of motion features.
In this way, we introduce a novel method to interpret the hidden states of transformer-based motion forecasting models.
Building on these insights, we fit control vectors to opposing motion features which allow for controlling motion forecasts at inference.
As a result, our method enables controlling transformer-based motion forecasting models using textual inputs, offering a novel and intuitive interface for interacting with and understanding these models.

## 2 Related Work

The concept of linking learned representations to natural language has gained significant attention (e.g., [, , ]).

Conditioning.
Numerous works leverage natural language to guide or condition generative models [, , ].
Tan et al. [] generate dynamic traffic scenes that match user-specified descriptions expressed in natural language.

Prompting.
Some works use language as an interface to prompt driving models.
Kuo et al. [] generate linguistic descriptions of predicted trajectories during decoding, capturing essential information about future maneuvers and interactions.
More recent works employ large language models (LLMs) to analyze driving environments in a human-like manner, providing explanations of driving actions and the underlying reasoning [, , , ].
This offers a human-centric description of the driving environment and the model’s decision-making capabilities.
Although these works create a language interfaces to interact with the underlying model, in contrast to our work, they do not analyze the alignment of internal representations with human-interpretable motion features.

Enriching.
Another line of work aims to use LLMs generalization capacities to enrich the context embeddings and provide additional information to improve motion forecasting accuracy.
Zheng et al. [] demonstrate that by integrating this enriched context information into motion forecasting models.
Wang et al. [] use LLMs for data augmentation to improve out-of-distribution generalization.
Others use pre-trained LLMs for better generalization during decision-making [, , ].

Instructing.
Natural language can also serve as a powerful tool for issuing instructions.
Instructions are explicit commands given to a model to perform a specific task and differ from language conditioned generation [, , ].
The challenge in instructing lies in connecting the abstractions and generality of language to environment grounded actions.
Huang et al. [] demonstrate the potential of using instructions to guide task execution in simulation environments for autonomous driving.

## 3 Method

Figure: Figure 1: Words in Motion. We use natural language to quantize motion features in an interpretable way. (b) The corresponding direction, speed, and acceleration classes are highlighted in blue. (c) To reverse engineer motion forecasting models, we measure the degree to which these features are embedded in their hidden states $H_{i}$ with linear probes. Furthermore, we use our discrete motion features to fit control vectors $V_{i}$ that allow for controlling motion forecasts during inference.
Refer to caption: /html/2406.11624/assets/x1.png

We reverse engineer recent motion forecasting models by using natural language to quantize motion features into discrete sets (i.e., subclasses).
Textual descriptions naturally represent human-interpretable classes of motion features (e.g., for speed or direction).
In this work, we focus on coarse-grained direction (e.g. moving right in [Figure 1](#S3.F1)), speed (e.g. moderate speed in [Figure 1](#S3.F1)), acceleration classes (e.g. decelerating in [Figure 1](#S3.F1)) and fine-grained motion classes (complete text representation in [Figure 1](#S3.F1)).

### 3.1 Motion classification using natural language

We classify motion features using four primary classes of direction, following insights from [].
Initially, we determine these classes by the cumulative sum of difference in yaw angles.
Trajectories with a sum less than 15 ​ ° 15° degrees are classified as straight.
For sums exceeding this threshold, a positive value labels the motion as right, while a negative value, exceeding the threshold in absolute terms, indicates a left direction.
Additionally, we introduce a stationary class for stationary objects, where direction lacks semantic significance.
We further classify motion based on speed. Speeds between $25\text{\,}\mathrm{km}\text{\,}{\mathrm{h}}^{-1}$ and $50\text{\,}\mathrm{km}\text{\,}{\mathrm{h}}^{-1}$ are labeled moderate.
Speeds above this range are high, while those below are low.
Negative speeds are classified as backwards.
Lastly, we analyze acceleration by comparing the integral of speed over time to projected displacement with initial speed.
Trajectories are decelerating if under $0.9$ times, accelerating if over $1.1$ times the projected displacement, and constant otherwise.
All threshold values are selected by analyzing the distribution of the used dataset.
These direction, speed, and acceleration categories enable us to construct sentence descriptions, as illustrated in [Figure 1](#S3.F1).

### 3.2 Neural collapse as a metric of interpretability

A recent line of work [, , ] introduces the term neural collapse to describe a desirable learning behavior of deep neural networks for classification.(^2^22Neural collapse is not to be confused with representation collapse [, ], where learned representations across all classes collapse to redundant or trivial solutions (e.g., zero vectors).)
It refers to the phenomenon that learned top-layer representations form semantic clusters until these collapse to their means.
In addition, the cluster means transform progressively into equidistant vectors when centered around the global mean.
Therefore, neural collapse facilitates classification tasks and is considered a desirable learning behavior for both supervised [] and self-supervised learning [].

We propose to measure neural collapse as a metric of interpretability.
Specifically, we focus on interpreting hidden states (i.e., latent representations) of motion sequences in motion forecasting models.
Our approach is motivated by the assumption that hidden states of motion sequences should embed human-interpretable motion features.
Therefore, we propose to evaluate if hidden states can be classified as human-interpretable motion features by linear probes [] (i.e., learned linear classifiers).
Simple linear probes ensure that the abstract hidden states are closely related to interpretable semantics.(^3^33 Ben-Shaul et al. [] show that linear probing accuracy is consistent with the accuracy of nearest class center classifiers, which are typically used to measure neural collapse.)
Adapted to motion forecasting, we choose the aforementioned language-based motion features.
In summary, we use natural language to quantize motion features in an interpretable way, and measure the degree to which they are embedded in hidden states using linear probing.

### 3.3 Natural language-based control vectors

We transfer the concept of control vectors from natural language processing [, , ] to motion forecasting.
Control vectors ($V_{i}$ in [Figure 1](#S3.F1)) are sets of vectors, which are added to the hidden states of transformer models to modify their predictions, without fine-tuning or prompt engineering.
In general, control vectors represent the difference of hidden states from opposite concepts or features.
Therefore, adding temperature-scaled ($\tau$ in [Figure 1](#S3.F1)) control vectors to the hidden states of a given sample controls the strength of the corresponding features.
However, this requires that the hidden states are arranged in a well-structured latent space, where samples are clustered according to classes or features (e.g., a high degree of neural collapse, see [Section 3.2](#S3.SS2)).

We use our natural language-based classes of motion features (see [Section 3.1](#S3.SS1)) to build pairs of opposing features.
Specifically, we generate speed control vectors using the high and low speed classes, acceleration control vectors using the accelerating and decelerating classes, and direction control vectors using the left and right classes.
For each pair, we build a dataset and extract the corresponding hidden states $H_{i}^{[m]}$ of the module $m=2$.
Afterwards, we compute the element-wise difference between the hidden states of opposing classes.
We choose the hidden state of the last token per motion sequence (with $i=-1$) as it is closest to the start of the prediction.
Finally, we follow [] and use principal component analysis (PCA) with one component to reduce the computed differences to one scalar per hidden dimension.

## 4 Experimental Setup

### 4.1 Motion forecasting models

We study two recent transformer models for motion forecasting in self-driving vehicles.
Wayformer [] and RedMotion [] models employ attention-based scene encoders to learn agent-centric embeddings of past motion, map, and traffic light data.
To efficiently process long token sequences, Wayformer uses latent query attention [] for subsampling, RedMotion lowers memory requirements via local-attention [] and redundancy reduction.
For Wayformer, we use the implementation by [] and a custom hierarchical fusion setup with a modality-specific encoder for past motion and a shared encoder for environment context.
For RedMotion, we use the publicly available implementation.
Therefore, both models contain specific motion encoders with three modules, which we combine with linear probes and control vectors to interpret and control hidden states with human-interpretable motion features (see [Figure 1](#S3.F1)).

### 4.2 Datasets

The Argoverse 2 Forecasting Dataset [] consists of $250,000$ scenarios of urban or suburban driving, each covering a period of 11 seconds.
These scenarios include $5$ seconds of past and $6$ seconds of future data.
The dataset captures interactions along more than $2000\text{\,}\mathrm{km}$ of roadways in six geographically diverse cities.

The Waymo Open Motion Dataset [] consists of over $1.1$ million data points extracted from $103,000$ urban or suburban driving scenarios, each spanning $20$ seconds.
The state of the traffic agents includes attributes such as position, dimensions, speed, acceleration, orientation, and angular velocity.
Each data point captures $1$ second of past data followed by $8$ seconds of future data.

### 4.3 Metrics

Linear probing accuracy. To measure the degree to which interpretable motion features are embedded in latent representations, we use linear probing [].
In this approach, we train linear probes (i.e., linear classifiers detached from the overall gradient computation) on top of the hidden states ($H_{i}$ in [Figure 1](#S3.F1)).
During training, we track their accuracy in classifying our quantized motion features on validation sets.

Representation quality.
Following [], we use the mean of the standard deviation of the $\ell_{2}$-normalized embedding to measure representation collapse.
Representation collapse refers to an undesirable learning behaviour where learned embeddings collapse into redundant or trivial representations.
Redundant representations have a standard deviation close to zero.
In a way, representing the opposite of neural collapse.
As shown in [], rich representations have a standard deviation close to $1/\sqrt{\text{dim}}$, where dim is the hidden dimension.

### 4.4 Training details and hyperparameters

We add linear probes for our quantized motion features to each hidden state of both models ($H_{i}^{[m]}$ in [Figure 1](#S3.F1), where $m\in\{0,1,2\}$ is the module number and $i$ is the token index).
These classifiers are learned during training using regular cross-entropy loss to classify our fine-grained motion, and coarse-grained speed, acceleration and direction classes from hidden states.
We decouple this objective from the overall gradient computation.
Therefore, these classifiers do not contribute to the alignment of the latent representations, but exclusively measure the corresponding neural collapse into interpretable clusters.

As the main loss, we use a common combination loss terms for motion forecasting.
Following [], we use the unweighted sum of the negative log-likelihood loss for positions, the cosine loss for the heading angle, and the Huber loss for velocities as motion loss.

We provide Wayformer models with the nearest $512$ map polylines and RedMotion models with the nearest 128 map polylines, and both models with a maximum of $48$ surrounding traffic agents as environment context.
For the Argoverse 2 Forecasting dataset, we use past motion sequences with $50$ timesteps (representing $5\text{\,}\mathrm{s}$) as input.
For the Waymo Open Motion dataset, we use past motion sequences with $11$ steps (representing $1.1\text{\,}\mathrm{s}$) as input.
We use AdamW [] in its default configuration as optimizer and set the initial learning rate to $0.0002$.
All models are configured to forecast $k=6$ motion modes per agent.
As post-processing, we do not perform trajectory aggregation, but follow [] and modify only the predicted confidences of redundant forecasts.

## 5 Results

Figure: (a) Acceleration

[Figure 2](#S5.F2) presents the distribution of motion subclasses across the datasets.
The Waymo dataset predominantly features vehicles with constant acceleration (65%) and traveling straight (49%), while the Argoverse 2 dataset showcases a higher proportion of accelerating instances (52%).
Both datasets predominantly capture low-speed scenarios, with 62% of Waymo instances and 53% of Argoverse 2 instances falling into this category.
Furthermore, a notable difference lies in the proportion of stationary vehicles, with Waymo exhibiting a significantly higher percentage (51%) compared to Argoverse 2 (28%).
Conversely, Argoverse 2 stands out with a much larger proportion of instances involving backward motion (24%) compared to Waymo (4%).
This disparity in motion characteristics highlights that the two datasets capture different driving environments and scenarios, with Waymo potentially focusing on highway or structured urban driving, while Argoverse 2 encompasses more diverse traffic situations.

Figure: Figure 3: Linear accuracies for RedMotion and Wayformer on the Argoverse 2 Forecasting dataset. Left plot: Direction validation top-1. Middle plot: Motion validation top-1. Right plot: Motion validation top-5.

[Figure 3](#S5.F3) shows the top-1 linear probing accuracy for our coarse-grained direction and the top-1 and top-5 linear accuracy for our fine-grained motion classes on the Argoverse 2 Forecasting dataset.
The scores are computed on the validation split over the course of training.

Both models achieve similar direction top-1 and motion top-5 accuracy scores, while the Wayformer model achieves a slightly higher motion top-1 accuracy.
Both models reach high direction top-1 accuracy and motion top-5 accuracy scores, $>$80% and $>92$% respectively.
This shows that both models likely exhibit neural collapse towards interpretable motion features.

Figure: Figure 4: Normalized standard deviation representation quality metric for RedMotion and Wayformer.

The representation quality metric normalized standard deviation of embeddings is shown in [Figure 4](#S5.F4).
Both RedMotion and Wayformer learn embeddings with a normalized standard deviation close to the desired value of $1/\sqrt{\text{dim}}$.

[Figure 5](#S5.F5) shows the linear probing accuracy scores for our coarse-grained acceleration, direction, and speed features on the Waymo Open Motion dataset.
Here, we report the scores for each hidden state $H_{i}$ in the RedMotion model (i.e., after each module in the motion encoder, see [Figure 1](#S3.F1)).
Similar accuracy scores are reached for all features at all three hidden states.
The accuracy scores for the acceleration and speed features progressively improve, while the scores for the direction class jump to a score of  80% early on.
Compared to the direction scores on the Argoverse 2 Forecasting dataset, the scores on the Waymo Open Motion dataset “jump” earlier.
We hypothesize that this is linked to the shorter input motion sequence on Waymo ($1.1\text{\,}\mathrm{s}$ vs. $5\text{\,}\mathrm{s}$), which limits the amount possible movements and thus simplifies classifying direction.
Overall, the highest scores are reached for the speed features, indicating that these are most relevant for motion forecasting.
Additionally, the rightmost plot in [Figure 5](#S5.F5) shows the linear probing results for classifying agent types (i.e., vehicle, cyclist, and pedestrian).
Across all hidden states, very high linear accuracy scores are reached indicating that the representations for agent types are fully collapsed and are particularly important for motion forecasting.

Figure: Figure 5: Top-1 linear accuracies at module 0, module 1 and module 2 for acceleration, direction, and speed on the Waymo Open Motion dataset.

Figure: Figure 6: Linear accuracy for our fine-grained motion classes and its aggregated components, along with their difference. The aggregated components’ accuracy, calculated as the product of individual component accuracies, is consistently higher than the motion accuracy. The difference between these accuracies varies from $0.00$ to $0.04$ throughout the training.

[Figure 6](#S5.F6) presents the linear accuracy measurements for both our fine-grained motion classes and its aggregated components (i.e., our coarse-grained direction, speed, and acceleration features and the agent types) across various steps.
The blue curve represents the accuracy of the aggregated components, calculated as the product of individual component accuracies.
The green curve, on the other hand, depicts the linear accuracy of our fine-grained motion classes.
As expected, the linear accuracy of the aggregated components consistently exceeds that of the motion curve.
This outcome is expected as it is generally less challenging to identify individual components with distinct probes than to identify all components simultaneously.

## 6 Controlling motion forecasts at inference

Building on the insight that hidden states are likely collapsed towards our quantized motion features, we fit control vectors to opposing motion features.
These control vectors allow for controlling motion forecasts at inference.
Specifically, we use our quantized motion features to build pairs of opposing features for the Argoverse 2 Forecasting and the Waymo Open Motion dataset.
Afterwards, we fit sets of control vectors ($V_{i}$ in [Figure 1](#S3.F1)) as described in [Section 3.3](#S3.SS3).

[Figure 7](#S6.F7) shows qualitative results on the Waymo Open Motion dataset for controlling a motion forecast with the set of control vectors for speed and different temperatures $\tau$.
Subfigure (a) shows the default (i.e., non-controlled) top-1 (i.e., most likely) motion forecast.
In subfigure (b) and (c), we apply our speed control vector to de- and increase the driven speed of a vehicle.
Both controls affect the future speed analogously, while increasing the speed also changes the route to fit the given environment context (i.e., lanes and surrounding vehicles).

Figure: (a) Default motion forecast
Refer to caption: /html/2406.11624/assets/img/baseline_speed_crop.png

[Figure 8](#S6.F8) shows a qualitative example from the Argoverse 2 Forecasting dataset.
In subfigure (b) and (c), we apply our acceleration control vector with $\tau=-20$ and $\tau=100$ to enforce a strong deceleration and a moderate acceleration.
Moreover, we include an example for our direction control in the Appendix.
Overall, these qualitative results support the finding that the hidden states of motion sequences are arranged with respect to our discrete sets of motion features.

Possible applications in self-driving vehicles include aligning motion forecasts to specific driving styles or areas.
For example, assuming a more aggressive driving style may be modeled by the speed or acceleration control vectors applied to other road users.

Figure: (a) Default motion forecast
Refer to caption: /html/2406.11624/assets/img/baseline_crop.png

## 7 Limitations

We limit the scope of our work to motion forecasting models that use agent-centric environment representations.
Models that use scene-wide [, ] or pairwise relative representations [, ] may be controlled differently.
Furthermore, we did not evaluate how downstream tasks such as motion planning can benefit from aligned motion forecasts.
These are promising directions for future research.

## 8 Conclusion

In this work, we take a significant step towards interpreting and controlling motion forecasting models.
We use textual descriptions to quantize motion features in an interpretable way.
These textual descriptions offer a human-interpretable way to classify motion sequences.
Most importantly, we demonstrate that neural collapse towards human-interpretable classes of motion features occurs in recent motion forecasting models.
Building on these insights, we fit control vectors to opposing motion features, which allow for controlling motion forecasts at inference.

We analyze “words in motion” and examine the representations associated with motion features.
Future work can explore detailed text descriptions by incorporating both static and dynamic scene elements.
Additionally, investigating the impact of how multiple motion decoders affect embeddings offers a promising direction for further research.
Furthermore, natural language-based motion classes could enable voice control.
Adding voice control to self-driving vehicles could significantly change how humans interact with these machines, enhancing their accessibility and usability.

#### Acknowledgments

The research leading to these results is funded by the German Federal Ministry for Economic Affairs and Climate Action within the project “NXT GEN AI METHODS”.

## Appendix

Figure: (a) Default motion forecast
Refer to caption: /html/2406.11624/assets/img/left_baseline_crop.png

Title origin. The title of our work “words in motion“ is inspired by our quantization method using natural language and a common notion in the computer architecture literature.
In computer architecture, a word is a natural unit of data for a processing unit (e.g., CPU or GPU). In our work, words are classes of motion features, which are embedded in the hidden states of motion sequences processed by motion forecasting models.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. In the introduction, the authors claim that they identify that interpretable features are embedded in hidden states of transformer-based models. Since this is a well-known observation, the reviewer is curious about how the work adds to the existing knowledge in the mechanistic interpretability literature.
2. In the conclusion, the authors claim that they take a significant step towards the mechanistic interpretability and controllability of the transformer models. This is a bold claim - how do the results from the paper justify the claim and add to the pre-existing knowledge in the broader transformer interpretability literature?
3. How do different feature quantization thresholds affect results? The rationale for these specific thresholds should be better justified by showing the dataset statistics. The authors mention the choice was based on insights from Seff et al., 2023 - what were the specific insights the authors used for their choice of classes?
4. In Section 3.2, the authors claim to use the mean of the standard deviation to measure collapse - is this correct or should this metric be just the standard deviation?`
5. For the control vectors found using pca, what is the variance explained in each case?
6. The implications of neural collapse on the post-training processes of a model like transferability and generalization is still an open question (Vignesh Kothapalli, 2024). Can the authors better justify the use of neural collapse to show the validity of their probe accuracies?
7. How would the control effects change when combining multiple control vectors (e.g., speed + direction)?
8. How do the control vectors found using one approach compare to the other, in terms of their cosine similarity?
9. What was the intuition behind choosing the temperature range of [-50,50]?
10. Can the authors comment on how to extend their method to continuous control features rather than discrete classes as they have used in the paper?
12. Will the authors release both code and pre-trained models, including the trained sparse autoencoders?
soundness: 2
strengths: 1. The paper applies interpretability approaches to transformer-based models beyond the natural language domain and propose neural collapse as a metric of interpretability.

2. The authors use sparse autoencoder-based steering for improving control vector linearity for motion control and zero-shot generalization.

3. The authors have done significant work to apply the proposed method to multiple motion forecasting architectures and datasets.
summary: The paper focuses on the interpretability and control of transformer-based motion forecasting models. The authors aim to interpret hidden states of motion transformers and control them during inference. They use neural collapse to assess whether human-interpretable features are embedded within hidden states. The authors fit control vectors for steering during inference for the interpretable features. They further finetune these vectors using sparse autoencoders. They show that enforcing sparsity leads to a more linear relationship between the strength of the steering and interpretable features. They apply this approach to various motion forecasting models with different fusion mechanisms and environment representations. Finally, they address domain shifts using the control vectors and enable zero-shot generalization.
weaknesses: 1. The paper relies heavily on neural collapse as a measure for the model learning clusters of interpretable features. The authors should verify the feature clusters are indeed distinct by comparing the within-class and between-class variance.
2. The L1 sparsity in the training objective is known to induce feature shrinkage and result in poor reconstructions (Wright and Sharkey, 2024). The authors should compare l1, l2, and reconstruction loss for other SAE architectures that do not induce feature shrinkage such as TopK or JumpReLU SAEs, and use the best-performing architecture to finetune control vectors and show their impact on control linearity. If training new SAEs seem out-of-scope, then they should report the % loss recovered from the SAEs and ensure that they are in an acceptable range.
3. The authors should test their approach on other naturally occurring domain shifts like various traffic densities, scenes with different weather conditions, etc.
4. Details on the training of the sparse autoencoders such as choice of learning rate and schedule, batch size, number of epochs, etc. as missing.
5. Other minor comments:
           a. The authors give a brief explanation of neural collapse in paragraph 2 of the introduction but do not explicitly mention the term.
           b. Along similar lines, the authors use terms like domain shift and fusion mechanisms for the transformer architectures and zero- 
               shot generation, they should be properly defined and contextualized for their specific problem.
           c. The reference for sparse autoencoders used in Section 4.4 is for the Gated SAEs paper, however, the authors do not use that 
               specific architecture. The reference should be changed appropriately.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: See weaknesses, above. I would be happy for the authors to provide commentary on the baselines and computational resources, then I would be happy to raise my score.
soundness: 3
strengths: - **Interpretability:** The method takes motion transformers, and maps hidden states to human-interpretable features, thus clarifying the model’s decision-making process. In general, interpretability is an important area.

- **Controllability:** Control vectors allow manipulation of specific motion features (e.g., speed, acceleration) at inference time without retraining, enabling intuitive model adjustments.

- **Zero-shot Generalization:** The interpretable control vectors support generalization to unseen scenarios, like different driving styles or environments, compensating for domain shifts in the data.

- **Nice integration across disciplines:** I like using neuro-inspired linear probing and their quantization method using natural language. I think this shows an elegant combination of techniques that can be leveraged to build interpretable systems.
summary: The paper introduces a method to interpret and control transformer-based motion forecasting models by analyzing hidden states through the lens of neural collapse. Using linear probes and control vectors, it maps hidden states to interpretable features, such as speed and direction, enabling insight into the model’s decision-making and the ability to manipulate forecasts without retraining. This approach enhances model interpretability, though its reliance on neural collapse and associated computational demands present some (minor) challenges, and testing other base models beyond SAE vs. PCA would be interesting.
weaknesses: - **Reliance on Neural Collapse:** The method's effectiveness depends on well-defined hidden state clusters. If neural collapse is weak, the extracted features and control vectors may be less reliable.

- **Limited Feature Scope:** The approach primarily focuses on basic motion features (e.g., speed, acceleration, direction) and could be expanded to capture more complex motion patterns and interactions with the environment.

- **Limited Baselines**: Currently, they primarily focus on comparisons for the control vectors are to PCA, but do not consider methods that inherently have dynamics, such as temporal convolutional networks (TCNs), nonlinear embedding methods that consider time-series inputs (one prominent example would be Schneider et al. 2023 Nature), or RNNs. While not critical for their argument, they should minimally discuss this in a limitation/discussion section. 

- **Computational Overhead:** Using sparse autoencoders increases computational demands, potentially extending training time and resource requirements; it would be ideal to estimate the computational resources used and those needed to utilize their code and models that they note will be released.

---

confidence: 5
contribution: 2
initial_rating: 3
presentation: 1
questions: See the weakness above.
soundness: 2
strengths: The problem is well-motivated: building an interpretable and controllable motion prediction network.
summary: The authors have proposed finding an interpretable feature in the motion transformer and then enabling the control of the prediction based on the found interpretable features. To this end, authors claim that the interpretable feature can be found with the linear probing the hidden state, then use a control vector combined with sparse autoencoder to control the output of the motion transformer.
weaknesses: 1. I think the overall writing needs to be improved in multiple aspects.
- First, it is hard to understand why the author has focused on motion transformers in the introduction. After reading section 3 (method part), I can understand why the author needs an interpretable method for motion transformers, but in the introduction, it is explained as an application area of their method, not the main focus.
- While the author suggests they have used neural collapse to measure the human interpretable feature, it is not known what neural collapse is in the introduction. 

2. Neural collapse is the most important term in the paper, but it is not defined clearly (or clearly mathematically), so it is unable to understand what the authors are using. Note that the authors have argued that they have used neural collapse as a metric, so we need a mathematical way to measure it. I think the author should define the following in a clear (or mathematical) way.
- What is "Neural collapse"?
-  What is an "interpretable feature"?
- What is "motion features"?

3. While the author mentioned they have used sparse autoencoder (SAE), it is not clear which one they have used [1,2,3]. 
- Which layer did the author train the SAE? SAE is usually trained on the hidden state of the transformer [1,2,3], and the layer is always selected carefully.
- Can the author report that the SAE is trained well? For instance, the reconstruction error of the SAE. (I think the author mentioned it is reported in the Appendix, but could not find it).

4. Can the author explain the goal of the motion transformer? What is the input of the motion transformer and output of the motion transformer?

5. It is not clear why the author used SAE for the collability. There are several activation steering methods [4,5]. I think it is great to claim the benefit of using SAE over other methods. 

Overall, I think the paper needs to improve in terms of writing, especially the introduction and the method part. Moreover, the author could improve the overall experimental details and analysis mentioned above.

Reference\
[1] Scaling and evaluating sparse autoencoders\
[2] Jumping Ahead: Improving Reconstruction Fidelity with JumpReLU Sparse Autoencoders\
[3] Improving Dictionary Learning with Gated Sparse Autoencoders\
[4] In-context Vectors: Making In Context Learning More Effective and Controllable Through Latent Space Steering\
[5] In-Context Learning Creates Task Vectors

---

confidence: 3
contribution: 2
initial_rating: 3
presentation: 2
questions: - How sensitive are the results to the thresholds mentioned in Appendix A.3?
soundness: 2
strengths: - The paper addresses the under-explored area of mechanistic interpretability in motion forecasting, providing a timely study at this intersection.
- The paper presents thorough quantitative and qualitative experiments, demonstrating the effectiveness of the interpretability method across different models and scenarios.
summary: The paper introduces a method for extracting interpretable representations in transformer-based motion forecasting models: (i) leverage neural collapse to identify a structured latent space where features form interpretable clusters, (ii) use sparse autoencoders to optimize "control vectors" for each interpretable feature. This method is evaluated on three motion forecasting models, demonstrating effective feature manipulation and zero-shot generalization.
weaknesses: - While the paper is new in connecting mechanistic interpretability with motion forecasting, the technical components (neural collapse and sparse autoencoders) have been studied in LLMs. The claim of `a significant step towards mechanistic interpretability and controllability of transformer models` (L452) seems quite overstated. Further clarification on the unique technical contributions would strengthen the paper.
- The paper extends interpretability techniques from LLMs to the motion domain; however, the contexts differ fundamentally: LLMs deal with discrete tokens, while motion forecasting operates in a continuous space. The proposed method quantizes continuous variables like speed and acceleration into discrete categories, which might oversimplify the difference. A deeper discussion on the impact of this discretization and the influence of specific parameters (e.g., in L160) would enhance understanding of the challenges and significance of applying mechanistic interpretability to motion forecasting.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: 1. Will using a sparse autoencoder significantly increase computational costs?

2. This paper is not well written. Some of the design motivations are not clearly explained.
soundness: 2
strengths: 1. The paper introduces an approach for interpreting hidden states in motion forecasting models by leveraging neural collapse and linear probing.

2. The paper uses sparse autoencoders to optimize control vectors and enhance the linearity between control temperatures and forecasts.

3. Applying the method to self-driving cars and motion prediction shows its relevance to real-world applications.
summary: This paper aims to interpret hidden states and control them at inference. It assesses whether human-interpretable features are embedded in hidden states of motion transformers using neural collapse. The latent space properties are used to fit control vectors for each interpretable feature. The control vectors are optimized using sparse autoencoding and enforcing sparsity results in a more linear relationship between control vector temperatures and forecasts.
weaknesses: 1. The related work section does not clearly and effectively highlight the similarities and differences between prior research and your work.

2. The study lacks baseline comparisons. Are there existing methods that could be used to evaluate your proposed approach?

3. The paper lacks novelty, as it does not introduce linear probes or control vectors but simply applies these existing methods to motion control.

Meta-review: The problem studied in this submission is considered interesting by the majority of reviewers, attracting favourable remarks about clear motivation and broader interest in studying interpretability techniques for transformers beyond the language domain. In addition, other positive comments included the ability to manipulate specific motion features (e.g., speed, acceleration) at inference time without retraining) as well as the application to multiple motion forecasting architectures and datasets.

On the flip side, several reviewers criticised the submission for requiring major revision in terms of clarity of writing while also requesting authors to address a perceived lack of baseline comparisons. While the reviewers included an ablation study testing various SAE alternatives, this does not go far enough. Other criticisms revolve around the novelty of the approach (beyond changing the application domain) and justification for claims made throughout the manuscript. Finally, the method is considered to rely heavily on neural collapse, for which reviewers may prefer stronger robustness guarantees.

On balance, this submission is unfortunately not yet ready for publication at ICLR.

Decision: Accept (Poster)