Words in Motion: Extracting Interpretable Control Vectors for Motion Transformers
Authors: Omer Sahin Tas
Royden Wagner
Venue: ICLR 2025 Poster (2025)
Keywords: control vectors
activation steering
sparse autoencoders
neural collapse
mechanistic interpretability
motion forecasting
Primary area: interpretability and explainable AI
TL;DR: We use linear probing and show that neural collapse toward interpretable features occurs in hidden states of motion transformers, indicating a structured latent space. We fit control vectors using opposing features and optimize them using SAEs.

Abstract: Transformer-based models generate hidden states that are difficult to interpret. In this work, we analyze hidden states and modify them at inference, with a focus on motion forecasting. We use linear probing to analyze whether interpretable features are embedded in hidden states. Our experiments reveal high probing accuracy, indicating latent space regularities with functionally important directions. Building on this, we use the directions between hidden states with opposing features to fit control vectors. At inference, we add our control vectors to hidden states and evaluate their impact on predictions. Remarkably, such modifications preserve the feasibility of predictions. We further refine our control vectors using sparse autoencoders (SAEs). This leads to more linear changes in predictions when scaling control vectors. Our approach enables mechanistic interpretation as well as zero-shot generalization to unseen dataset characteristics with negligible computational overhead.

Published as a conference paper at ICLR 2025
WORDS IN MOTION: EXTRACTING INTERPRETABLE CONTROL VECTORS FOR MOTION TRANSFORMERS
Ömer Sahin Taş Royden Wagner*
FZI Research Center for Information Technology Karlsruhe Institute of Technology
ABSTRACT
Transformer-based models generate hidden states that are difficult to interpret. In this work, we analyze hidden states and modify them at inference, with focus on motion forecasting. We use linear probing to analyze whether interpretable features are embedded in hidden states. Our experiments reveal high probing accuracy, indicating latent space regularities with functionally important directions. Building on this, we use the directions between hidden states with opposing features fit control vectors. At inference, we add our control vectors to hidden states and evaluate their impact on predictions. Remarkably, such modifications preserve the feasibility of predictions. We further refine our control vectors using sparse autoencoders (SAEs). This leads to more linear changes in predictions when scaling control vectors. Our approach enables mechanistic interpretation well zero- shot generalization to unseen dataset characteristics with negligible computational overhead.
INTRODUCTION
Accurately predicting sequential data in an interpretable way is desirable for many real-world applications. However, these two objectives often conflict: methods achieving higher accuracy tend to rely on the increased complexity of their underlying models (Kaplan al., 2020; Bahri et al., 2024). This, in turn, renders them difficult to interpret in terms of semantically meaningful concepts.
Interpretability methods aim uncover how models process information, often by analyzing whether learned features align with semantically meaningful concepts. The manifold hypothesis (Bengio et al., 2013) suggests that nigh-dimensional data often lies lower-dimensional manifold. Deep learning models learn representations that approximately capture the manifold's geometry by mapping data into a space where related inputs closer together (Rifai al., 2011). Therefore, we focus on the structure of the learned representations to make predictions of the model's 's behavior.
Deep learning models are trained with loss functions that encourage the clustering of data samples in latent space (Papyan et al., 2020). Together with regularizers that prevent overfitting, clusters become more distinct over the course of training, i.e. neural collapse (Galanti et al., 2022: Wu & Papyan, 2024). Following Ben-Shaul et al. (2023), we use linear probes (Alain & Bengio, 2017) to measure neural collapse toward interpretable features hidden states. High probing accuracy implies separability of features, which suggest functionally important directions in hidden states. Building on the insight that interpretable features are embedded hidden states, we fit control vectors to the directions between hidden states with opposing features.
To further enhance this approach, we use sparse autoencoders to extract more distinct features from hidden states (Bricken et al., 2023). We evaluate sparse autoencoders with fully-connected, convolutional, and MLPMixer layers; and different activation functions. Our experiments with sparse autoencoders of varying sparse intermediate dimensions show that enforeing sparsity leads to more linear changes in prediction when scaling control vectors.
We apply our method recent multimodal motion transformers (Nayakanti al., 2023; Zhang et al., 2023b; Wagner et al., 2024). They process features of past motion sequences (i.e., past positions,
"Equal contribution. Emails: tas@izi. de, royden. www.eeeeit...................... Our implementation is available https: //t::////ttt/ httpg/tttttooooooooooeoooootettt

---

Published as a conference paper at ICLR 2025
orientation, acceleration, and speed) and environment context (i.e., map data and traffic light states), and transform them into future motion sequences. Like other transformer models, they rely on learned representations of these features, resulting in hidden states that are difficult to interpret and control. We focus on analyzing interpretable motion features that are physically measurable, such as speed, acceleration, direction, and agent type. By leveraging these features, our approach enables interpretable control over generated forecasts and facilitates zero-shot generalization. Specifically, in this work:
We argue that, fit control vectors, latent space regularities with separable features necessary. We use linear probing and show that neural collapse toward interpretable features occurs in hidden states of recent motion transformers, indicating structured latent space.
We fit control vectors using hidden states with opposing features. By modifying hidden states at inference, we show that control vectors describe functionally important directions. Similar to the vector arithmetic in word2vec, we obtain predictions consistent with the current driving environment.
We use sparse autoencoders to optimize control vectors. Notably, enforcing sparsity leads to more linear changes in predictions when scaling control vectors. We use linearity measures to compare these results against Koopman autoencoder and SAEs with various layers and activation functions, including convolutional and MLPMixer layers.
2 RELATED WORK
Concept-based interpretability. Kim et al. (2018) propose explaining predictions with human- interpretable concepts, rather than relying on sample-based raw features. They first choose a set of examples that represent distinct concepts and measure the influence of high-level concepts on the model's 's decisions, thereby providing global explanations. In recent text-to- text-to-image image diffusion setting, Conceptor (Chefer al., 2024) decomposes a concept into a weighted combination of interpretable elements.
Structure of the latent space. Mikolov et (2013) show that consistent regularities naturally emerge from the training process of word embeddings. This phenomenon, commonly referred to as the word2vec hypothesis, suggests that learned embeddings capture both semantic and syntactic relationships between words through consistent vector offssets in latent space. While the observed linear offsets naturally fit a flat latent space, non-Euclidean geometric models (e.g., Riemannian manifolds) can better capture structural distortions (Arvanitidis al., 2018). In those cases, "vector arithmetic" can be seen as an approximation to geodesic operations on a curved latent space.
Neural collapse. A recent line of work (Papyan et al., 2020; Galanti et al., 2022; Wu & Papyan, 2024) introduces the term neural collapse to describe a desirable learning behavior of deep neural networks for classification.' It refers the phenomenon that learned top-layer representations form semantic clusters, which collapse to their means at the end of training. In addition, the cluster means transform progressively into equidistant vectors when centered around the global mean. Therefore, neural collapse facilitates classification tasks and is considered desirable learning behavior for both supervised (Papyan al., 2020) and supervised learning (Ben-Shaul et al., 2023).
Hidden state activations. Transformers consist of attention blocks, followed by simple feed-forward networks, whose hidden state activations are analyzed for interpretability. Elhage et al. (2022) explore two key hypotheses that describe how these activations capture meaningful structures: the linear representation hypothesis (Pennington et al., 2014) and the superposition hypothesis (Arora et al., 2018). These hypotheses essentially state that the neural networks represent features directions in their activation space, and that representations can decomposed into independent features.
Control v are used for form of activation steering, where concept-based vectors are added to activations (i.c., hidden states) of transformer models. In natural language processing (Zou et al.,
Neural collapse not to he confused with representation collapse (Hua al., 2021; Barbero al., 2024), where learned representations across all classes collapse to redundant or trivial solutions (c-g., zero vectors). Also referred to steering vectors (Subramani et al., 2022), style vectors (Konen et al., 2024), activation addition (Turner al., 2024).
2

---

Published as a conference paper at ICLR 2025
2023; Subramani et al., 2022; Turner et al., 2024; Heo et al., 2025), control vectors allow targeted adjustments to model outputs by modifying hidden states without the need for fine-tuning or p engineering. Control vectors are set of vectors that capture the difference of hidden states with opposing concepts or features (Rimsky et al., 2023). This approach requires a well-structured latent space, where samples are clustered according to classes features (e.g., high degree of neural collapse, see Section 3.2).
key goal of interpretability research decompose models and gain mechanistic interpretation of how their components function. Sparse autoencoders (SAEs) leverage the linear representation hypothesis and approximate the model's activations with linear combination of feature directions. By enforcing sparsity in latent space, they separate features into distinct, interpretable representations (Bricken et al., 2023; Cunningham et al., 2024; Gao et al., 2025). Related autoencoders linearize learned representations either by manifold flattening (Psenka et al., 2024) or using Koopman operators (Lusch et al., 2019; Azencot et al., 2020).
Sparse autoencoders.
Our method differs from prior works in several aspects. We measure neural collapse in multimodal models for motion forecasting (i.e., regression) instead of unimodal image classifiers (Papyan et al., 2020) or language models (Wu & Papyan, 2024). Unlike Conmy & Nanda (2024), we do not manually suppress SAE features in control vectors. Furthermore, we do not use our SAEs during inference (Bricken et al., 2023), but to optimize control vectors beforchand, resulting in negligible computational overhead.
METHOD
3.1 MOTION FEATURE CLASSIFICATION USING NATURAL LANGUAGE
In contrast to natural language, where words naturally carry semantic meaning, motion lacks prede- fined labels. Therefore, we identify human-interpretable motion features by quantizing them into discrete subclasses in natural language.
Initially, we classify motion direction using the cumulative sum of differences in yaw angles, as- signing either left, straight, or right. Additionally, we introduce stationary class for stationary objects, where direction lacks semantic significance. We define further classes for speed, dividing the speed values into four intervals: high, moderate, low, and backwards, Lastly, we analyze the change in acceleration by comparing the integral of speed over time to the projected displacement with initial speed. Accordingly, we classify acceleration profiles as either accele e raa ng, dece erat t i ng. or constant (see Figure la). Our thresholds for motion features are based on insights from Ettinger et al. (2021); Seff et (2023). The threshold values are detailed in Appendix A.3.
3.2 NEURAL COLLAPSE AS A METRIC OF INTERPRETABILITY
We use neural collapse as a metric of interpretability. Specifically, we focus on interpreting hidden states (i.e., activations or latent representations) and evaluate whether hidden states embed inter- pretable features. We measure how close abstract hidden states are related to interpretable semantics using linear probing accuracy (Alain & Bengio, 2017).² We train linear probes (i.e., linear classifiers detached from the overall gradient computation) on top of hidden states (H,, in Figure 1). During training. we track their accuracy in classifying interpretable features on validation sets. Adapted to motion forecasting, we choose the aforementioned motion features as interpretable semantics.
Besides linear probing accuracy, following Chen & He (2021), we the mean of the standard deviation of the fz-normalized embedding to measure representation collapse. Representation collapse refers to undesirable learning behavior where learned embeddings collapse into redundant or trivial representations (Hua et al., 2021; Barbero et al., 2024). Redundant representations have a standard deviation close zero. In way, representing the opposite of neural collapse. As shown in (Chen & He, 2021), rich representations have standard deviation close to 1 1/Vdhh /dim, where dim is the hidden dimension.
Ben-Shaul et al. (2023) show that linear prohing accuracy is consistent with the accuracy of nearest class center classifiers, which are typically used to measure neural collapse.
3

---

Published as a conference paper at ICLR 2025
Control
Xm
Motion decoder
vectors
MD
H
Va
XT
Module m
M
H
Vi
A vehicle is moving right moderata speed while decelezar ang.
Mation tokens
Motion encuder
Su
V
Enc.
Dec.
S1
Vi
Probe X78
Sparse autoencoder
(a) Interpretable motion features
(b) Controllable motion encoder
Figure 1: Words in Motion. (a) We classify motion features in an interpretable way, in natural language. (b) We measure the degree to which these interpretable features are embedded in the hidden states H,: of transformer models with linear probes. Furthermore, we use our discrete features and sparse autoencoding to fit interpretable control vectors V.. that allow for modifying motion forecasts inference. The training the sparse autoencoder is shown with arrows (->) and the fitting of control vectors with blue arrows (+).
3.3 INTERPRETABLE CONTROL VECTORS
We use our interpretable features to form pairs of opposing features. For each pair, we build a dataset and extract the corresponding hidden states. Next, we compute the element-wise difference between the hidden states of samples with these opposing features. Finally, following Zou et al. (2023) we apply principal component analysis (PCA) with single component as pooling method. This reduces the computed differences to a single scalar per hidden dimension to generate control vectors (V,, in Figure lb).
We optimize our control vectors using SAEs (Bricken et al., 2023). SAEs extract distinct features in hidden states by encoding and reconstructing them from sparse intermediate representations (St, in Figure lb). We hypothesize that sparse intermediate representations enable more linear decomposition of our interpretable features, and hence, more distinct control vectors. Therefore, generate intermediate control vectors Vi, by pooling the differences between hidden states with H.!!) Specifically, we compute
opposing features (H. por
spes = ReLU Wene HH baec) + bene
where W and denote weights and biases of the SAE. Similarly, compute S and obtain the intermediate control vectors as
= PCA S
Sneg L.:
Leveraging the Johnson-Lindenstree Lemma, 4 we use the SAE decoder to project the intermediate control vectors back the hidden dimension the motion encoder
V W dee Vi, base
This enables using sparse autoencoders of arbitrary sparse intermediate dimensions for generating control vectors of fixed dimension. inference, we scale the control vectors with a temperature parameter (r in Figure 1b) to control the strength of the corresponding features of given sample.
'Johnson & Lindenstrauss (1984) state that set of points in high-dimensional space can be projected into lower-dimensional space while approximately preserving the pairwise distances between points.
4

---

Published as a conference paper at ICLR 2025
4 EXPERIMENTAL SETUP
4.1 MOTION FORECASTING MODELS
We study three recent motion transformers for self-driving. Wayformer (Nayakanti et al., 2023) and RedMotion (Wagner et al., 2024) models employ attention-based scene encoders learn agent- centric embeddings of past motion, map, and traffic light data. To efficiently process long sequences, Wayformer uses latent query attention (Jaegle et al., 2021) for subsampling. RedMotion lowers memory requirements via local-attention (Beltagy et al., 2020) and redundancy reduction. HPTR (Zhang et al., 2023b) models Iearn pairwise-relatin environment representations via kNN-based attention mechanisms. For Wayformer, we use the implementation by Zhang et al. (2023b) and the early fusion configuration. Therefore, we analyze the hidden states generated by an MLP-based input projector for motion data, which consists of three layers. For RedMotion and HPTR, we the publicly available implementations. We configure RedMotion with a late fusion encoder for motion data, and HPTR using custom hierarchical fusion setup with modality-specific encoder for past motion with a shared encoder for environment context. Further details on model architectures and fusion mechanisms are presented in Appendix A.4 and A.5.
4.2 LINEAR PROBES
We add linear probes for our quantized and interpretable motion features (see Section 3.1) to hidden state of all models (H (m) in Figure I, where m E {0, 1,2] is the module number and is the temporal index). These classifiers are learned during training using regular cross-entropy loss to classify speed, acceleration, direction, and the agent classes from hidden states. We decouple this objective from the overall gradient computation. Therefore, these classifiers do not contribute to the alignment of hidden states, but exclusively measure neural collapse toward interpretable features.
4.3 CONTROL VECTORS
Using our interpretable motion features, we build pairs of opposing features. Specifically, we generate speed control vectors representing the direction from low to high speed, acceleration control vectors representing the direction from decelerating to accelerating, and direction control vectors representing the direction from turning left turning right, and agent control vectors representing the direction
from pedestrian to vehicle. For each pair, we use the hidden states H (m) from module m = 2 and the last embedding per motion sequence (with i = 1)), as it is closest to the start of the prediction.
4.4 TRAINING DETAILS AND IYPEERPARAMETERS
Motion transformers. We provide Wayformer and HPTR models with the nearest 512 map polylines, and RedMotion model with the nearest 128 map polylines. All models process maximum of 48 surrounding traffic agents as enviromment context. For the Argoverse Forecasting (abbr. AV2F) dataset, we past motion sequences with 50 time steps (representing 5s input. For the Waymo Open Motion (abbr. Waymo) dataset, we use past motion sequences with 11 steps (representing 1. s) as input. For Wayformer and RedMotion, we use the unweighted sum of the negative likellhooo loss for positions modeled as mixture of Gaussians and cross-entropy for confidences as motion forecasting loss. For HPTR, we additionally use the cosine loss for the heading angle and the Huber loss for velocities. We use AdamW (Loshchilov & Hutter, 2019) in its default configuration as optimizer and set the initial learning rate x 10-4. All models have hidden dimension of 128 and are configured to forecast k- 6 trajectories per agent. As proceprocessing, we follow Konev (2022) and reduce the predicted confidences of redundant forecasts.
Sparse autoencoders. We train SAES as an auxiliary model with sparse intermediate dimensions of 512. 256, 128, 64, 32, and 16. The total loss combines Ez reconstruction loss with an & spar- sity penalty: En ensures accurate reconstruction, while é promotes sparsity by minimizing small, noise-like activations. The & must be carefully scaled to avoid deadening important features (Ra- jamanoharan al., 2024a). We scale it scaled by 3 x 10-4, We optimize the models over 10 000 epochs using the Adam optimizer (Kingma & Ba, 2015) and a batch size of 528. The final loss values provided in Table 7 in the appendix.
5

---

Published as a conference paper at ICLR 2025
5 RESULTS
5.1 EXTRACTING INTERPRETABLE FEATURES FOR MOTION
Our approach relies on a well-structured latent space, where samples are clustered with respect to interpretable features. First, we ensure that features are not highly correlated, as confirmed by the Spearman feature correlation analysis in Appendix A.6. Next, we report linear probing accuracy for interpretable features during training on the AV2F and Waymo datasets.
Figure 2 shows the linear probing accuracies for our interpretable motion features for the AV2F dataset. The scores are computed on the validation split over the course of training. All models achieve similar accuracy scores, while the Wayformer model achieves slightly higher scores for classifying acceleration and lower scores for agent classes. Overall, we measure high linear probing accuracy for all intepretable features. This shows that all models likely exhibit neural collapse toward interpretable features.
Probing acceuracy
Probing acccuracy
Probing acceuracy
Probing accouracy
1.00
0.76
0.70
0.98
0.74
0.65
0.72
0.75
0.96
0.70
0.53
0.94
0
Step -10*
Step -10
Step
2
3
Step -10
(a) Speed
(b) Acceleration
(c) Direction
(d) Agent type
Figure 2: Linear probing accuracies for RedMotion, Wayformer, and HPTR on the validation split of the AV2F dataset.
The representation quality metric normalized standard de- viation of embeddings is shown in Figure 3. Both HPTR and RedMotion learn to generate embeddings with nor- malized standard deviation close to the desired value of 1/ 1/Vdim, where dim is the hidden dimension. The scores for Wayformer are lower, which reflects differences between attention-based and MI.--hased motion encoders.
std-é2 -10
/imm
2 Step-10
Figure shows the linear probing accuracies for our inter- pretable features the Waymo dataset. Here, report the scores for each of the three hidden states H; in the RedMo- tion model (i.e., after each module m in the motion encoder, see Figure 1). Similar accuracy scores are reached for all features at all three hidden states. The accuracies for the speed and acceleration classes continuously improve, while those for direction classes reach 0.80 early on. Compared to the direction scores on the AV2F dataset, the scores on the Waymo dataset "jump" earlier. We hypothesize that this is linked to the shorter input motion sequence on Waymo (1.1s s vs. 5s, which limits the amount possible movements. In contrast to the AV2F dataset, higher accuracies are achieved for classifying speed. Overall, the highest scores reached for classifying agent types, as on the AV2F dataset.
Figure 3: Normalized standard devia- tion representation quality metric for RedMotion, Wayformer, and HPTR.
Probing acccuracy
Probing acceuracy
Probing acceuracy
Probing acceuracy
0.95
0.80
1.000
0.91
0.80
0.75
0.93
0.999
0.78
0.92
0.70
0.91
0.76
Step 10
1
Step 100
1
Step Step-10
1
2
Step 100
1
(a) Speed
(b) Acceleration
(c) Direction
(d) Agent type
Figure 4: Linear probing accuracies at module O, module and module for classfiying speed, acceleration, direction, and agent type on the validation split of the Waymo dataset.

---

Published as a conference paper at ICLR 2025
In addition to linear probing, measure neural collapse using (CDNV) (Galanti et al., 2022), see Appendix A.19. On the Waymo dataset, the within-class variance values and the mean distance norm for RedMotion are 10,68 and 11.24, respectively, resulting in CDNV of 0.95. On the AV2F dataset, these values are 5.73 and 2.32, yielding a CDNV of 2.46. We hypothesize that the higher CDNV value AV2F is caused by the longer past motion sequence (i.e., 5 vs. 1.18 on Waymo), allowing for greater range of potential movements.
normalized variance
5.2 MODIFYING HIDDEN STATES OF MOTION TRANSFORMERS AT INFERENCE
Building on the insight that hidden states are likely collapsed toward our interpretable features, we fit control vectors using opposing features. These control vectors allow for modifying motion forecasts at inference. Specifically, we build pairs opposing features for the AV2F and the Waymo dataset. Then, fit sets of control vectors (V, in Figure 1) described in Section 3.3. At inference, add the control vectors generated for the last temporal index (i = -) to all embeddings (i E (0....,49) for AV2F, E (O, , 10} for Waymo).
5.2.1 QUALITATIVE RESULTS
Figure shows qualitative example from the AV2F dataset, where modify hidden states using our control vector for acceleration scaled with different temperatures т. Subfigure 5a shows the default (i.e., non-controlled) top-1 (i.e., most likely) motion forecast. subfigures 5b and Sc, apply our acceleration control vector with r -20 and = 100 to enforce strong deceleration and moderate acceleration, respectively.
(a) Default motion forecast
(b) Enforced strong deceleration
(c) Enforced acceleration
Figure 5: Modifying hidden states to control a vehicle at an intersection. We add our acceleration control vector scaled with -20 and 100 to enforce strong deceleration and moderate acceleration. The focal agent is highlighted in orange, dynamic agents are blue, and static agents are grey. Lanes are black lines and road markings white lines.
Figure 6 shows a qualitative example from the Waymo dataset. Subfigure 6a shows the default motion forecast. In subfigures 6b 6c, we apply our speed control vector to decrease and increase the driven speed of vehicle. Both modifications affect the future speed in similar manner, while increasing the speed also changes the route to the given environment context (i.e., lanes).
(a) Default motion forecast
(b) Speed controlled 32
(c) Speed controlled r 100
Figure 6: Modifying hidden states to control vehicle before a predicted right turn. In this example, increasing the speed also changes the route to fit the given environment context (i.e., lanes).

---

Published as a conference paper at ICLR 2025
In Appendix A.8, we include example of our direction control vector. Overall, these qualitative results support the finding that the hidden states of motion sequences are arranged with respect to our discrete sets of motion features.
5.2.2 SIMILARITY-BASED COMPARISON OF CONTROL VECTORS
In this section, evaluate how control vectors obtained using SAEs differ from those derived via plain PCA. For comparison, we train SAEs with varying sparse intermediate dimensions: 512, 256, 128, 64, 32, and 16. For each control vector, calculate its pairwise angles with the control vectors for controlling other features. Table I presents the angular distances between control vectors of speed, acceleration, direction, and agent generated with plain PCA and our SAE with sparse intermediate dimension 128. As expected, the similarity between speed and acceleration, speed and agent, and acceleration and agent is notably high, while the similarity involving direction and other vectors is significantly lower. This result aligns with expectations, as positive speed and acceleration controls lead to faster motion, and our agent control vector represents transition between agent types from pedestrian to vehicle, which is associated with faster motion, as well. Angular-distane results for the remaining SAE dimensions are in Table in the appendix. The similarity of the control vectors generated using the SAE with an intermediate dimension of 128 is the highest.
Table I: Comparison of control vectors, with angles measured in degrees.
Plain PCA & Plain PCA
speed
accelenation
direction
agent
SAE & SAE
speed
acceleration
discetion
spood
10.9
spand acceleration
0.0
120.6
acceleration
0.0
126.8
0.0
122.9
7.0
direction agent
0.0
128.7
directionn
0.0
125.8
0.0
0.0
5.3 QUANTITATIVE EVALUATION OF SAES FOR OPTIMIZING CONTROL VECTORS
We empirically analyze the temporal-causal relationship between modifications on hidden states of past motion and motion forecasts. Specifically, we measure the linearity of relative speed changes in forecasts when scaling our speed control vectors. We use the Pearson correlation coefficient, the coefficient of determination (R²), and the straightness index (S-idx) (Benhamou, 2004) linearity measures. Given the large range of scenarios in the Waymo dataset, we focus on relative speed changes within a range of +50 (see Appendix A. A.14). Higher linearity implies improved controllability.
We compute linearity measures for control vectors optimized using regular SAEs (Bricken et al., 2023) with varying sparse intermediate dimensions. We achieve the highest scores using the SAE with dimension of 128 (see Table 2). Therefore, use this dimension in the rest of our evaluations.
In the following, we evaluate autoencoders with differ- ent activation functions and layer types. Following Ra- jamanoharan et al. (2024b), we use JumpReLU with a threshold 0 0.001 and regular ReLU activation func- tions. Moreover, we evaluate regular SAEs with fully- connected layers, with MLPMixer (Tolstikhin al., 2021) layers (Sparse MLPMixer), and with convolutional layers (ConvSAE). For Sparse MLPMixer and ConvSAE, we use large patch and kernel sizes to approximate the global receptive fields of fully-connected hidden units in regular SAFS. Furthermore, we evaluate a consistent Koopman autoencoder (KoopmanAE) (Azencot al., 2020) to include method that models temporal dynamics between embeddings (see Appendix A.15).
Table 2: Scaling sparse autoencoders.
Autoencoder
Pearson
S-idx
SAE
0.990
0.974
0.984
0.990
0.985
SAE-128
0.993
0.984
0.988
SAE-64
0.991
0.976
0.985
SAE-32
0.990
0.959
0.985
SAE-16
0.982
0.770
0.958
Table 3 presents linearity measures for different control vectors derived from both plain PCA pooling and SAE methods. Overall, the regular SAES (Bricken al., 2023) achieve the highest Pearson and R scores. JumpReLU activation functions improve the R² scores marginally compared to ReLU activation functions. The SAE version of Cunningham et al. (2024) does not improve the linearity scores. We hypothesize that this is due to reduced decoding flexibility since they transpose the encoder weights instead of learning the decoder weights (i.e., Wa WT) )
8

---

Published as a conference paper at ICLR 2025
The ConySAE with kernel size 64 and the KoopmanAE achieve the highest straightness index, yet the lowest R² scores. As shown Figure 7 and Figure 16 in the appendix. the range of temperatures т is much higher for this ConvSAE and significantly lower for the KoopmanAE than for e.g. the regular SAE. This lowers the R² score but does not affect the straightness index. For the ConvSAE, et al., 2024b). Therefore, the JumpReLU configuration of this SAE-type leads to significantly smaller range (see Appendix A.17), which in leads higher R scores (see Table 3). For the KoopmanAE, the opposite is likely, since activation shrinkage is caused by sparsity terms, which are not included the loss function Azencot al. (2020). Notably, activation steering with our SAE-based control vector has an almost 1-to-1 ratio between and relative speed changes (i.e.,
hypothesize that this is due strong activation shrinkage (Rajamanoharan
T = 50 corresponds roughly- -50%). This improves R² scores and enables an intuitive interface. Furthermore, improved controllability with SAEs indicates that sparse intermediate representations capture more distinct features.
Table Linearity measures for optimized control vectors: Pearson correlation coefficient, coefficient of determination (R²), and straightness index (S-idx).
Autoencoder
Activation function
Pooling
Patch/kernel size
R²
S-idx
PCA
0.988
0.969
0.981
SAE (Bricken 2023)
ReLU
PCA
0.993
0.981
0.988
SAE (Rajamanoharan al., 2024b)
JumpReLU
PCA
0.993
0.986
0.988
SAE (Cunningham et al., 2024)
ReI.U
PCA
0.987
0.971
0.980
Sparse MLPMixer
ReLU
PCA
64
0.992
0.980
0.966
Sparse MLPMixer
JumpReLU
PCA
64
0.902
0,981
0.986
Sparse MLPMixer
ReLU
PCA
32
0.990
0.978
0.985
Sparse MLPMixer
JumpReLU
PCA
32
0.991
0.980
0.986
ConvSAE
RcLU
PCA
64
0.986
0.383
0.991
ConvSAR
JumpReL.U
PCA
64
0.987
0.861
0.978
ConvSAE
ReLU
PCA
32
0.622
0.986
ConvSAE
JumpReLU
PCA
32
0.989
0.623
0.986
KoopmanAE (Azencot al., 2020)
tanh
PCA
0.991
0.057
1.000
50
Calibration curve
50
Calibration curve Lincar reference
Calibration curve Lincar reference
relatve speed change %
Linear reference
relutive speed change in %
relutive speed change
25
25
0-
-25
-50
-50
-50
20
20
(a) Plain PCA
(b) SAE
(c) ConvSAE 64
Figure 7: Calibration curves plain PCA-based speed control vectors and control vectors optimized using SAES for relative speed changes in forecasts of +50%.
In the appendix, we present an ablation study analyzing our method's sensitivity to hidden states from different modules (see Table 9) and varying speed thresholds (see Table 10). Our method performs best with sparse intermediate dimension of 128 and hidden states from module m = 2; and is more sensitive low than high speed thresholds.
5.4 RELATION OF PROBING ACCURACY TO LINEARITY MEASURES FOR CONTROL VECTORS
We train RedMotion model on the AV2F dataset using the same trajectory lengths in the Waymo dataset (I.1s s past and 8 future), while leaving all other hyperparameters as described in Section 4.4. Table 4 shows the probing accuracy and linearity measures of speed control vector for this model (see Appendix A. for the calibration curve). Compared with model trained on the Waymo dataset, the AV2F model achieves both lower probing accuracy and significantly lower linearity measures. These results support our argument that latent space regularities with separable features are necessary to fit precise control vectors.

---

Published as a conference paper at ICLR 2025
Table 4: Higher probing accuracy enables higher linearity measures. We train RedMotion models on the Waymo and AV2F datasets using the same trajectory lengths. We report the probing accuracies for speed classes and the linearity measures for the corresponding PCA-based control vectors.
Dataset
Probing accuracy
Pearson
R²
S-idx
AV2F
0.753 0.945
0.877
0.275
0.891
Waymo
0.988
0.969
0.981
5.5 ZERO-SHOT GENERALIZATION WITH CONTROL VECTORS
Domain shifts between training and test data significantly degrade the performance of many learning algorithms. Zero-shot generalization methods compensate for such domain shifts without further training or fine-tuning (Kodirov et al., 2015; Xian et al., 2017; Mistretta et al., 2024). In motion forecasting, common domain shifts lower future speeds, respectively. We simulate this domain shift by reducing the future speed in the Waymo validation split by approximately 50%. Specifically, we take the first half of future waypoints and linearly upsample this sequence to the original length.
or less aggressive driving styles that result in higher or
Table 5 shows the results of a RedMotion model trained the regular training split this vali- dation split with domain shift. We provide an overview of the used motion forecasting metrics in Appendix 18. Without the of our control vectors, high distance-based based errors, miss, and overlap rates are obtained. Using the calibration curve in Figure 7b, we compensate for this domain shift by applying our SAE-128 control vector with a temperature т -50. This significantly reduces the distance-bart -baee errors, the overlap, and the miss rates without further training. In addition, we show the results of applying our control vector with a temperature of r -30 and r 770, which improves all scores over the baseline as well.
Table 5: Zero-shot generalization to a Waymo dataset version with reduced future speeds. Best scores are bold, second best are underlined.
Control vector
Temperature r
minADE.j
Brier minADE.
minFDE.
Brier minFDE.
Overlap rate.
Miss rate |
None
4.617
0,220
0.580
SAE- 128
-30
1685
4.838
2,870
8.429
0.177
0.224
-50
1.174
1.798
0.174
0,236
SAE-122 128
-70
1.808
3.576
2.035
3,676
0.189
0.302
CONCLUSION
In this work, we take a step toward mechanistic interpretability and controllability of motion trans- formers. We analyze "words in motion" by examining the representations associated with quantized motion features. Specifically, we show that neural collapse toward interpretable classes of features occurs in recent motion transformers. The high degree of neural collapse indicates well-separated latent space, that enables to fit precise control vectors to opposing features and modify predictions inference. We further refine this approach by optimizing our control vectors using sparse autoen- coding, resulting in higher linearity. Finally, we compensate for domain shift and enable zero-shot generalization to unseen dataset characteristics. Our findings highlight the effectiveness of sparse dictionary learning and the use of SAES for improving interpretabilityy
We assumed flat latent space and relied on vector arithmetic. We leave detailed investigation of how SAE parameterization might help address potential latent space curvature for future work. Furthermore, we have empirically shown connection between neural collapse and the structure of the latent space for using control vectors, although our analysis remained limited to probing accuracy and class-distance normalized variance. Our findings enable new applications in robotics and self-driving. We identify safety validation in latent space as a promising direction, particularly for end-to-end driving. Using control vectors to modify internal representations and adjust trajectories via instruction- based inputs is also valuable application. Finally, future work can explore the use of other embedding methods (e.g.. Schneider et al. (2023)), as well as incorporate features from other modalities by capturing both static and dynamic scene elements.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
The research leading to these results is funded by the German Federal Ministry for Economic Affairs and Climate Action within the project "NXT GEN AI METHODS", The authors thank anonymous reviewers for their valuable feedback and insightful suggestions.
REFERENCES
Guillaume Alain and Yoshua Bengio. Understanding intermediate layers using linear classifier probes. In ICLR,, 2017.
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: Visual Language Model for Few-Shot Learning. NeurlPS, 2022.
Sanjeev Arora, Yuanzhi Li, Yingyu Liang, Tengyu Ma, and Andrej Risteski. Linear Algebraic Structure of Word Senses, with Applications to Polysemy. Transactions of the Association for Computational Linguistics, 6:483-495, 2018.
Georgios Arvanitidis, Lars Kai Hansen, and Søren Hauberg, Latent Space Oddity: the Curvature of Deep Generative Models. In ICLR, 2018
Omri Azencot, N Benjamin Erichson, Vanessa Lin, and Michael Mahoney. Forecasting Sequential Data Using Consistent Koopman Autoencoders. In ICML,, 2020.
Yasaman Bahri, Ethan Dyer, Jared Kaplan, Jachoon Lee, and Utkarsh Sharma. Explaining neural scaling laws. Proceedings of the National Academy of Sciences, 2024.
Federico Barbero, Andrea Banino, Steven Kapturowski, Dharshan Kumaran, João GM Araújo, Alex Vitvitskyi, Razvan Pascanu, and Petar Veličković. Transformers need glasses! Information over-squashing in language tasks. In NeurlPS, 2024.
Iz Beltagy, Matthew E. Peters, and Arman Cohan. Longformer: The Long-Document Transformer. arXiv:2004.05/500 2020.
Ido Ben-Shaul, Ravid Shwartz-Ziv, Tomer Galanti, Shai Dekel, and Yann LeCun. Reverse Engineering Self-Supervised Learning. In A. Oh, T. Neumann, A. Globerson, K. Saenko, M. Hardt, and
S. Levine (eds.), NeurIPS, 2023.
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation Learning: A Review and New Perspectives. IEEE Transactions Pattern Analysis and Machine Intelligence, 35(8):1798-188) 2013.
Simon Benhamou. How to reliably estimate the tortuosity of an animal's path: straightness, sinuosity, or fractal dimension? Journal of Theoretical Biology, 2004.
Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis, Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models. CVPR, 2023.
Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nicholas I. Turner, Cem Anil, Amanda Askell, et al. Towards Monosemanticity: Decompos- ing Language Models With Dictionary Learning. Transformer Circuits Thread, 2023.
Tim Brooks, Aleksander Holynski, Alexei A Efros. InstructPix2Pix: Learning to Follow Image Editing Instructions. In CVPR, 2023.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, et al. Language Models are Few-Shot Learners. In NeurlPS, 2020.
Hila Chefer, Oran Lang, Mor Geva, Volodymyr Polosukhin, Assaf Shocher, michal Irani, Inbar Mosseri, and Lior Wolf. The Hidden Language of Diffusion Models. In ICLR, 2024.
Xinlei Chen and Kaiming He. Exploring Simple Siamese Representation Learning. In CVPR, 2021.
11

---

Published as a conference paper at ICLR 2025
Arthur Conmy and Neel Nanda. Activation steering with SAEs. Alignment Forum Blog Post, 2024. URL https: : /tt:://wwwww wwtp:wwwwwll.......mm..........w gnment forum. wrt.umorr///a/.oooooooaa postspossss.cc wwtpssss/caaoaaaaabbccoooaa BCtta/bbbbbbbbbeeyyygc progress-update- 1. Progress Update #1 from the GDM Mech Interp Team.
Hoagy Cunningham, Aidan Ewart, Logan Riggs, Robert Huben, and Lee Sharkey. Sparse Autoen- coders Find Highly Interpretable Features in Language Models. In ICLR, 2024.
Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, et al. Toy models of superposition. Transformer Circuits Thread, 2022.
Scott Ettinger, Shuyang Cheng, Benjamin Caine, Chenxi Liu, Hang Zhao, Sabeek Pradhan, Yuning Chai, Ben Sapp, Charles R Qi, Yin Zhou, et al. Large Scale Interactive Motion Forecasting for Autonomous Driving: The Waymo Open Motion Dataset. In ICCV, 2021.
Daocheng Fu, Xin Li, Licheng Wen, Min Dou, Pinlong Cai, Botian Shi, and Yu Qiao. Drive Like Human: Rethinking Autonomous Driving with Large Language Models. In WACV, 2024.
Tomer Galanti, András György, and Marcus Hutter. On the Role of Neural Collapse in Transfer Learning. In ICLR, 2022.
Leo Gao, Tom Dupre la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Ilya Sutskever, Jan Leike, and Jeffrey Wu. Scaling and evaluating sparse autoencoders. In ICLR, 2025.
Lin Guan, Karthik Valmeekam, Sarath Sreedharan, and Subbarao Kambhampati, Leveraging Pre- trained Large Language Models to Construct and Utilize World Models for Model-based Task Planning. NeurlPS, 2023.
Juyeon Heo, Christina Heinze- -Dezl, Oussama Elachqar, Kwan Ho Ryan Chan, Shirley You Ren, Andrew Miller, Udhyakumar Nallasamy, and Jaya Narain. Do LLMs "know" internally when they follow instructions? In ICLR, 2025.
Tianyu Hua, Wenxiao Wang, Zihui Xue, Sucheng Ren, Yue Wang, and Hang Zhao. On feature decorrelation self-supervised learning. In ICCV, 2021.
Yidong Huang, Jacob Sansom, Ziqiao Ma, Felix Gervits, and Joyce Chai. DriVLMe: Enhancing LLM--based Autonomous Driving Agents with Embodied and Social Experiences. In CVPR Workshop on Vision and Language for Autonomous Driving and Robotics, 2024.
Andrew Jaegle, Felix Gimeno, Andy Brock, Oriol Vinyals, Andrew Zisserman, and Joao Carreira. Perceiver: General perception with iterative attention. In JCML, 2021.
William Johnson and Joram Lindenstrauss. Extensions of Lipschitz maps into Hilbert space. Contemporary Mathematics, 26:189-206 01 1984. doi: 10. 10..1900ccomm/06/777740000
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv:2001. 003361 2020.
Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, al. Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV). In ICML, 2018
Diederik P. Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. In ICLR, 2015.
Elyor Kodirov, Tao Xiang, Zhenyong Fu, and Shaogang Gong. Unsupervised domain adaptation for zero-shot learning. In ICCV, 2015.
Kai Konen, Sophie Jentzsch, Diaoulé Diallo, Peer Schütt, Oliver Bensch, Roxanne El Baff, Dominik Opitz, and Tobias Hecking. Style vectors for steering generative large language models. In EACL. 2024.
Stepan Konev. MPA: MultiPath+ Based Architecture for Motion Prediction. arXiv preprint arXiv:2206. 10041, 2022.
12

---

Published as a conference paper at ICLR 2025
Yen-Ling Kuo, Xin Huang, Andrei Barbu, Stephen G McGill, Boris Katz, John Leonard, and Guy Rosman. Trajectory prediction with linguistic representations. In ICRA, 2022.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual Instruction Tuning, NeurIPS, 2024.
Ilya Loshchilov and Frank Hutter. Decoupled Weight Decay Regularization. In ICLR, 2019.
Bethany Lusch, Nathan Kutz, and Steven L Brunton. Deep learning for universal linear embeddings of nonlinear dynamics. Nature Communications, 9()):4950, 2019.
Jiageng Mao, Junjie Ye. Yuxi Qian, Marco Pavone, and Yue Wang. A Language Agent for Au- tonomous Driving. In Conference on Language Modeling (COLM), 2024.
Tomáš Mikolov, Wen-tau Yih, and Geoffrey Zweig. Linguistic Regularities in Continuous Space Word Representations. In Proceedings of the North American Association for Computational Linguistics, 2013.
Marco Mistretta, Alberto Baldrati, Marco Bertini, and Andrew D Bagdanov. Improving zero-shot generalization of learned prompts via unsupervised knowledge distillation. In ECCV, 2024.
Nigamaa Nayakanti, Rami Al-Rfou, Aurick Zhou, Kratarth Goel, Khaled S Refaat, and Benjamin Sapp. Wayformer: Motion Forecasting via Simple & Efficient Attention Networks. In ICRA, 2023.
Long Ouyang. Jeffrey Wu, Xu Jiang. Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, et al. Training language models to follow instructions with human feedback. NeurlPS, 2022.
Vardan Papyan, X. Y. Han, and David I.. Donoho. Prevalence of Neural Collapse during the terminal phase of deep learning training. PNAS, 2020.
Jeffrey Pennington, Richard Socher, and Christopher D Manning. GloVe: Global Vectors for Word Representation. In Conference on Empirical Methods in Natural Language Processing (EMNLP), 2014.
Michael Psenka, Druv Pai, Vishal Raman, Shankar Sastry, and Yi Ma. Representation Learning via Manifold Flattening and Reconstruction. Journal of Machine Learning Research, 25(132):1-47, 2024.
Maria Abi Raad, Arun Abuja, Catarina Barros, Frederic Besse, Andrew Bolt, Adrian Bolton, Bethanie Brownfield, Gavin Buttimore, Max Cant, Sarah Chakera, al. Scaling Instructable Agents Across Many Simulated Worlds. arXiv:2404. 10179, 2024.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In /CML, 2021.
Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Tom Lieberum, Vikrant Varma, Janos Kramar, Rohin Shah, and Neel Nanda. Improving Sparse Decomposition of Language Model Activations with Gated Sparse Autoencoders. NeurIPS, 2024a.
Senthooran Rajamanoharan, Tom Lieberum, Nicolas Sonnerat, Arthur Conmy, Vikrant Varma, János Kramár, and Neel Nanda. Jumping Ahead: Improving Reconstruction Fidelity with JumpReLU Sparse Autoencoders. arXiv:2407. 140.3355 2024b.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. ICML, 2021.
Salah Rifai, Pascal Vincent, Xavier Muller, Xavier Glorot, and Yoshua Bengio. Contractive Auto- Encoders: Explicit Invariance During Feature Extraction. In ICMI, 2011.
Nina Rimsky, Nick Gabrieli, Julian Schulz, Meg Tong. Evan Hubinger, and Alexander Matt Turner Steering Llama 2 via Contrastive Activation Addition. In Annual Meeting of the Association for Computational Linguistics (ACI.), 2023.
13

---

Published as a conference paper at ICLR 2025
Victor Sanh, Albert Webson, Colin Raffel, Stephen H Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler. Teven Le Scao, Arun Raja, et al. Multitask Prompted Training Enables Zero-Shor Shot Task Generalization. In ICLR, 2022.
Steffen Schneider, Jin Hwa Lee, and Mackenzie Weygandt Mathis. Learnable latent embeddings for joint behavioural and neural analysis. Nature, 617(7960):3603366 2023.
Ari Seff, Brian Cera, Dian Chen, Mason Ng, Aurick Zhou, Nigamaa Nayakanti, Khaled S Refaat, Rami Al-Rfou, and Benjamin Sapp. MotionLM: Multi-Agent Motion Forecasting as Language Modeling. ICCV. 2023.
Hao Shao, Yuxuan Hu, Letian Wang, Steven L. Waslander, Yu Liu, and Hongsheng Li. LMDrive: Closed-Loop End-to-End Driving with Large Language Models. In CVPR, 2024.
Mohit Shridhar, Lucas Manuelli, and Dieter Fox. CLIPort: What and Where Pathways for Robotic Manipulation. In CoRL, 2021.
Chonghao Sima, Katrin Renz, Kashyap Chitta, Li Chen, Hanxue Zhang, Chengen Xie, Ping Luo, Andreas Geiger, and Hongyang Li. DriveLM: Driving with Graph Visual Question Answering. In ECCV, 2024.
Nishant Subramani, Nivedita Suresh, and Matthew Peters. Extracting Latent Steering Vectors from Pretrained Language Models. In Findings of the Association for Computational Linguistics (ACL), 2022.
Shuhan Tan, Boris Ivanovic, Xinshuo Weng, Marco Pavone, and Philipp Krachenbuchl. Language Conditioned Traffic Generation. In CoRL, 2023.
Guy Tever, Brian Gordon, Amir Hertz, Amit H. Bermano, and Daniel Cohen-Or. MotionCLII: Exposing Human Motion Generation CLIP Space. In ECCV, 2022.
Ilya O Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Un- terthiner, Jessica Yung, Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, et al. MLP-Mixer: An all-MLP Architecture for Vision. In NeurlPS, 2021.
Alexander Matt Turner, Lisa Thiergart, Gavin Leech, David Udell, Juan J. Vazquez, Ulisse Mini, and Monte MacDiarmid. Steering Language Models With Activation Engineering. arXiv:2308. 10248, 2024.
Royden Wagner, Ömer Şahin Tas, Marvin Klemp, Carlos Fernandez, and Christoph Stiller. RedMo- tion: Motion Prediction via Redundancy Reduction. TMLR, 2024.
Haiyang Wang. Chen Shi. Shaoshuai Shi, Meng Lei, Sen Wang, Di He, Bernt Schiele, and Liwei Wang. DSVT: Dynamic Sparse Voxel Transformer With Rotated Sets. In CVPR, 2023a.
#sun-Hsuan Wang, Alaa Maalouf, Wei Xiao, Yutong Ban, Alexander Amini, Guy Rosman, Sertac Karaman, and Daniela Rus. Drive Anywhere: Generalizable End-to-end Autonomous Driving with Multi-modal Foundation Models. arXiv:23 17642, 2023b.
Wayve Technologies Ltd.
Exploring Natural Language for hitps:/ bitps://wayve.ail/ /tw.///eaa/thiiii..... lingo-nann wawttarrraaannaaaaaaaaaaa aut anomous driving/. Blog Post.
LINGO-1:
Autonomous Driving, 2023.
URL
Licheng Wen, Daocheng Fu, Xin Li, Xinyu Cai, Tao MA, Pinlong Cai, Min Dou. Botian Shi, Liang He, and Yu Qiao. Di A Knowledge-DDrin Driven Approach Autonomous Driving with Large Language Models. In ICLR, 2024.
Benjamin Wilson, William Qi, Tanmay Agarwal, John Lambert, Jagjeet Singh, Siddhesh Khandelwal, Bowen Pan, Ratnesh Kumar, Andrew Hartnett. Jhony Kaesemodel Pontes, et al. Argoverse 2: Next Generation Datasets for Self-Driving Perception and Forecasting. arXiv:2301.00493, 2023.
Menghua Wu, Hao Zhu, Linjia Huang, Yiyu Zhuang, Yuanxun Lu, and Xun Cao. High-fidelity 3D Face Generation from Natural Language Descriptions. In CVPR, 2023.
14

---

Published as a conference paper at ICLR 2025
Robert Wu and Vardan Papyan. Linguistic Collapse: Neural Collapse in (Large) Language Models. In NeurlPS, 2024.
Yongqin Xian, Bernt Schiele, and Zeynep Akata. Zero-Shot Learning the Good, the Bad and the Ugly. In CVPR, 2017.
Zhenhua Xu, Yujia Zhang, Enze Xie, Zhen Zhao, Yong Guo, Kwan- Yee K. Wong, Zhenguo Li, and Hengshuang Zhao. DriveGPT4: Interpretable End-to-End Autonomous Driving Via Large Language Model. IEEE Robotics and Automation Letters, ((10)8186-8199) 2024.
Lvmin Zhang, Anyi Rao, and Mancesh Agrawala. Adding Conditional Control to Text-to-Image Diffusion Models. In ICCV, 2023a.
Zhejun Zhang, Alexander Liniger, Christos Sakaridis, Fisher Yu, and Luc Van Gool. Real-Time Motion Prediction via Heterogeneous Polyline Transformer with Relative Pose Encoding. In NeurIPS, 2023b.
Xiaoji Zheng, Lixiu Wu, Zhijie Yan, Yuanrong Tang, Hao Zhao, Chen Zhong, Bokui Chen, and Jiangtao Gong. Large Language Models Powered Context-aware Motion Prediction. In IEEF/RSI International Conference on Intelligent Robots and Systems (IROS), 2024.
Ziyuan Zhong, Davis Rempe, Yuxiao Chen, Boris Ivanovic, Yulong Cao, Danfei Xu, Marco Pavone, and Baishakhi Ray. Language-Guin Traffic Simulation via Scene- Diffusion. In CoRL, 2023.
Brianna Zitkovich, Tianhe Yu, Sichun Xu, Peng Xu, Ted Xiao, Fei Xia, Jialin Wu, Paul Wohlhart, Stefan Welker, Ayzaan Wahid, Quan Vuong, Vincent Vanhoucke, al. RT-2: LanguueLe Action Models Transfer Web Knowledge to Robotic Control. In CoRL, 2023.
Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Dombrowski, et al. Representation Engineering:
Xuwang Yin, Mantas Mazeika, Now-Down Approach to AI Transparency. arXfv:2310.01405. 2023.
15

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1 TITLE ORIGIN
The title of our work "words in motion" is inspired by our quantization method using natural language and by common notion in the computer architecture. In computer architecture, word is a basic unit of data for processing unit (e.g., CPU or GPU), In our work, words are classes of motion features that are embedded in the hidden states of motion sequences processed by motion transformers.
A.2 NATURAL LANGUAGE AS AN INTERFACE FOR MODEL INTERACTION
Linking learned representations to natural language and using it as interface for model interaction has gained significant attention (e.g., Radford et al. (2021 I); Alayrac et al. (2022); Liu al. (2024)). Broadly, approaches incorporating language in models can be categorized into four types. We present these approaches along with applications in robotics and self-driving below.
Conditioning. Numerous works natural language condition generative models in diverse tasks such as image synthesis (Ramesh et al., 2021: Zhang et al., 2023a), video generation (Blattmann et al., 2023), and 3D modeling (Tevet et al., 2022; Wu al., 2023). Tan al. (2023); Zhong et al. (2023) generate dynamic traffic scenes based on user-specified descriptions expressed in natural language.
Prompting. Some works language request assistance or information. This includes obtaining explanations of underlying reasoning, and human-centric descriptions of model behavior (Brown al., 2020; Sanh et al., 2022). Kuo al. (2022) generate linguistic descriptions of predicted trajectories during decoding, capturing essential information about future maneuvers and interactions. More recent works employ large language models (LLMs) to analyze driving environments in human-like like manner, providing explanations of driving actions and the underlying reasoning (Xu et al., 2024; Fu al., 2024; Sima al., 2024; Wayve Technologies Ltd., 2023). This offers a human-centric description of the driving environment and the model's 's recision-making capabilities.
interface to interact with models, enabling users to
Enriching. Another line of work leverages LLMs' generalization abilities to enrich context embed- dings, providing additional information for better prediction and planning (Guan et al., 2023). Zheng et al. (2024) integrate the enriched context information of LLMs into motion forecasting models. Wang et al. (2023b) use LLMS for data augmentation to improve out-of- di-diistibuttin generalization. Others use praited LLMs for better generalization during decision-makingg (Mao et al., 2024; Wen et al., 2024; Shao et al., 2024).
Instructing. Natural language can be used issue explicit commands for specific tasks, distinet from conditioning (Ouyang et al., 2022; Brooks et al., 2023). The main challenge is connecting the abstractions and generality of language with environment-grouncee Shridhar et al. (2021) enable robotic control through language- instruction. Zitkovich et al. (2023) incorporate web knowledge, enriching saaion-anguage-acttio models for more generalized task performance. Huang et al. (2024) demonstrate the use of instructions to guide task execution in self-driving, with experiments in simulation environments.
actions (Raad et al., 2024).
Although these works align learned text representations with embeddings of other modalities, in contrast to our work, they do not measure the functional importance of features. To our knowledge, no prior work has explored the mechanistic interpretability of transformers in robotics applications.
A.3 PARAMETERS FOR CLASSIFYING MOTION FEATURES
We classify motion trajectories with a sum less than 15t degrees straitt. When the cumulative angle exceeds this threshold, a positive value indicates ritht direction, while a negative value exceeding the threshold in absolute terms indicates left direction. We classify speeds between 25 km h and 50 km -1 as moderate, speeds above this range as high, those below as low, and For acceleration, we classify trajectories
negative speeds as backwards the integral of speed over time to projected displacement with initial speed is less than 0.9 times. this ratio is greater than 1.1 times, we classify them accelerate ing. For all other values, classify the trajectories as having constant speed. We determine all threshold values by analyzing the distribution of the dataset.
inng if
16

---

Published as a conference paper at ICLR 2025
53.32
51.48
02.19
# Laato
27.88
18.49
moderate
31.17
26,09
decelerating
straight
11.69
48.62
4.16
high
16.76
11.65
12.66
constant
right
65.32
12.26
24.17
backwards
52.08
10:77
4.47
acce_erating
left
22.99
(a) Speed
(b) Acceleration
(c) Direction
Figure 8: Distributions of our motion features for the Argoverse Forecasting AV2F) and the
Waymo Open Motion (abbr. Waymo) datasets. All numbers are percentages.
Figure 8 presents the distribution of motion subclasses across the datasets. Both datasets predomi- nantly capture low-speed scenarios, with 62% of Waymo instances and 53% of AV2F instances falling into this category. Furthermore, a notable difference lies in the proportion of stationary vehicles, with AV2F exhibiting significantly higher percentage (51%) compared to Waymo (28%). The Waymo dataset predominantly features vehicles with constant acceleration (65%) and traveling straight (49%), while the AV2F dataset has higher proportion of accelerating instances (52%). Additionally, AV2F has much larger proportion of instances involving backward motion (24%) compared to Waymo (4%). This disparity in motion characteristics highlights that the two datasets capture different driving environments and scenarios, with Waymo potentially focusing on highway or structured urban driving, while AV2F contains more diverse traffic situations.
A.4
OF MULTIMODAL MOTION TRANSFORMERS
We study multimodal motion transformers (Nayakanti et al., 2023; Wagner et al., 2024; Zhang et al., 2023b), which process motion, lane and traffic light data. The archtterchie of these models is shown in Figure 9. These models generate motion M, map Kj. and traffic light 7i embeddings using MLPs. Modality-ppecifie encoders aggregate information from multiple embeddings with attention mechanisms (e.g.. across multiple past timesteps for motion embeddings). Afterwards, in the motion decoder, learned motion queries Q (i.c., form learned anchors) cross-attend M, K, and T. Finally, an MIP projects the last hidden state of Q into multiple motion forecasts, which are represented as Gaussians for future positions in bird's-eye- e-eee-ieww, along with their associated confidences. The differences between the models lie in the type of attention and fusion mechanisms they employ, as well as the used reference frames.
Motion MLP
Mr
Motion encoder
Motion decoler
F1
Map MLP
Map encoder
MLP
Traflic light MLP
T
Traftic light
Qu
Q
Q
Figure 9: Motion transformer meta architecture of RedMotion, Wayformer and HPTR.
A.5 EARLY, ITERARCHICAL AND LATE FUSION IN MOTION ENCODERS
Fusion types for motion transformers are defined based on the information they process in the first attention layers. In early fusion, the first attention layers process motion data of the modeled agent, other agents, and environment context. In hierarchical fusion, they process motion data of the modeled agent, and other agents. In late fusion, they exclusively process motion data of the modeled agent.
17

---

Published as a conference paper at ICLR 2025
A.6 FEATURE CORRELATION
Combined Heatmap of Spearman Correlation (Color) and Distances (Annotations)
low
090
0:547
00001
0.453
8:221
6,200
0:422
0.348
0-274
120
0.379
0.100
moderate
01000
0:278
0 989
0.438
0.353
100
(.00
6.1-33
0.987
0-330
0.350
0.318
0.8
high
0.000
0.993
0.072
0.367
0.067
C.003
0.005
0.383
0.353
0.427
K.050
0,419
rackwards
0.000
0.400
0.601
(.207
0.975
0.716
0.835
R.021
0.080
0.459
0.6
decolerae ing
6.377
6,487
0.005
0.415
0.430
8.204
0,415
0.310
Spearman Correlation
constant
0,200
0,092
(1300)
0.244
0.245
0.425
0.389
0.4
acceleration
acce eeat ing
CL000
(.000)
0.204
0.:17
11205
KU100
0.375
0.312
stationary
0.002
0.827
0.600
0.729
0:539
0.2
straight
(00)
01
196
12121
0.000
0.200
right
0-737
0.166
left
vehi cle
-0.2
pedestrian
oyelist
moderate
low
backwa rds
high
dece wellerrttin ing
constant
acpe lerating
stationary
straight
right
left
vehi c.
pedest rian
cyclist
Figure 10: Heatmap representing Spearman correlation between feature cluster means for the Waymo Open Motion dataset. The values in the matrix indicate pairwise distances between clusters, normalized by the largest distance.
Combined Heatmap of Spearman Correlation (Color) und Distances (Annotations)
1
1-455
modorate
0.550
260
0.500
0.900
0.009
0.230
0.506
high
0.000
0.520
0.436
0.700
0.001
0.685
0.564
0.545
0.731
0.8
backwards
0,000
0..37
0,772
0.200
0.772
0,709
0J350
0.010
dece decelerating
0.647
0,274
0.627
0.803
02252
0.320
0.425
0.6
Spearman Correlation lationiion
constant
0,390
6.741
(.007
156
0.402
0.454
0.446
accelerating
C.203
0.394
0.3577
0.549
0.307
0.4
otationary
6.127
0.007
0.21
0:540
0.400
0.111
straight
000
o 177
0.269
0.688
0.440
0.2
right
0000
0.350
0.590
0.356
left
0.171
0.506
0.300
vehicle
0000
0.321
0.224
pedestrian
0.361
oyclist
.00
0.2
moderate
low
backwa rds
high
dece lar.ierrting
constand ant
aoce brating
conary
straigm ght
right
left
vehicce
pedeste strian
cyclist
Figure 11: Heatmap representing Spearman correlation between feature cluster means for the Argoverse 2 Forecasting dataset. The values in the matrix indicate pairwise distances between clusters, normalized by the largest distance.
18

---

Published as a conference paper at ICLR 2025
A.7
EXPLAINED VARIANCE
Speed
Acceleration
Direction
Agent
Explained Variance
0.66
Hidden dims 512
0.8
10.08
10.d3
0.2
0.4
10.4
0.03
10 02
00
0.01
10
08
0.022
0.02
0.02
00
0.12
(1 01.08
0.06
0.01
0.03
0.02
D. 0..08
0.04
0.02
0.02
0.01
10
1
2
10
8
910
2
Explained Variance
Hidden dime 256
0.8
0.61
00.08
11.20
0.4
0.2
0.
1.00
10.0
0..02
00001
0.001
10.01
0.11
10.03
D0.02
10.02
0..01
0.12
0.09
0.05
8.05
1001
1.004
10.03
0.08
0.04
0.03
D.02
02
8
10
9
9010
9.10
9
110
Explained Variance
0..07
D
10.37
Hidden 128
0,36
0.1
0.2
0.
05
0.02
0.01
0.01
0.01
10.1
10.02
0..02
0.001
0.01
O. 0.
0.088
0.05
0.04
10.01
0.3
10.03
0.08
00.04
10.03
0..0
0.01
10
3
5
9
11
1
9
2
4
5
Explained Variance
0.08
0.64
0.53
Hidden dim: 64
0.6
0.31
0.4
1600
0.01
02
0.02
0.01
10.1
0,00
10:03
0.002
0.02
0..01
0.00
0.06
1004
0.03
0.03
0.08
0.055
10.03
0..02
10.02
10
9
010
Explained Variance
0.08
0.57
Hidden dim: 32
0.51
0.6
0.35
0.4
0.06
05.05
1002
10.02
10
00.02
10.01
10.10
0.03
D.00
0.02
10.01
0.00
0.07
0.03
0004
1.00
1.02
0.13
0.05
00.04
0.02
0.01
1234
S
Explained Variance
03
0.65
b.31
Hidden dinc
0.6
0.30
0.4
0.09
0.07
03
0.22
0.02
00.01
0.11
OF
D2
0.02
0.01
10.11
O. 01
0 0G
0.05
1.04
10.03
0.13
0.05
0..04
D 02
0.01
Index
Index
Index
Index
Figure 12: Explained variance for SAE across hidden latent dimensions 512, 256, 128, 64, 32, 16.
Speed
Acceleration
Direction
Explained Variance
0.688
0.67
0.6
0.4
0.08
.G
03
D.02
0.01
D.01
0.000
0.
0 04
0.02
0.01
D.0L
0.15
0.23 2
05
0
02
0.01
0.11
08
0.02
0.01
0..00
2
Index
Index
Index
Indes
Figure 13: Explained variance for Plain-PCA.
19

---

Published as a conference paper at ICLR 2025
A.8 ADDITIONAL QUALITATIVE RESULTS
Figure 14 shows a qualitative example for our direction control vector from the Argoverse Forecast- ing dataset. The left control leads to accelerated future motion, which is consistent with the common driving style of slowing down in front of a curve and accelerating again when exiting the curve. A strong right control makes the focal agent stationary. We hypothesize that cancels out the actually driven left turn, resulting in virtually stationary past.
(a) Default motion forecast
(b) Left control r 10
(c) Right control r 100
Figure 14: Modifying hidden states to control a left turning vehicle. In subfigure (b) and (c), we apply our left-direction control vector and right-direction control vector. The focal agent is highlighted in orange, dynamic agents are blue, and static agents are grey. Lanes are black lines and road markings are white lines.
(a) = -00
(b) T
(c) r - -5
(d) r = 25
(c) Default motion forecast
(f) r 25
g 50
(h) r = 75
( = 100
Figure 15: Control vectors applied all agents, showing consistent multi-agent behavior.
20

---

Published as a conference paper at ICLR 2025
A.9 COMPARISON OF CONTROL VECTORS USING PLAIN PCA AND SAE ACROSS VARIOUS SPARSE INTERMEDIATE DIMENSIONS
Table 6: Comparison of control vectors obtained with and without SAES across sparse intermediate dimensions (512, 256, 128, 64, 32, 16). The tables on the left represent the pairwise angular distances SAE model, whereas those the right represent the angular
between control vectors of the distances between control vectors of SAE and those derived from plain PCA. The control vector with a sparse intermediate dimension of 128 achieves the highest overall similarity.
speed
direction
agent
speed
agent
apoed
0.0
10.2
121.8
7.6
speed
20.7
28.6
123.5
25.4
acceleration
120.7
acceleration
19.1
23.0
128.5
direction
direction
agent
agent
SAE-256 & SAE-256
aucelemation
direction
Pluin PCA SAE-256
accelenstion
direction
speed
0.0
9.9
120.9
7.9
speed
21.3
26.8
121.8
22..3
acceleration
0.0
123.7
7-2
acseliration
20.3
21.0
128.7
18.7
disection
0.0
126.3
dircetion
114.7
116.9
13.7
120.1
20.3
23.
130.2
18.7
NAE-128 & SAR-128
spesd
accelennion
direction
agent
Plain PCA SAF-125
spoad
accelontionn
dicection
agent
speed
speed
acceleration
acceleration
19.2
direction
0.0
125.8
direction
115.2
117.1
12.1
120.5
agent
0.0
agent
19.5
21.8
130.4
17.1
Spead
direction
agent
SAE-64
sposd
accclonation
agent
speed
0.0
9.7
121.0
8.00
speed
18.1
23.7
124.7
19.3
acceleration
accoloration
19.3
128.0
direction
direction
120.5
agent
agent
SAL-32 SAE-32
speed
acceleration
direction
Pluin SAE-32
acceteration
direction
speed
0.0
9.8
120.3
8.3
speed
14.7
18.8
126.0
15.5
accoleration
0.0
122.8
7.0
acceleration
18.0
15.5
130.3
14.1
discetiion
0.0
125.8
direction
114.4
136.9
10.9
120.22
18.1
SAK-16 & SAE-16
speed
seceleration
disectios
Egent
Plain PCA & SAR-16
speed
sccelevation
dicecton
agent
spoed
9.5
126.6
acceleration
scceleration
direction
110.2
131.9
apeit
0.0
agent
28.0
28.8
131.0
22.5
A. LOSS METRICS FOR SAES
We report the results for the epoch with the lowest total loss f2-loss x 10-4, e flsss.. Note that the & reconstruction loss is computed as the average of all partial losses for all embeddings, while the & sparsity loss is computed as the all partial losses.
Table 7: Loss metrics for SAES
sparse intermediate dimensions, trained for 10.000 epochs.
Dim
Best epoch
Total loss
E reconstruction loss
sparsity loss
512
9805
4.01
1.52
8270.70
256
9845
3.72
1.38
7823.98
128
9820
4.14
1.6G
8608.95
64
9348
1.56
1.89
88911997
9864
16
9956
17.44
13.37
13.576.57
21

---

Published as a conference paper at ICLR 2025
A.11 INFERENCE LATENCY
Table 8 shows inference latency measurements of a RedMotion model on the Waymo Open Motion dataset with and without activation steering with our control vectors. Our activation steering adds only about ms to the total inference latency. Since most datasets are recorded at 10 Hz (e.g.. Wilson al. (2023); Ettinger al. (2021)), is common define the threshold for real-time til-timme capability of self-driving stacks as 1100 mss Considering the inference latency of recent 3D perception models (e.g., approx. 40 ms for Wang et al. (2023a)), which must be called before motion forecasting, activation steering should not add significantly to the forecasting latency.
Table 8: Inference latency without and with activation steering with our control vectors. We measure the inference latency on one A6000 GPU using the PyTorch Lightningg profiler and plain eager execution. We report the mean of 1000 iterations per configuration for the predict_sted including pre- and post-pne post-processing.
Activation steering
Focal agents
Inference Intency
True
51.08 ms
A.12 CONTROL VECTORS ACROSS MODULES IN SPARSE AUTOENCODERS
Table 9: Generating control vectors for hidden states of different modules. Control vectors for speed generated earlier modules achieve lower linearity scores for activation steering. Linearity measures for controlling: Pearson correlation coefficient, coefficient of determination (R²), and straightness index.
Autoeneoder
Module m
Pearson
S-idx
SAE-128
2
0.993
0.984
0.988
SAE-123 128
SAE-128
0
0.959
0.654
0.933
A.13 SENSITIVITY ANALYSIS FOR VARIOUS SPEED THRESHOLDS
Table 10: Generating speed control vectors with different thresholds for low and high speed. Decreasing the threshold for high speed marginally improves linearity scores, while increasing the threshold for low speed significantly worsens the linearity scores.
Autnencuder
Low speed
High speed
Pearson
S-idx
SAF-128
25 km h1
>> 50 km h-1
0.993
0.984
0.988
SAE-128
<< 25 km
25 50 km h
0.994
0.987
0.989
SAE-128
25 50 km h
>> 50 km h
0.355
-0.731
0.533
A.14 14 CHOOSING RANGE OF RELATIVE CHANGES IN FUTURE SPEED
Given the large range of scenarios in the Waymo dataset, we focus on relative speed changes within a range of +50% capture the most relevant speed variations (see Figure in Ettinger et al. (2021)). Considering the approximated mean and standard deviation for each agent type (vehicles: μ 12ms a z 5 ams" 1 pedestrians: μ a 1.5 1 R 3 m1) the +500% range corresponds to speeds within approximately lo of the mean for each agent type.
0.7 ms- and cyclists: μ = 7ms!!
22

---

Published as a conference paper at ICLR 2025
A.15 EVALUATING KOOPMAN AUTOENCODER
A consistent Koopman autoencoder (KoopmanAE) (Azencot et al., 2020) is a bidirectional method that models temporal dynamics between embeddings. The learned latent space approximates Koopman-invariant space where dynamics evolve linearly. Adapted to the SAE configurations, we train an encoder and decoder with one layer each and a latent dimension of 128. We use learned linear projections to decode Koopman operator approximations C, E R 12128x12 from intermediate representations. For the first 10 time steps, we encode the embedding and predict the next embedding using C, while for the last 10 time steps, we encode the embedding and predict the previous embedding using D.
Afterwards, we use the KoopmanAE instead of an SAE to fit control vectors (see Section 3.3). Figure 16 shows the calibration curve for the resulting speed control vector. The range of values approximately x smaller than for the SAE-based control vector shown in Figure 7b.
Calibration curve
Linear reference
speed change
25
0-
-25
-10-2
Figure 16: Calibration curve for a speed control vector optimized using the KoopmanAE. The range of values is significantly lower than for plain PCA and SAE-based control vectors (cf. Figure 7), yielding lower R² scores as shown in Table 3.
A.16 PLAIN PCA-BAR BASED SPEED CONTROL VECTOR FOR THE AV2F DATASET
Figure 17 shows the calibration curve for a plain PCA-based speed control vector for a RedMotion model trained on the AV2F dataset (cf. Section 5.4). To suppress the effects of different trajectory lengths, we trained this model on configuration of the AV2F dataset with the same trajectory lengths as in the Waymo dataset. cannot reduce the speed by more than 3%. Therefore, we center the range of values instead of the range of relative speed changes to compare this control vector with those for the Waymo dataset.
contrast the control vectors for the Waymo dataset, this control vector
50
Calibration
Lincar reference
spoed change
25
Figure 17: Calibration curve for plain PCA-based speed control vector for the AV2F dataset. In contrast to the control vectors for the Waymo dataset, this control vector cannot reduce the speed by more than 3%.
Moreover, used the low and moderate speed classes to fit this control vector, the low and high speed classes did not yield good results. We hypothesize that this due to the different distributions of the datasets shown in Figure 8.
23

---

Published as a conference paper at ICLR 2025
A.17 JUMPRELU COMPENSATES ACTIVATION SHRINKAGE CONVSAES
Calibration
30-
Calibration curve
Linexr reference
relalive spood change innngninn
relaive
-20
(a) ConvSAE 64
(b) ConvSAE JumpReLU
Figure 18: Ju compensates activation shrinkage as reflected in smaller range of values for the same range relative speed changes.
The range of temperatures is much higher for the ConvSAE than for the JumpReLU version of this sparse autoencoder (ConvSAE = 64 JumpReLU), We hypothesize that this due to activation shrinkage (Rajamanoharan et al., 2024b). Therefore, the JumpReLU configuration of this SAE-type leads to a significantly smaller 7 range, which in turn leads to higher R² scores (see Table 3).
A.18 MOTION FORECASTING METRICS
Following Wilson et al. (2023); Ettinger al. (2021), we use the average displacement error (mi- nADE), the final displacement еπтог (minFDE), and their respective Brier variants, which account for the predicted confidences. Furthermore, we compute the miss rate, and overlap rate evaluate motion forecasts. All metrics are computed using the minimum mode. Accordingly, the metrics for the trajectory closest to the ground truth are measured.
A.19 NEURAL COLLAPSE
Neural collapse metries capture structural patterns in feature representations, focusing clustering, geometry, and alignment. Class-distance normalized variance (CDNV), also referred to as "NC1", quantifies the degree to which features form class-wise clusters by measuring the variance within feature clusters of each class relative to the distances between class means. CDNV provides a robust alternative to methods that compare between- and within-cluster variation for assessing feature separability (Galanti et al., 2022).
02 +
03
CCV
Vc Vc d
2 16 12.1 /
24


Reviews:
[{"review_id": "rwUqBHskmW", "reviewer": "Reviewer_6xxM", "summary": "The paper focuses on the interpretability and control of transformer-based motion forecasting models. The authors aim to interpret hidden states of motion transformers and control them during inference. They use neural collapse to assess whether human-interpretable features are embedded within hidden states. The authors fit control vectors for steering during inference for the interpretable features. They further finetune these vectors using sparse autoencoders. They show that enforcing sparsity leads to a more linear relationship between the strength of the steering and interpretable features. They apply this approach to various motion forecasting models with different fusion mechanisms and environment representations. Finally, they address domain shifts using the control vectors and enable zero-shot generalization.", "questions": "1. In the introduction, the authors claim that they identify that interpretable features are embedded in hidden states of transformer-based models. Since this is a well-known observation, the reviewer is curious about how the work adds to the existing knowledge in the mechanistic interpretability literature.\n2. In the conclusion, the authors claim that they take a significant step towards the mechanistic interpretability and controllability of the transformer models. This is a bold claim - how do the results from the paper justify the claim and add to the pre-existing knowledge in the broader transformer interpretability literature?\n3. How do different feature quantization thresholds affect results? The rationale for these specific thresholds should be better justified by showing the dataset statistics. The authors mention the choice was based on insights from Seff et al., 2023 - what were the specific insights the authors used for their choice of classes?\n4. In Section 3.2, the authors claim to use the mean of the standard deviation to measure collapse - is this correct or should this metric be just the standard deviation?`\n5. For the control vectors found using pca, what is the variance explained in each case?\n6. The implications of neural collapse on the post-training processes of a model like transferability and generalization is still an open question (Vignesh Kothapalli, 2024). Can the authors better justify the use of neural collapse to show the validity of their probe accuracies?\n7. How would the control effects change when combining multiple control vectors (e.g., speed + direction)?\n8. How do the control vectors found using one approach compare to the other, in terms of their cosine similarity?\n9. What was the intuition behind choosing the temperature range of [-50,50]?\n10. Can the authors comment on how to extend their method to continuous control features rather than discrete classes as they have used in the paper?\n12. Will the authors release both code and pre-trained models, including the trained sparse autoencoders?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper applies interpretability approaches to transformer-based models beyond the natural language domain and propose neural collapse as a metric of interpretability.\n\n2. The authors use sparse autoencoder-based steering for improving control vector linearity for motion control and zero-shot generalization.\n\n3. The authors have done significant work to apply the proposed method to multiple motion forecasting architectures and datasets.", "weaknesses": "1. The paper relies heavily on neural collapse as a measure for the model learning clusters of interpretable features. The authors should verify the feature clusters are indeed distinct by comparing the within-class and between-class variance.\n2. The L1 sparsity in the training objective is known to induce feature shrinkage and result in poor reconstructions (Wright and Sharkey, 2024). The authors should compare l1, l2, and reconstruction loss for other SAE architectures that do not induce feature shrinkage such as TopK or JumpReLU SAEs, and use the best-performing architecture to finetune control vectors and show their impact on control linearity. If training new SAEs seem out-of-scope, then they should report the % loss recovered from the SAEs and ensure that they are in an acceptable range.\n3. The authors should test their approach on other naturally occurring domain shifts like various traffic densities, scenes with different weather conditions, etc.\n4. Details on the training of the sparse autoencoders such as choice of learning rate and schedule, batch size, number of epochs, etc. as missing.\n5. Other minor comments:\n           a. The authors give a brief explanation of neural collapse in paragraph 2 of the introduction but do not explicitly mention the term.\n           b. Along similar lines, the authors use terms like domain shift and fusion mechanisms for the transformer architectures and zero- \n               shot generation, they should be properly defined and contextualized for their specific problem.\n           c. The reference for sparse autoencoders used in Section 4.4 is for the Gated SAEs paper, however, the authors do not use that \n               specific architecture. The reference should be changed appropriately.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4A4bjuXMDx", "reviewer": "Reviewer_FS1W", "summary": "The paper introduces a method to interpret and control transformer-based motion forecasting models by analyzing hidden states through the lens of neural collapse. Using linear probes and control vectors, it maps hidden states to interpretable features, such as speed and direction, enabling insight into the model’s decision-making and the ability to manipulate forecasts without retraining. This approach enhances model interpretability, though its reliance on neural collapse and associated computational demands present some (minor) challenges, and testing other base models beyond SAE vs. PCA would be interesting.", "questions": "See weaknesses, above. I would be happy for the authors to provide commentary on the baselines and computational resources, then I would be happy to raise my score.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- **Interpretability:** The method takes motion transformers, and maps hidden states to human-interpretable features, thus clarifying the model’s decision-making process. In general, interpretability is an important area.\n\n- **Controllability:** Control vectors allow manipulation of specific motion features (e.g., speed, acceleration) at inference time without retraining, enabling intuitive model adjustments.\n\n- **Zero-shot Generalization:** The interpretable control vectors support generalization to unseen scenarios, like different driving styles or environments, compensating for domain shifts in the data.\n\n- **Nice integration across disciplines:** I like using neuro-inspired linear probing and their quantization method using natural language. I think this shows an elegant combination of techniques that can be leveraged to build interpretable systems.", "weaknesses": "- **Reliance on Neural Collapse:** The method's effectiveness depends on well-defined hidden state clusters. If neural collapse is weak, the extracted features and control vectors may be less reliable.\n\n- **Limited Feature Scope:** The approach primarily focuses on basic motion features (e.g., speed, acceleration, direction) and could be expanded to capture more complex motion patterns and interactions with the environment.\n\n- **Limited Baselines**: Currently, they primarily focus on comparisons for the control vectors are to PCA, but do not consider methods that inherently have dynamics, such as temporal convolutional networks (TCNs), nonlinear embedding methods that consider time-series inputs (one prominent example would be Schneider et al. 2023 Nature), or RNNs. While not critical for their argument, they should minimally discuss this in a limitation/discussion section. \n\n- **Computational Overhead:** Using sparse autoencoders increases computational demands, potentially extending training time and resource requirements; it would be ideal to estimate the computational resources used and those needed to utilize their code and models that they note will be released.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0YyxwQmak5", "reviewer": "Reviewer_KZpZ", "summary": "The authors have proposed finding an interpretable feature in the motion transformer and then enabling the control of the prediction based on the found interpretable features. To this end, authors claim that the interpretable feature can be found with the linear probing the hidden state, then use a control vector combined with sparse autoencoder to control the output of the motion transformer.", "questions": "See the weakness above.", "limitations": "", "rating": 3, "confidence": 5, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "The problem is well-motivated: building an interpretable and controllable motion prediction network.", "weaknesses": "1. I think the overall writing needs to be improved in multiple aspects.\n- First, it is hard to understand why the author has focused on motion transformers in the introduction. After reading section 3 (method part), I can understand why the author needs an interpretable method for motion transformers, but in the introduction, it is explained as an application area of their method, not the main focus.\n- While the author suggests they have used neural collapse to measure the human interpretable feature, it is not known what neural collapse is in the introduction. \n\n2. Neural collapse is the most important term in the paper, but it is not defined clearly (or clearly mathematically), so it is unable to understand what the authors are using. Note that the authors have argued that they have used neural collapse as a metric, so we need a mathematical way to measure it. I think the author should define the following in a clear (or mathematical) way.\n- What is \"Neural collapse\"?\n-  What is an \"interpretable feature\"?\n- What is \"motion features\"?\n\n3. While the author mentioned they have used sparse autoencoder (SAE), it is not clear which one they have used [1,2,3]. \n- Which layer did the author train the SAE? SAE is usually trained on the hidden state of the transformer [1,2,3], and the layer is always selected carefully.\n- Can the author report that the SAE is trained well? For instance, the reconstruction error of the SAE. (I think the author mentioned it is reported in the Appendix, but could not find it).\n\n4. Can the author explain the goal of the motion transformer? What is the input of the motion transformer and output of the motion transformer?\n\n5. It is not clear why the author used SAE for the collability. There are several activation steering methods [4,5]. I think it is great to claim the benefit of using SAE over other methods. \n\nOverall, I think the paper needs to improve in terms of writing, especially the introduction and the method part. Moreover, the author could improve the overall experimental details and analysis mentioned above.\n\nReference\\\n[1] Scaling and evaluating sparse autoencoders\\\n[2] Jumping Ahead: Improving Reconstruction Fidelity with JumpReLU Sparse Autoencoders\\\n[3] Improving Dictionary Learning with Gated Sparse Autoencoders\\\n[4] In-context Vectors: Making In Context Learning More Effective and Controllable Through Latent Space Steering\\\n[5] In-Context Learning Creates Task Vectors", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rU8M2ET2aQ", "reviewer": "Reviewer_QCbU", "summary": "The paper introduces a method for extracting interpretable representations in transformer-based motion forecasting models: (i) leverage neural collapse to identify a structured latent space where features form interpretable clusters, (ii) use sparse autoencoders to optimize \"control vectors\" for each interpretable feature. This method is evaluated on three motion forecasting models, demonstrating effective feature manipulation and zero-shot generalization.", "questions": "- How sensitive are the results to the thresholds mentioned in Appendix A.3?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The paper addresses the under-explored area of mechanistic interpretability in motion forecasting, providing a timely study at this intersection.\n- The paper presents thorough quantitative and qualitative experiments, demonstrating the effectiveness of the interpretability method across different models and scenarios.", "weaknesses": "- While the paper is new in connecting mechanistic interpretability with motion forecasting, the technical components (neural collapse and sparse autoencoders) have been studied in LLMs. The claim of `a significant step towards mechanistic interpretability and controllability of transformer models` (L452) seems quite overstated. Further clarification on the unique technical contributions would strengthen the paper.\n- The paper extends interpretability techniques from LLMs to the motion domain; however, the contexts differ fundamentally: LLMs deal with discrete tokens, while motion forecasting operates in a continuous space. The proposed method quantizes continuous variables like speed and acceleration into discrete categories, which might oversimplify the difference. A deeper discussion on the impact of this discretization and the influence of specific parameters (e.g., in L160) would enhance understanding of the challenges and significance of applying mechanistic interpretability to motion forecasting.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gVHpxuwJvn", "reviewer": "Reviewer_5JLb", "summary": "This paper aims to interpret hidden states and control them at inference. It assesses whether human-interpretable features are embedded in hidden states of motion transformers using neural collapse. The latent space properties are used to fit control vectors for each interpretable feature. The control vectors are optimized using sparse autoencoding and enforcing sparsity results in a more linear relationship between control vector temperatures and forecasts.", "questions": "1. Will using a sparse autoencoder significantly increase computational costs?\n\n2. This paper is not well written. Some of the design motivations are not clearly explained.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The paper introduces an approach for interpreting hidden states in motion forecasting models by leveraging neural collapse and linear probing.\n\n2. The paper uses sparse autoencoders to optimize control vectors and enhance the linearity between control temperatures and forecasts.\n\n3. Applying the method to self-driving cars and motion prediction shows its relevance to real-world applications.", "weaknesses": "1. The related work section does not clearly and effectively highlight the similarities and differences between prior research and your work.\n\n2. The study lacks baseline comparisons. Are there existing methods that could be used to evaluate your proposed approach?\n\n3. The paper lacks novelty, as it does not introduce linear probes or control vectors but simply applies these existing methods to motion control.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)