# Action matters for object representation learning from sequences

Mohamed Massamba Sene, Jean-Charles Quinton, Frédéric Armetta, Mathieu Lefort

► To cite this version:

Mohamed Massamba Sene, Jean-Charles Quinton, Frédéric Armetta, Mathieu Lefort. Action matters for object representation learning from sequences. European Conference On Artificial Intelligence (ECAI), Artificial Intelligence and Cognition workshop (AIC), Oct 2025, Bologna, Italy. hal-05324845

HAL Id: hal-05324845

https://hal.science/hal-05324845v1

Submitted on 21 Oct 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Action matters for object representation learning from sequences

Mohamed Massamba Sene<fnref n="1" />, Jean-Charles Quinton<fnref n="2" />, Frédéric Armetta<fnref n="1" /> and Mathieu Lefort<fnref n="1" />

## Abstract

According to the sensorimotor contingencies theory, building from developments in cognitive sciences, action, and especially the rules governing the changes in sensations caused by action, are essential to construct meaningful representation of the environment. However, most of current approaches to self-supervised learning are currently learning static representations from large datasets. In this article, we study how state-of-the-art deep learning methods can learn sensorimotor representations from sequences of interactions in a simplified deterministic Tetris-like environment, partially observable by the model. Especially, we emphasize the role of action to facilitate the emergence of representations related to objects present in the environment (here Tetris manipulable shapes). To that purpose, several model configurations were compared: a multi-task Transformer, a multi-task State Space Model, and ablated variants omitting action either in the input and/or in the pretext tasks. All models were trained in a self-supervised setting using predictive tasks of a masked subpart of the input sequence. Results show that explicitly including actions in the input and in the pretext tasks significantly improves the quality of learned representations, particularly in ambiguous situations. These findings support the relevance of the sensorimotor framework for structuring representation learning.

## Keywords

Sensorimotor theory, Representation learning, Self-supervised learning, Sequence modeling, Tetris

## 1. Introduction

Self-Supervised Learning (SSL) produces rich representations from unlabeled data by learning from pretext tasks and exploiting inherent data structure. In computer vision and natural language processing, SSL has proven highly effective when abundant data is available, learning generic representations that transfer well across tasks [1]. While traditionally applied to static data, SSL also demonstrates abilities in modeling temporal dynamics [2]. These approaches typically rely on static data transformations or masking strategies to create learning objectives without explicit supervision.

In Deep Reinforcement Learning, SSL principles extend to temporal modeling through world models and predictive approaches. Agents learn to predict transformation consequences and environmental dynamics without explicit supervision [3]. Although recent work, such as Interactive VideoGPTs, demonstrates scalable world model learning from sensorimotor sequences [4], these approaches tend to prioritize predictive accuracy over the emergence of exploitable object representations.

Sensorimotor contingency theory, rooted in cognitive sciences, offers a complementary perspective on representation learning. This framework emphasizes that agents structure perception by learning and mastering sensorimotor contingencies, regularities linking actions to sensory changes [5]. Bickhard's interactivism provides a compatible yet complementary theoretical perspective: an "object" as a perceived entity emerges from anticipations in a network of potential interactions [6]. The notion of object is grounded in the ability to maintain conditional and anticipatory interactions, rather than static encoding of entities in the world.

<figure><img src="image_2.png" /></figure>

<figure><img src="image_3.png" /></figure>

<figure><img src="image_4.png" /></figure>

<footnote n="1">¹Univ. Lyon, UCBL, CNRS, INSA Lyon, LIRIS UMR 5205, Villeurbanne, France</footnote>

<footnote n="2">²Univ. Grenoble Alpes, CNRS, Grenoble INP, LJK UMR 5224, Grenoble, France</footnote>
This raises fundamental questions about the link between sensorimotor principles and self-supervised learning, especially: Does explicit action integration in input and learning lead to the emergence of meaningful object representations? We investigate these questions using a simulated discrete and non-stochastic Tetris-like environment where state inference from partial observation of the visual environment remains non trivial, such as determining which shape is being perceived and manipulated. Despite its simplicity, this environment allows us to study how advanced architectures like Transformers [7] and State Space Models [8] enable the emergence of object representations from sensorimotor sequences, examining whether explicit action integration improves object understanding compared to standard passive observation.

After reviewing related work on SSL and sensorimotor theories (Section 2), we describe our simulated Tetris environment and experimental setup (Section 3). We then detail our proposed architectures (Section 4) and present experiments examining how action integration influences representation learning and object emergence (Section 5), concluding with future perspectives (Section 6).

## 2. Related Work

The seminal work from O'Regan and Noe [5] emphasized that perception is constituted by an agent's mastery of sensorimotor contingencies linking action and sensation. In a similar direction, more recent research work explored how such invariants can be extracted to support higher-level representations, such as object-like structures [9]. Mathematical models demonstrated how naive agents can discover both the topology and metric regularities of their environment through sensorimotor prediction [10]. These models rely on simple agents and basic learning mechanisms, providing a proof of concept for learning grounded structure from raw interaction.

Building on these theoretical foundations, practical implementation of sensorimotor learning requires architectures capable of processing the sequential nature of perception-action interactions and capturing temporal dependencies. While RNNs, LSTMs and GRUs address sequence processing, they suffer from vanishing gradients and efficiency limitations. Transformers mitigate these issues through attention mechanisms but incur quadratic complexity in sequence length [7]. State Space Models (SSMs), such as Mamba, achieve comparable performance while reducing computational cost via selective state updates, thus ensuring linear complexity [11]. These architectures can be adapted for self-supervised learning, enabling models to learn representations without labeled data. In reinforcement learning, such architectures can be used to learn world models, i.e., the latent dynamics of the environment through observation-action sequences. However, existing studies rarely evaluate how the incorporation of actions alongside observations impacts the quality of learned representations [12]. We investigate this gap by explicitly modeling perception-action dynamics, moving beyond single-modality setups to assess the groundedness of internal representations under active control.

The challenge of learning meaningful representations from sensorimotor data has motivated significant research in representation learning for control systems. Recent advancements in state representation learning (SRL) for control have extended the taxonomy proposed by Lesort [13], relying on the distinction between forward and inverse models. Latent world models, exemplified by Dreamer [14], leverage forward predictive coding to facilitate control in latent spaces, employing imagination-based planning. In contrast, inverse models focus on learning action-relevant embeddings by predicting actions between latent states [15]. More recent work introduces contrastive or bisimulation-based objectives [16] that align representation quality with policy performance. These methods aim for invariance to irrelevant features while preserving action-relevant dynamics.

Our work builds upon these representation learning advances by integrating the sensorimotor contingency framework with modern sequence architectures. Our approach contributes to this line of work by learning action-conditioned latent dynamics from raw sensorimotor data, without extrinsic rewards, bridging the gap between SRL and SSL. Our approach integrates action into SSL, extracting structure from sensorimotor sequences in a self-supervised manner, similar to world model learning approaches but without external objectives.
### 3. Simulation Environment and Dataset Creation

Our simulation environment, inspired by the Tetris game, provides a structured setting for studying object recognition and manipulation in a discrete space. The seven possible tetrominos (T, I, J, L, O, Z, S), named after their shapes, evolve on a discrete grid. Each tetromino can be manipulated via discrete actions: single-cell movements (left, right, down, up) and 90° rotations around their geometric center (clockwise, counterclockwise). The simulated agent's visual input was deliberately restricted to a 3 × 3 window, so that tetrominos are only partially visible and a reasonable number of actions allows constructing a full picture of the piece (see figure 1). This enforces the model to gather successive pieces of evidence, and to link them through actions associated to transitions between partial observations.

<figure><img src="image_5.png" /><figcaption>**Figure 1:** Full 5 × 5 view of the tetromino and an example of a 3 × 3 observation window (light gray square). The limited local view often results in partial visibility of tetromino shapes.</figcaption></figure>

Rather than analyzing static inputs, we generate random sequences of actions and corresponding partial observations (see figure 2). Sequences are created using an adaptive strategy that balances movement and rotation actions by favoring under-represented actions at each step, which ensures uniform coverage across sequence lengths (1-5 actions), and distributes tetromino types uniformly across the dataset. The dataset is augmented via geometric transformations with coherent action remapping, including horizontal/vertical reflections and 90°, 180°, and 270° rotations around the grid center. These transformations preserve the environment logic while theoretically guaranteeing a perfect balance between actions.

We distinguish between unambiguous sequences (that could only be generated by a unique tetromino) and ambiguous sequences (compatible with multiple tetrominos due to limited observations). This natural ambiguity defines a theoretical upper bound on classification accuracy, as some sequences remain inherently indistinguishable.

<figure><img src="image_6.png" /><figcaption>**Figure 2:** Example sequence showing how discrete actions (up, counterclockwise rotation, left) alter the tetromino position and orientation. The model will only perceive the 3x3 view in light gray.</figcaption></figure>

### 4. Studied Models

Our modular architecture centers on a generic sequential encoder capable of processing sequences alternating observations and actions (as described in section 3). The architecture comprises four main components as shown in Figure 3. Each input sequence begins with an initial observation preceding the first action, followed by up to five action-observation pairs, totaling a maximum of six observations. The sequence is padded with null elements as needed to handle variable lengths.

Each observation is encoded as a 9-dimensional binary vector obtained by flattening the 3×3 grid, where a value of 1 indicates the presence of a visible tetromino block in a given cell, and 0 indicates
<figure><img src="image_7.png" /><figcaption>Figure 3: General architecture of tested multi-task models</figcaption></figure>

its absence. Actions are encoded in ordinal format<fnref n="1" />. Each sequence element is projected into a fixed-dimension representation space via a linear layer, then summed with sinusoidal positional encoding to preserve temporal structure. The encoder block can be instantiated as either a Transformer (multi-head attention layers with feedforward blocks) or Mamba (a State Space Model architecture combining convolutions and sequential operations). Multi-task prediction heads use encoder representations to feed four linear predictive heads designed to capture temporal structure and underlying action/state dynamics: masked observation prediction (inferring a missing observation from sequential context), masked action prediction (inferring a missing action), next observation prediction (anticipating the following observation given the past), and inter-observation action inference (predicting the action linking two successive states). The input contains masked information, depending on the prediction task. Since the inputs differ, the training processes for each task are conducted separately.

From this modular architecture, we derive six configurations to evaluate the impact of action in representation learning. We test three training paradigms: Multi-task models (MT) where all four predictive heads are trained jointly as fully self-supervised representation learners exploiting the full sensorimotor signal; Action-reduced models (AR) designed to assess the impact of action-related signals by removing action prediction heads while keeping actions in input sequences; and Observation-only models (OO) where action information is suppressed from the input data (thus only containing successive observations), with action-related tasks necessarily removed, evaluating learning capacity based solely on observation transitions. Each paradigm is tested with both Transformer (prefix T-) and SSM encoders (prefix S-) to assess the generalizability of our results across different encoders, yielding configurations T-MT, S-MT, T-AR, S-AR, T-OO, and S-OO.

The total loss is computed using task-adapted losses: mean squared error (MSE) for observation prediction tasks to accommodate future non-binary inputs, and cross-entropy (CE) for action prediction tasks. The loss scaling constant $N$ corresponds to the number of tasks on which the model is evaluated (2 instead of 4 for the AR and OO models):

$$ \mathcal{L}_{\text{total}} = \frac{1}{N} \left( \sum_{\text{obs tasks}} \mathcal{L}_{\text{MSE}} + \sum_{\text{action tasks}} \mathcal{L}_{\text{CE}} \right) \quad \text{where } N \in \{2, 4\} \qquad (1) $$

<footnote n="1">¹One-hot encoding of the action yielded poorer results during preliminary testing and was therefore not selected, despite its seemingly better fit to the categorical nature of the action set.</footnote>
<table><caption>**Table 1**
Performance results using normalized evaluation counts</caption><thead><tr><th>Model type<br>Encoder</th><th>MT<br>Transformer</th><th>MT<br>SSM</th><th>AR<br>Transformer</th><th>AR<br>SSM</th><th>OO<br>Transformer</th><th>OO<br>SSM</th></tr></thead><tbody><tr><td>Bits received (M)</td><td>14.4</td><td>14.4</td><td><strong>12.22</strong></td><td>14.5</td><td>13.15</td><td><strong>6.84</strong></td></tr><tr><td>Relative Accuracy</td><td><strong>99%</strong></td><td><strong>97%</strong></td><td>91%</td><td>90%</td><td>89%</td><td>87%</td></tr><tr><td>Accuracy Predictable seq.</td><td><strong>96%</strong></td><td><strong>96%</strong></td><td>95%</td><td>94%</td><td>94%</td><td>92%</td></tr><tr><td>Accuracy Ambiguous seq.</td><td><strong>97%</strong></td><td><strong>94%</strong></td><td>77%</td><td>73%</td><td>65%</td><td>64%</td></tr></tbody></table>

The most striking differences emerge on ambiguous sequences, where action contribution becomes essential. Multi-task models (MT) reach 94-97% relative accuracy, significantly outperforming action-reduced variants (AR: 73-77%) and observation-only models (OO: 64-65%). This confirms that action information in sensorimotor learning is particularly critical for learning representations that can resolve perceptual ambiguity, showing that self-supervised learning with action enables more effective disambiguation compared to observation-only approaches

# 6. Conclusion and Perspectives

This work explores how explicit action integration affects the learning of object representations from sensorimotor sequences in deep learning models. Inspired by sensorimotor contingency theory, we created a simplified Tetris environment to generate sequences of visual observations paired with actions, evaluating Transformers and State Space Models (SSM) in a self-supervised multi-task setting.

Results show that incorporating action signals in inputs as well as prediction targets significantly boosts object recognition performance, particularly on ambiguous sequences with gains exceeding 30 percents with a comparable quantity of informative training data. These improvements occur across both model architectures, with action integration proving beneficial regardless of the specific encoder type used. This consistency supports the importance of action signals for learning object representations is architecture-independent, therefore also supporting our core hypothesis about the constitutive role of action in object representation. These findings emphasize the value of grounding learning in structured sensorimotor data where observations and actions are temporally coupled, supporting approaches that leverage the inherent structure of agent-environment interactions for representation learning.

Future directions include validating the results in richer environments with continuous dynamics and multi-object interactions to test generalization to real-world complexity. Most importantly, we aim to move beyond passive sequence processing toward true interactive learning, where agents engage in continuous interaction without resets. In sensorimotor contingencies theory, active interaction with the environment plays also a central role in mastering the regularities. It would thus be interesting to additionally introduce active learning to speed up learning and shape the representation structure, as well as active perception to discriminate between objects more efficiently.

# Declaration on Generative AI

During the preparation of this work, the authors used GPT-4 for: Text Translation and Improve writing style. After using these tools, the authors reviewed and edited the content as needed and take full responsibility for the publication's content.

# References

[1] S. Albelwi, Survey on self-supervised learning: Auxiliary pretext tasks and contrastive learning methods in imaging, Entropy 24 (2022). URL: https://www.mdpi.com/1099-4300/24/4/551. doi:10.3390/e24040551.
[2] M. C. Schiappa, Y. S. Rawat, M. Shah, Self-supervised learning for videos: A survey, ACM Computing Surveys 55 (2023) 1–37. URL: http://dx.doi.org/10.1145/3577925. doi:10.1145/3577925.

[3] D. Pathak, P. Agrawal, A. A. Efros, T. Darrell, Curiosity-driven exploration by self-supervised prediction, in: Proceedings of the 34th International Conference on Machine Learning, volume 70 of *Proceedings of Machine Learning Research*, PMLR, 2017, pp. 2778–2787. URL: https://arxiv.org/abs/1705.05363.

[4] J. Wu, S. Yin, N. Feng, X. He, D. Li, J. Hao, M. Long, ivideogpt: Interactive videogpts are scalable world models, 2024. URL: https://arxiv.org/abs/2405.15223. arXiv:2405.15223.

[5] J. K. O'Regan, A. Noë, A sensorimotor account of vision and visual consciousness, Behavioral and Brain Sciences 24 (2001) 939–973. URL: http://nivea.psycho.univ-paris5.fr/OREGAN-NOE-BBS/OREgan;Noe.BBS.pdf. doi:10.1017/S0140525X01000115.

[6] M. Bickhard, Autonomy, function, and representation, Artificial Intelligence - AI 17 (2000).

[7] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, I. Polosukhin, Attention is all you need, Advances in Neural Information Processing Systems 30 (2017).

[8] A. Gu, K. Goel, C. Ré, Efficiently Modeling Long Sequences with Structured State Spaces, arXiv (Cornell University) (2021). URL: https://arxiv.org/abs/2111.00396. doi:10.48550/arxiv.2111.00396.

[9] N. Le Hir, O. Sigaud, A. Laflaquière, Identification of invariant sensorimotor structures as a prerequisite for the discovery of objects, Frontiers in Robotics and AI 5 (2018) 70. doi:10.3389/frobt.2018.00070.

[10] A. Laflaquière, J. K. O'Regan, B. Gas, A. V. Terekhov, Discovering space - grounding spatial topology and metric regularity in a naive agent's sensorimotor experience, Neural Networks 105 (2018) 371–392. doi:10.1016/j.neunet.2018.06.001.

[11] A. Gu, T. Dao, Mamba: Linear-Time Sequence Modeling with Selective State Spaces, arXiv (Cornell University) (2023). URL: https://arxiv.org/abs/2312.00752. doi:10.48550/arxiv.2312.00752.

[12] J. Ding, Y. Zhang, Y. Shang, et al., Understanding world or predicting future? a comprehensive survey of world models, arXiv preprint arXiv:2411.14499 (2024).

[13] T. Lesort, N. Díaz-Rodríguez, J. Goudou, D. Filliat, State representation learning for control: An overview, Neural Networks 108 (2018) 379–392.

[14] D. Hafner, T. Lillicrap, J. Ba, M. Norouzi, Dream to control: Learning behaviors by latent imagination, arXiv preprint arXiv:1912.01603 (2019).

[15] P. Agrawal, A. Nair, P. Abbeel, J. Malik, S. Levine, Learning to poke by poking: Experiential learning of intuitive physics, in: Advances in Neural Information Processing Systems, volume 29, 2016.

[16] A. Zhang, N. Ballas, J. Pineau, Learning invariant representations for reinforcement learning without reconstruction, in: International Conference on Machine Learning, PMLR, 2021, pp. 12674–12685.

# A. Global Classification Metrics of the Models

## A.1. Multi-task models (-MT)
<figure><img src="image_8.png" /><figcaption>Supplementary Figure 1: Classification report and confusion matrix for the multi-task Transformer (T-MT)</figcaption></figure>

<figure><img src="image_9.png" /></figure>

<figure><img src="image_10.png" /></figure>

<figure><img src="image_11.png" /><figcaption>Supplementary Figure 2: Classification report and confusion matrix for the multi-task SSM (S-MT)</figcaption></figure>

## A.2. Removal of action predictive heads (-AR)

<figure><img src="image_12.png" /></figure>

<figure><img src="image_13.png" /><figcaption>Supplementary Figure 3: Classification report and confusion matrix for the ablated Transformer (T-AR)</figcaption></figure>
<figure><img src="image_14.png" /></figure>

<figure><img src="image_15.png" /><figcaption>Supplementary Figure 4: Classification report and confusion matrix for the ablated SSM (S-AR)</figcaption></figure>

### A.3. Removal of actions from input sequences (-OO)

<figure><img src="image_16.png" /></figure>

<figure><img src="image_17.png" /><figcaption>Supplementary Figure 5: Classification report and confusion matrix for the no action Transformer (T-OO)</figcaption></figure>

<figure><img src="image_18.png" /></figure>

<figure><img src="image_19.png" /><figcaption>Supplementary Figure 6: Classification report and confusion matrix for the no action SSM (S-OO)</figcaption></figure>
## B. Class based accuracy

**Supplementary Table 1**

Comparison of class-wise accuracy between models T-MT, S-MT, and the theoretical maximum

<table><thead><tr><th>Label</th><th>S-MT</th><th>T-MT</th><th>Theoretical Maximum</th></tr></thead><tbody><tr><td>I</td><td>86%</td><td>86%</td><td>88%</td></tr><tr><td>J</td><td><strong>91%</strong></td><td><strong>97%</strong></td><td>98%</td></tr><tr><td>L</td><td><strong>100%</strong></td><td><strong>100%</strong></td><td>100%</td></tr><tr><td>O</td><td>88%</td><td>88%</td><td>90%</td></tr><tr><td>S</td><td><strong>81%</strong></td><td><strong>88%</strong></td><td>89%</td></tr><tr><td>T</td><td>97%</td><td>100%</td><td>100%</td></tr><tr><td>Z</td><td><strong>92%</strong></td><td><strong>90%</strong></td><td>93%</td></tr></tbody></table>

## C. Optimal hyperparameters

**Supplementary Table 2**

Optimal hyperparameters

<table><thead><tr><th>Parameter</th><th>T-MT</th><th>T-AR</th><th>T-OO</th><th>S-MT</th><th>S-AR</th><th>S-OO</th></tr></thead><tbody><tr><td>Model dimension</td><td>256</td><td>144</td><td>576</td><td>320</td><td>384</td><td>128</td></tr><tr><td>Attention heads</td><td>4</td><td>8</td><td>8</td><td>-</td><td>-</td><td>-</td></tr><tr><td>Layers</td><td>4</td><td>4</td><td>4</td><td>-</td><td>-</td><td>-</td></tr><tr><td>State dimension</td><td>-</td><td>-</td><td>-</td><td>64</td><td>96</td><td>192</td></tr><tr><td>Conv kernel size</td><td>-</td><td>-</td><td>-</td><td>6</td><td>2</td><td>2</td></tr><tr><td>Expansion factor</td><td>-</td><td>-</td><td>-</td><td>4</td><td>2</td><td>5</td></tr><tr><td>Learning rate (×10<sup>-5</sup>)</td><td>10</td><td>125</td><td>9.4</td><td>50</td><td>62.5</td><td>10</td></tr><tr><td>Weight decay (×10<sup>-5</sup>)</td><td>10</td><td>1.5</td><td>3.75</td><td>1.5</td><td>1</td><td>5</td></tr></tbody></table>