MULTIMODALITY AS SUPERVISION: SELF-SUPERVISED SPECIALIZATION TO THE TEST ENVIRONMENT VIA MULTIMODALITY
Authors: Kunal Pratap Singh
Ali Garjani
Rishubh Singh
Muhammad Uzair Khattak
Jason Toskov
Efe Tarhan
Andrei Atanov
Oğuzhan Fatih Kar
Amir Zamir
Venue: ICLR 2026 Poster (2026)
Keywords: specialization
multimodal
transfer learning
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: Studying 'multimodality as self-supervision', to learn a representation that achieves SOTA in the test environment without using external/internet-based data

Abstract: The common approach for developing a vision model is generalism, which involves training on a large diverse dataset to cover the varied deployment environments and leads to a model that is expected to solve the problem everywhere. However, many practical applications need to operate in a specific test space, e.g., a robot deployed in a single house, and do not necessarily need to generalize to novel environments. In this work, we explore whether we can use rich multimodal data only from the test environment to pre-train a representation in a self-supervised way, without access to any external data.
We find that this approach can match and, in most cases, outperform generalists pre-trained on large-scale Internet datasets, including popular off-the-shelf models, CLIP and DINOv2. We study the effectiveness of this approach by evaluating the models on various datasets and downstream tasks, such as semantic segmentation, captioning, and object detection, as well as a set of ablations and analyses to extract insights. This approach raises intriguing points on substituting data with (multi)modality, enabling an alternative scenario where the need for external Internet-scale datasets for pre-training models is reduced. It also shows that merely benefiting from test-space data was insufficient for achieving competitive results, and multimodality was essential for that purpose.

Published as a conference paper at ICLR 2026
MULTIMODALITY AS SUPERVISION: SELF-SUPERVISED SPECIALIZATION TO THE TEST EN- VIRONMENT VIA MULTIMODALITY
Kunal Pratap Singh* Ali Garjani"
Rishubh Singh
Muhammad Uzair Khattak
Efe Tarhan
Jason Toskov Andrei Atanov
Oguzhan Fatih Karl
Amir Zamir
Swiss Federal Institute of Technology Lausanne (EPFL)
sttps://tttt-vlssoo..ep..
Test-Space Training
RGB
Depth
Gross Modal
Cross Modal
Deploy
Learning
Pre-training
Space
Space
Generalist Pre-training
Pre-training
Physical World
External Data
(Internet, Non- Test Spaces)
Test Space
Figure Left: Multimodality as Supervision. The sensed data in a deployment environment is often multimodal, which, besides RGB images, can contain various modalities, such as depth, motion sensing, surface normals, tactile, etc. This enables Cross-Modal learning, i.e., predicting the response of one sensor from another, frame learning a self-supervised representation for the test space via multimodality.
method for self-supervised pre-training. We use this concept to
Right: Test-Space Training. The common approach to train and deploy vision models in desired test space is generalist pre-training It uses large, diverse external data, such as images from the Internet or other external spaces. As alternative, performs self-supervised pre-training on unlabeled data from the test space, yielding performant representation for that space. We evaluate this approach on several downstream tasks (semantic segmentation here) and show that TST can outperform generalist pre-training baselines, including those trained on large-scale basedd datasets (Bachmann al. 2024] Changpinyo et al. 2021
study Test-Space Training (TST), which
Oquab et al. 2023 Radford et al. 2021) or many other external spaces.
ABSTRACT
Many practical applications, e.g., deploying household robot, involve devices that are equipped with a rich set of sensors that enable multimodal sensing in their test environment. The multimodal data collected from these sensors can be used to perform cross-modal learning, therefore using multimodality as source of self-supervision. To study this, we propose setup where we restrict a user device to just given test environment. This allows us to study the ability of multimodal pre-training to learn distribution agnostic its generalization abilities. Under this setup, develop framework called Test-Space Training (IST), which performs multimodal data collection in the test environment and performs self-super pre-training on it. We evaluate these models various downstream tasks in the
Work done while EPFL. Now Apple.

---

Published as a conference paper at ICLR 2026
environment. TST reveals various interesting insights, such collecting rich multimodal data only from the test environment and leveraging cross-modal learning, we achieve competitive results with generalist models (Oquab et al. 2023; Radford et al. 2021), pre-trained on large-scale internet datasets. We also present of analyses and ablations that raise intriguing points on substituting data with (multi monuliity enabling an alternative scenario where the need for external Internet-scale datasets for pre-training models reduced, and how varying pre-training data enables a tradeoff between a model's 's abilities to specialise to test environment, and generalize to held-out spaces.
INTRODUCTION.
Many user devices, e.g., a household robot (Wu et al.
2023),
augmented reality glasses (Lv et al. 2024
Zehtabian al. 202T), an iPhone (Baruch et al. that enable mulumooall data collection. They can acquire modalities beyond RGB images. such as depth, motion sensing, and haptic feedback. Such data enables cross-modal learning (Smith Gasser 2005), i.e., predicting the response of one sensor from another, thereby leveraging multimodality as a signal for supersuuee learning (Fig [
202T
are equipped with rich array of sensors
left). Evidence in developmental psychology (de Sa 2003; Smith & Gasser (2055) also suggests that multimodality is employed by biological organisms to bootstrap better representations of their environment.
In this work, we are interested in studying the potential of multimodality as source of self-supervisio building vision models. To so in a controlled manner, we construct a sandbox setup wherein we reduce the operating space of the user device specific physical space, test space. This implics that self-supervised pre- pre-training of visual representations and downstream evaluations both happen in the same space. This has several desirable properties. It alleviates the burden generalisation, thereby allowing us to attribute the performance solely to the effectiveness of multimodal pre-training. Additionally, it also represents practical evaluation scenarios such as household robotics (Wu et al. 2023), where user devices are expected to be highly performant in their own space, regardless of their generalization abilities elsewhere.
refer to it, the
To Iearn representations under this setup, we propose Test-Space Training (Fig. L right), which enables multimodal pre-trainnn pre-training data collection in the same space that the device would be deployed on. We use this data to perform self-supervised pre-training via cross-modas learning (Bachmann al. 2022). leading to TST-MM.
Through various analyses and ablations, we contribute the following key findings:
iST-MM the most performant in the test space. leverages sensory and off-the-shelf modalities with just test space, for pre- vre-training, and can serve as viable alternative for off-the-shelf generalists (Tab. D. currently covers half the way our sandbox (Fig. [T]. indicating room for further improvement in multimodal pre- pre-training algorithms.
Scaling modalities in the test space can substitute external data. We find that scaling the number of modalities in the test space is more effective (Fig. 4 than scaling unimodal data from external source.
Specialization-G ppeciaizattonnGGeerralaaatoo Tradeoff. We explore how, with the same number of samples, the source of pre- training data can enable model to exhibit varying abilities, from being specialized to a test space, to generalizing over set of held-out spaces (Fig. 9h.
2 RELATED WORK
Self-supervised learning (SSL) has been effective in learning visual (et al. 2020 He et al. 2021
Bao et al. 2022 Oquab et al. 2023) and natural language (Devlin et al. 2019; Brown et al. 2020
OpenAl 2023) representations. In vision, one line of work uses masked image modeling (He et al. 2021) as a scalable approach to pre-train self-supervised models. It masks an input image, and attempts to reconstruct it in the form of pixels (He al. 2021: Chen et al.) 2020a| Dosovitskiy et al.
2021 El-Nouby et al, 2022aa,, tokens (Bao et al. 2022) or features (Zhou et al.) 2022 Baevski
al.
2022 On the other hand, approaches like Sim (Chen et al. 2020b) and DINOv2 (Oquab
2

---

Published as a conference paper at ICLR 2026
et al. 2023) use contrastive learning (Caron et al.) 2001 Chen et al. 2020b; He al. 2020 Chen
& He 2020) and knowledge distillation (Buciluà et al.
2006 Hinton et al.) 2015) respectively,
to pre-train representations. Both classes of SSL pre- training approaches are typically trained on
large-scale Internet-based datasets (Changpinyo et al. 2021] Deng et al. 2009 Schuhmann et al. 2022 Gadre al. 2024) and exhibit remarkable downstream generalization abilities. In contrast to the self-supervised learning works discussed above, which aim to pre-train on a large-scale dataset to generalize to a variety of downstream scenarios, we are interested in studying multimodality as a source of self-supervision in controlled setup. We do so by restricting our pre- wre-training and downstream evaluation space single test space, and performing cross- cross-modal learning on the multimodal data collected in that space.
Multimodal learning aims to build models that can relate information from different sources of underlying reality (Balruvsaitis et al. (017). This can involve training separate encoders or unified model on various sources of modalities, like image, video, 3D, text, audio, etc. (Arandjelovic &
Zisserman 2017 Lu et al. 2019; Jaegle et al. 2022, Radford 2 al. 2021; : Girdhar et al. [0022] Lu
ct ala.]] a Girdhar et 2023). MultiMAE (Bachmann et al. (022) uses multimodal masked modeling to learn cross predictive coding across multiple modalities. 4M (Mizrahi et al., 2023
Bachmann et al. 2024) extends this idea further to train a multimodal foundation model across tens of modalities. These approaches build on large-scale Internet datasets with image-text pairs (Changpinyo et a 2021 [Byoon et al.] 2022 [Schuhmann et al. 2022). 'ST leverages cross-modal pre-training on test space data, as opposed to Internet-based data, learn performant vision model for that space. Results in Sec. 4soo the value of TST over these multimodal baselines.
Pre-training data has been critical component in the recent advances in self-supervised (Oquab
et al. 2023; Caron et al. (22TT) and multimodal foundation models (Mizrahi et al. 2023 OpenAl
2023 Ramesh et al.] 2021). Prior work (Gadre et al.] 2024: Li et al.j 2024b] Fang al. (2023)
has studied the role of pre-training data curation criteria in large-scale imge--tex models (Radford al., (02T). Similarly, (HI-Nouby et al. 202Tb) investigates the role of external pre-training data for downstream performance on various target transfer tasks (Lin et al., 2014; et al.) 2017). They show that, pre-training via masked modelling directly on just the target task images performs on par with pre-training large-scale external dataset such as Imagenet (Russakovsky et al. 2014). Our work also similarly sheds light on the importance of pre-training in building models specialised for a particular test space, opposed to prior work, that was more focused towards generalization abilities. We present more analysis on the role of pre-training data in building specialist models in Sec. 4.5
Test-time adaptation adapts model to distribution shifts at test-time (see (Xiao & Snoek 2024] for
a recent survey). One prominent approach in the community is test-time training (TTT) (Sun et al. 2020, Wang et al. [202T] Liu et al. 2021b Gandelsman al. 2022h] Boudiif et al. [022] Gao et al. 2.23), which optimizes a self-supervised objective at test-time to finetune the model. Contrastingly, as opposed optimizing for specific test instance, we focus learning a vision model for a given test space during pre-training and not model adaptation at test-time. Concretely, we specialize in given test space, as opposed to a specific test instance. Note that TTT can be complementary to TST and improve performance (see App. T We present additional related works in App. B on domain adaptation, embodied active learning, and supervised learning.
METHOD
We provide an overview of our framework, TST, in Fig. In Sec. 3.1 we describe our sandbox setup for studying multimodality as supervision. In Sec. 3.2 we provide details on the multimodal data collection process in the test space. Sec. 3.3 describes how we leverage cross-modal learning to pre-train on this data, and finally, Sec. 3.4 outlines our evaluation setup.
3.1 PROBLEM SETTING
As described in Sec. [ we construct a sandbox setup to study multimodality as a source of self- supervision in a controlled manner. We restrict the user device to a specific physical space. This implies not having external sources, like internet-based datasets (Changpinyo et al. (021) pre-train a model on, or a lack of label supervision in the test space. It also assumes that the pre-traiee pre-training and downstream evaluation happen in the same space. This reduces the need to generalize novel
3

---

Published as a conference paper at ICLR 2026
1. Data Collection
Pre-training
3. Transfer
Deployment
Tret
Edeernl Debouet
Selt Superdsed trakning
Transfering Pre- triined Model)
Capttoninn Thansferred
Detection Modil
Segmentation Transfiored
Multimodal
Figure 2: TST framework. 1) First, we collect (multimodal) data from the test space (Sec. 3.2) 2) We then use this data for self-supervised multimodal pre-training (Mizrahi et al. 2023 Oquab al. 2023) (Sec. 3.3 After pre-training, the model is fine-tuned on small external transfer dataset to solve a desired downstream task, e.g. semantic segmentation (Sec. 3.4 4) This model is subsequently deployed and evaluated in the same test space where was pre-trained (Sec.
distribution, thereby allowing us to study the efficacy of multimodal pre-training by tracking the performance in the test space.
To this end, our framework, Test-Space Training (TST), proposes to collect unsupervised, multimodal pre-training data in the test space. Concretely, assume access to the sensory data sampling function in the test space, denoted as x (space and use it to collect a pre-training dataset Drr {a;} (Sec.3. 3.2). Besides RGB images, also leverage other sensors available the device, e.g., depth and surface normals, In real-world deployment, this set can be expanded significantly to other common sensors, such as IMU, microphone, radar, and occasionally haptics. We use this data to tre-train& a self-supervised model f X - h that maps RGB images into representations (Sec. 3.3). We evaluate this model on downstream tasks via transfer learning (.4..
3.2 MUL.TIMODAL DATA COLLECTION
As noted in Sec. 3.1 assume access to the sensory data sampling function in the test space, denoted as & ~ Pspace (x) to collect pre-training data, Drr. This can represent capturing data at various vantage points (Eftekhar et al. (02T),
video walkthrough (Baruch et al., [02T]] Yeshwanthi et al. 2023) to cover the test space. In addition to RGB frames, we also collect data from various sensors and modalities available
the user device being deployed in the test space. Additionally, can also process this data to create more optional modalities as illustrated in Fig. 2 As we later show in Sec. [4.4 and Fig. 4 scaling this rich set of modalities in the multimodal, test-space data is more effective than scaling to diverse unimodal data from external sources. It is also worth noting that such dataset of potentially repetitive images from the same space is related to findings developmental psychology research suggesting that infants observe highly redundant visual data (Jayaraman et al.) 2015 Slone et al. 2019). We defer more implementation details for our data collection to Sec. |4 1 This stage results in multimodal sensory dataset, which use for self-supervised pre-training.
3.3 SULF-SUPERVISED PRE-TRAINING
We employ self-supervised learning to pre-train a model / on the multimodal data DPT collected in the test space. Akin to standard generalist self self-supervised pre-training, this model learns task-agnostic representations that are useful for various downstream tasks. Note that although the objective of this work is studying the potential of multimodality as source of self-supervision, our setup described in Sec. 3.1 and the TST framework can also support other self-supervisee objectives. Amongst different self-supervised objectives, comparing both unimodal (RGB-only (He et al.) (02Th), i.c. TST-MAE, and multimodal (Bachmann et al.) 2022, Mizrahi et al.) 2023 Bachmann et al.) (024). ie. TST-MM, variants within our framework. Among these, we found cross-modal learning, or more specifically, multimodal masked modeling to be most effective for
setup, consistently achieving
superior performance (Sec. 4 App. H). We explore the role of choice of modalities (Fig. (5), and
how performance scales with the number of modalities (Fig. (6 for TST-MM in Sec. 4.4 Note that
4

---

Published as a conference paper at ICLR 2026
the TST framework can also support other self-supervised objectives such DINOv2 (Oquab et al. 2023) and MAE (He et al. 2021) (App. Qq. Unless specified otherwise, we refer to the multimodal version (TST-MM) TST. We describe more details on the architecture and modalities in Sec. 4
3.4 TRANSFER
We evaluate the effectiveness of the pre-trained model f uusin transfer learning. We add task- specific head 9 and finetuning the resulting model g o f on various downstream tasks, following standard practice in self-supervised learning (Mizrahi et al., 2023 He et al. 2021). For this, we consider a small transfer dataset D, with task-specific annotations, collected in an external space, disconnected from the test space. Importantly, we do not have access any task speciff annotations from the test space itself. i.c. Dr and DTT are sampled from different distributions. We benchmark
against several off- off-the-shelf vision models (Radford et al. [021 Oquab et al. 2023 Bachmann et al.) 2024), by finetuning them on the transfer data, Di, as discussed in Sec. 4.3
EXPERIMENTS
We present the results as follows. Sec. 4.1 describes our experimental setup and baselines. Sec. 4.2 analyzes how far TST can go with sensory modalities only. Then, Sec. 4..3 further scales these modalities with off-the-shelf pseudolabel networks and compares against off off-the-shelf Internet-based generalists (Bachmann et al. 2024 Oquab et al. 2023 Radford et al.) 202I) and task-specific
baselines (Kirillov et al.) 2023 Li al. 2022 Cheng et al.). Next, we present various analyses on the role of multimodality in TST. T. Sec. 4.5 explores the role of pre-training data from the test space.
4.1 EXPERIMENTAL. SETUP
Datasets. We experiment using three datasets:
1. Scannet++ (Yeshwanth et al. 2023) is large-scale dataset of real-world indoor spaces containing aub-millimeter resolution scans, paired with DSLR and iPhone RGB images. We use scenes as the test space, and use mix of iPhone and DSLR images from these scenes for pre-training.
Replica (Straub et al. 2019) provides high-quality 3D reconstructions of real indoor spaces. We use 5 scenes the test space, and use rendered RGB-D images for pre-training.
3. ProcTHOR (Deitke et al. 2022) includes procedurally generated house-like environments. We use 5 procedurally generated houses as the test space, unless specified otherwise.
Pre-training. For training models with TST-M and TST-MAE on the dataset DP collected from a test space, we leverage multimodal masked modelling (Bachmann al. 2024) described in Sec. [.3 and train an encoder-decoder transformer model. We use mnodality-specitn tokenizers (Bachmann al. 2024) to convert all modalities into tokens. We train models across two encoder sizes, VIT-S and ViT-B, which have and 12 encoder layers, respectively. Additionally, we found that mixing RGB images from the transfer was beneficial in pre-training. Please see App. M for ablation on this choice. Note that we do not use any task labels from the transfer set during pre-training, making this stage task-agnostic. For results in Sec. 4.3 we initialize the model from scratch, whereas for adaptation (Sec. 4.3 Adaptation through TST) we initialize from 4M-21 (Bachmann et al. 2024)
Notations. We refer to TST variants with different objectives as T-M-MM for multimodal masked modeling, TST-MAE for unimodal masked modeling, TST-DINO for DINOv2 (Oquab et al. 2023). Unless specified otherwise, we refer to the multimodal version (TST-MM) as TST.
Transfer and Evaluation. For all datasets, we use an external set of scenes that are different from the test space to collect small transfer set (D,) with task-specific annotations. We evaluate the transferred models in the test space semantic segmentation (Scannet++, Replica, ProcTHOR), object detection (Scannet++, ProRTHOR) and image captioning (HrocTHOR). We provide more details the transfer and evaluation setup in the Appendix.
Modalities. For Scannet++ (Yeshwanth et al. 2023). we use RGB images captured by DSLR and iPhone cameras. For Replica (Straub et al. 2019) and ProcTHOR (Deitke al. (022). render RGB-D D from the test space using onboard sensors. We also include Surface normals and Canny Edges as they can be extracted from RGB and depth respectively using simple operations. We refer to these
5

---

Published as a conference paper at ICLR 2026
Scratch
TSTMAE
TSTMM (Semoo ont)
DINO
Number Modalltte
50.72
53.00
mbJ
50.33
47
Figure How far can we go with no exter-
Number of Spaces
220
3005
nal access? We compare results of pre-training
using large-scale Internet data (DINOv2 (Oquab)
Figure 4: Modality scaling vs data scaling, We study the tradeoff between collecting unimodal pre-training data from more spaces versus scaling modalities the test space (here, houses). The size of each circle is proportional to the mloU performance on segmentation. We find that seal- ing the number of modalities within the test space yields better performance than scaling data by in- cluding external spaces, underscoring the efficacy of TST-MM. All models use the ViT-S backbone.
et al., (023) 142M images) with using only data collected from a test space with onboard sensors, TST-MM (Sensors). We show segmenta- tion and detection results on a test space from the
Scannet++, We find that, with no external access,
TST-MS with sensory modalities, and just mul-
timodal data from the test space, is competitive with scrge-scale Internet-based pre-training.
modalities (RGB, Depth, Surface normals, and Canny Edges) as sensory in Sec. 4.2 and thereafter. In Sec.
we discuss how we can further scale the number of modalities using ohl-tte-shelf networks.
Baselines. We compare against several baselines:
Scratch no pre-training. We compare against both unimodal scratch, which takes only RGB images input, and multimodal scratch, which inputs all modalities available to TST during transfer training and evaluation. The latter baseline indicates that the performance is not owed to merely having multiple modalities, but rather performing cross-modal pre-training.
Large-scale generalist pre-training baselines. We evaluate 4M-21 (Bachmann et al. 2024), DI- NOv2 (Oquab et al. 2023), and CLIP (Radford al., (202I) recent strong generalist (self- supervised) baselines, trained on large-scard scalescal datasets via unimodal and multimodal learning. To ensure fair comparison, we finetune these models, with the same transfer dataset (D₂) as TST.
Task specialist baselines. We perform evaluations using Mask2Former (Cheng et al.). ViTDet (Li ct al. 2022). SAM (Kirillov et al. 2023), and VL- (Liu et al. 2023) as established task-specific baselines for semantic segmentation, object detection, and image captioning. Similar to generalist baselines, we finetune these models, with the same transfer dataset, D., that we use for TST.
We finetune the baselines with the same transfer data Dr, and only RGB images input for transfer and evaluation (except for the multimodal scratch baseline). An extensive hyperparameter experimental setup, including
search was done for fair comparison. We provide more details pre-trase pre-training and transfer training hyperparameters in App. X
4.2 HOW FAR CAN WE GO WITH NO EXTERNAL ACCESS?
We first explore how far we can go when bootstrapping vision representation for the test space, with no external access. Fig. 3 shows, on Scannet++ Yeshwanth et al. (2023), that pre-training with just multimodal sensory data from the test space can perform competitively with generalist models (Oquab et al. 2023), pre-trained on large-scale external (internet-based) data. In addition, we find that TST-MAF MAE improves upon training from scratch, yet multimodality with TST-MM performs the best. This suggests that with TST, we can build highly performant models for the test space, without any external access, and achieve competitive performance with large-scale Internet-based pre-training (Oquab et al. 2023).
4.3 SCALING MODALITIES IN THE TEST SPACE
Sec. 4.2 discusses how far TST can go with no external access, with only sensory modalities. To take this one step further, we draw inspiration from recent progress in multimodal foundation
1 with only sensory modalities as descrihed in Sec, Taask-ppecific methods used for each result, in order: SAM Kirillov et al. 2023 (segmentation), ViTDet
2022 (detection), and L.LVVA (Liu et al. 2023 (captioning)

---

Published as a conference paper at ICLR 2026
Semantic Segmentation
Object Detection
Captioning
Method
Scumnet++ mloU
ProT
Replica mioU
Scanmet++
PreCTHOR
PreTTHOR
mloU
mAP
mAP
CIDEr
SPICE
No Pro-training
Unimodal Scratch pre gee-training
7.49
28.62
9.23
2.35
24.59
17.1
14.8
Multimodal Scratch no pre training
10.03
19.19
10.5
4M (RGB-onlv)) MAE et 2021
13.74
46.29
18.18
18.31
37.17
30.4
19.1
Generalist
(Rachmann
26.30
25.91
41.43
20.3
Pre-training
DINO (Oquab & Fi RU25)
30.60
$4.50
26.72
23.67
40.28
14.7
13.5
CLIP (Radtord oi
23.19
48.66
20,92
19:75
38.47
18.4
16.2
Task Specialist
Task Specific Viethods
34.75
56.72
28.51
23.59
44.10
40.60
21.00
Specialist
TST-MM
60.88
32.87
49.38
34.3
20.4
Pre training
TSI-MK (sI-tt(uulapied)
36.44
60.59
34.53
35.83
51.25
39.9
20.5
Table 1: Multimodal Test-Space Training (TST-MM) outperforms both strong generalists and task specialists across tasks. We evaluate semantic segmentation, object detection, and image captioning. All models use ViT-B backbones, except SAM (Kirillov et al., (023) (ST-- TM (adapted) refers to tne-tuning- 4M-21 test-space data. On segmentation and detection, TST-MM consistently outperforms Internetbaseed generalists and matches or surpasses specialists. On cap- tioning, TST-MM (from scratch), despite no text pre-training, matches 4M-21 trained on CC12M
image-text pairs; TST-MP (adapted) surpasses 4M-21 and approaches LLaVA-1.5
(Liu et al. 2023
models (Bachmann et al. 2023), and further scale our set of modalities. Next, we compare TST with this scaled set of modalities (TST-MM), against Internet-based generalists and task specialist models. Lastly, we describe how TST can also enable adapting a pre-trained generalist to the test space.
Additional Modalities. We create new modalities by pseudolabeling the collected RGB frames. We use neural network feature maps (Radford et al. |222] Girdhar al. (002)). SAM edges (Kirillov) ctt al. 2023), bounding boxes from ViTDet (Li et al. 2022). and semantic segmentation masks from also include these pseudolabeling networks
Mask2Former (Cheng et al.). For fair comparison, as baselines and show that TST-MM, trained from scratch, outperforms all of them (see Tab. [ and App. D. Note that TST-MM, which leverages these additional modalities, longer qualifies for external access, as is akin to distilling the off-the-shelf pseudolaheling networks which were trained on large-scale external data. However, note that this distillation happens only on test-space data, therefore alleviating the need to directly access external data by leveraging off- eff-the-model outputs as modalities. This is reminiscent of recent findings on attention transfer (Li et al. 2024a), which show that only distilling the attention patterns performs similarly to full finetuning of the model.
TST vs. generalists. Tab. I shows quantitative results for TST-MM. We compare against generalist models (MAE, DINOv2, 4M-21, and CLIP) trained on large-scale Internet datasets. This suggests that, we can outperform generalist models by using multimodal data from the test space. Figure T0 shows qualitative improvements of TST over generalist Internet pre-training.
TST vs. task specialists. We also show that TST-MM outperforms or is on par with off-the-shelf task specialist models semantic segmentation (Kirillov et al., 2023) and object detection (Li et al., 2022). For image captioning, despite not seeing any text data during pre-training, TST-MM performs par with 4M-21 (Bachmann et al. 2024) that was pre-trained on large-scale image-text data (Changpinyo et al.
2021 showing the effectiveness of the learned representation.
Adaptation through TST. Tab. Talso presents results when TST-MM. adapts existing generalist model to the test space. Unlike TST-MM models discussed above that start from scratch, we start from a pre-trained 4M-21 model and fine-tune it on data from the test space, using multimodal masked modeling. The resulting model, TST-MM (adapted), significantly improves over 4M-21 (Bachmann) al. 2024) in the test space. Therefore, TST can also serve as an adaptation mechanism for Internet-based models, making them more performant in the test space for downstream tasks.
4.4 MULTIMODALITY TST
As elaborated in Sec. 3.2 TST collects multimodal data in the test space. This allows us to leverage cross- mross- learning, and specifically, multimodal masked modelling for performing self- learning, which is a key component in enabling performant models in the test space. We perform the following controlled analysis to draw insights its role in TST.
7

---

Published as a conference paper at ICLR 2026
Mormalzznd Parformnnc
Caere Mamm
SAN Capss
SAA
CLP
Loge
Al
Figure 5: Contribution of different modalities to TST performance. We study the effect of each modality on TST by doing a drop one combination from TST -MM and add one to only TST). We use the ViT-S backbone. We find that even though some modalities provide higher gains than others when added to the RGB-only TST-MAE, the performance of TST-MM stays relatively stable, agnostic to the choice of the dropped modality. This indicates that no single modality is responsible for TST-Mi''s performance, but rather their collective interplay, i.e., multimodality. Can we substitute large-scale data with more modalities? We study the trade-off between using smaller-scale but modality-rich test-space data, versus large-scale unimodal external data (RGB- only). We use ProcTHOR to generate similar spaces (IID to the test space), and leverage them as an external source of data. Starting from unimodal pre- pre-training in the test space, Fig.4sshows that scaling data via additional modalities yields significantly better performance than increasing the amount of unimodal data from external sources. Additionally, we find that although scaling unimodal external data improves over the unimodal baseline in the test space. However, as we increase the number of modalities, we find that the performance is relatively stable when external data is added. This suggests: For building high-peen perffrrefoning models a specific test space, collecting data within that space using richer set of modalities is more often effective than relying on large-scale, unimodal data collected from external sources.
(RGB-
Is the choice of modalities important for the effective- ness of the multimodal pre-training? We investigate whether all modalities contribute similarly to multimodal pre- pre-training, as shown in Sec. 4.2 and 4.3] or there is single modality that contributes the most. We present two ablations in Fig. 5) First, we examine all pairs of two modalities starting with RGB, i.e., all {RGB, X} com- binations. Adding any modality improves performance, with some showing greater benefits than others (e.g., SAM edges increase performance by an absolute 7.8%). but none matches the performance of using all modalities. Second, we examine all sets of eight modalities by removing one modality, except RGB (which remains as the input during finetuning and evaluation). We find low variance between different sets, indicating that no single modality is irre- placeable and that other modalities can compensate for the absence of useful ones. For example, removing the SAM edges modality reduces results by only 1.5%, compared to its absolute 7.8% improvement when added to RGB alone. Thus, performance can be improved by simply collecting a larger set of modalities instead of engineering an optimal set.
28
Number of Modelities
Figure 6: Scaling the number of modal- ities for TST-MM. We report the perfor- mance of IST-MM ber of modalities. We begin with only the RGB modality and incrementally add more. Increasing the number of modal- ities improves performance, while the variance in performance due to any spe- cific modality decreases.
scale the num-
How does the performance of TST-MM scale with modalities? Fig. Gisows the performance of TST-MM as we increase the number of modalities. Due the combinatorial complexity of studying all possible combinations, we only sample all possible options for two (RGB+X) and eight (All-X) modalities, where here X is the modality added or dropped. For other modality counts, we randomly sample 8 modality sets and report the average performance on the plot. We find that the performance of TST-MM scales well with more modalities, agnostic of the exact modality combination, and with decreasing variance between subsets.
4.5 MEASURING SPECIALIZATION WITH TST
We define specialization as the measure of how performant a model is, on a downstream task, in a given test space while forgoing other spaces. E.g., model is specialized to space A if it performs well in test space A but is inferior in Space B. This is in contrast to generalization conventional machine learning, which measures performance all spaces. As described in Sec. 3.2 TST collects
8

---

Published as a conference paper at ICLR 2026
mlo (tat space)
50
50
60
40
mell: Test
49
Specia izalion
non-lest spaces TST, tast space pre training
48
Generalization
-50
mioli He di
30
1500 Number Pro training Spaces
1500
Number of Pre training Spaces
trade- off. We pre-train ViT-S models on data collected from growing number of spaces, starting with a single test space and adding data from other IID spaces. The blue curve shows performance the test space (specialization); the red curve shows performance on 100 held-out IID spaces (gener- alization). As add more pre-training spaces, test-space performance decreases while held-out performance improves, revealing the specializa- fion-generalization trade-off.
Figure 9: Specialization-generaaiine
Figure 8: How many spaces is one test space worth? We study if test-space data for pre- training can be substituted with data from sim- ilar but nonidentical spaces. We compare perfor-
mance on the test space between TST and models
pre-trained with ViT-S backbone on an increas-
ing number of IID houses. We find that using as many as 3000 spaces cannot match pre-training in the exact test space, underscoring the useful- ness of test-space specialization with TST.
pre- vre-training data in test space, to pre-train a model for that space. In this section, we explore what the role of this pre-training data from the test space itself.
First, we measure it by cross-evaluating models pre-trained on two different test spaces, showing that space-specific pre-training performs the best. Then, we explore if we can substitute data from the test space with data from many (thousands of) similar spaces during pre-training, effectively asking: how many spaces the test space worth? Third, we explore whether single model can exhibit both specialization and generalization capabilities and show the specializationn-eeeeeee generalization trade-off.
TST effectively specializes to a test space. Fig. 7 shows the performance of models pre-trained and evaluated on different test spaces. We find that in all cases pre-training the model in the corresponding test space is the best, demonstrating the practical value of specialization. We observe similar trends for other pre-training objectives (App. Q
Pre-tahiing Space
II
45.19
35.68
35.86
11
29.38
42.05
29.32
III
38.20
39.17
51.69
Figure 7: Measuring specializa- tion. We perform cross-space anal- ysis by pre-training and evaluating performance on different spaces. Each column and row represents pre-training and test space. Perfor- mance best along the diagonal, where pre- pre-training and evaluation are in the same space.
How many spaces is a single test space worth? Here we ask, if not space, data from how many similar spaces can substitute test-space data? Similar to Sec. 4.4 we use Proc- THOR (Deitke et al., 2022) to generate large number of similar houses and pre-tra train models using an increasing number of them. Fig. [ shows the performance of each model on the test space not seen during pre-training compared pre-training on the corresponding test space. We find that even thousands of similar spaces not enough to substitute pre-training on the exact same space that we deploy in.
trade-off. We observed that the best performance given test space is achieved when pre-trained on data from the same space. However, we would expect this specialized model to not generalize well to new houses. Can keep (or improve) this specialization performance while gaining generalization capabilities by adding more houses during pre-training in addition to the test house? Fig. ? shows that we add more houses during pre-training, the performance on the held-out new houses increases, as expected. However, the performance on the original test space drops compared to the specialist single-space pre-training, demonstrating specialization and generalization trade-off of the pre-trae pre-trained model.
ppecialiattion-generaiizzzz
DISCUSSION AND FUTURE WORK
We present sandbox setup to study multimodality source of self-supervision. It constrains the operating space of a multimodal device to a specific test space, allowing us to study the efficacy of multimodal pre-training independent of its generalization abilities.

---

Published as a conference paper at ICLR 2026
Input
Scratch
(Ours)
Segmentation
Semantio
Object Detecttion
Figure 10: TST-MM predictions across different tasks. We present qualitative results for TST-MM MM against various baselines, including scratch (no pre- training) and Internet- based pre-training real-world scenes from Scannet++ Yeshwanth et al.] 2023). TST-MM predictions are notably more consistent across both tasks. Note how TST-MM predicts the same object (magnified in red boxes) more accurately and robustly across various viewpoints, as compared to generalist models like 4M-21 (Bachmann et al. 2024) and DINOv2 (Oquab et al. 2023).
Mutsmodal
Scratch
TBT-MAE
(Sencors only)
TST-WM
7.62
Upper
7.48
15
54.43
Scratch
MAE
4M-21 TST-MM
Fully supertised
Large
Figure 11: Quantitative summary of TST. The lower bound is scratch (i.c., no pre-training and learning using the external transfer set only). The upper bound is approximated by fully supervised model trained with a large number of annotated segmentation images. The gap between the lower and upper bounds is the playfield for the pre-traie pre-training methods to fill. All methods share the same model architecture. The results here correspond to semantic segmentation on Scannet++.
To perform this study, we present TST, framework that performs unsupervised, multimodal pre- training data collection the test space, and then pre-trains a model on it using cross-modal learning. We present consolidated picture of our results in Fig. 11 and highlight the following key takeaways:
TST-MM yields the most performant model in the test space. It currently sits halfway between the bounds. It leverages sensory and off-the- the-shelf pseudolabelled modalities, with just test space data for pre-training. Note that this model does not qualify for claiming "no external access" (Sec. 4.3)
Distilling fest-space data external data. The distillation of pseudo-labeler network outputs as modalities (Sec. (4.3) is more effective on test-space data than on external data. Comparing TST-MM with 4M-21 (Bachmann et al.) 2024) shows that, as the two are equivalent in nearly all aspects, except that 4M-21 1 distills the pseudo-labelers on external data (CC12M) while TST-MM distills the same pseudo-labelers only on test-space data. The same observation was consistently made for various settings in Fig. 23 of the App. R
How far can we go with no external access? As discussed in Sec. 4.2 the TST-MM (Sensors only) uses no external data, directly or indirectly, as it only uses on-device sensors as modalities. It covers roughly 1/3 of the way, providing a nontrivial value, and is competitive with internet-baat based generalist, 4M-21, trained on large-scale internet data (CC12M (Changpinyo et al.) 2021)).
Multimodality as supervision is effective. Generally, multimodality useful the multimodal models consistently outperform their single-modal counterparts (e.g.. TST-MM vs TST-MAE or 4M-21 (Bachmann et al., 2024) vs MAE (He et al. 202T))
Note that TST serves formulation of leveraging multimodality and data from the deployment space to build specialist models, as opposed to a method that fully solves the problem. In future research, we are interested in advancing our understanding of the role of pre-training data sources to downstream performance in multimodal pre-training. Additionally, we are also interested in advancing the current methodology to inch closer to the upper bound by improving pre-trale pre-training objectives that leverage multi-view consistency (Luo et al. 2020), and various hardware-basce sensors such IMU, gyroscope, magnetometer.
10

---

Published as a conference paper at ICLR 2026
Acknowledgements. This work supported under project ID 43 part of the Swiss Al Initiative, through a grant from the ETH Domain and computational resources provided by the Swiss National Supercomputing Centre (CSCS) under the Alps infrastructure. This material is based work that is partially funded by an unrestricted gift from Google. This work has received funding from the Swiss State Secretariat for Education, Research and Innovation (SERT). We also thank Daniel Filipe Jana and the SCITTA team for their support. The authors also thank Chandan Yeshwanath for their help with the Scannet++ (Yeshwanth et al. 2023 dataset, and Roman Bachmann for useful discussions.
REFERENCES
Relja Arandjelovic Andrew Zisserman. Look, listen learn. Proceedings of the IEEE international conference on computer vision. pp. 609-617, 2017.
Gregor Bachmann, Sotiris Anagnostidis, and Thomas Hofmann. Scaling mlps: A tale of inductive bias. arXiv preprint arXiv:2306. 13575, 2023.
Roman Bachmann, David Mizrahi, Andrei Atanov, and Amir Zamir. MultiMAE: Multi-modal multi task masked autoencoders. European Conference on Computer Vision, 2022.
Roman Bachmann, Oguzhan Fatih Kar, David Mizrahi, Ali Garjani, Mingfei Gao, David Griffiths, Jiaming Hu, Afshin Dehghan, and Amir Zamir, 4M-21: An any-to-any vision model for tens of tasks and modalities. arXiv 2024, 2024.
Alexei Bacvski, Wei-Ning Hsu, Qiantong Xu, Arun Babu, Jiatao Gu, and Michael Auli. Data2vee: A general framework for supf-supe learning speech, vision and language. In International Conference on Machine Learning, pp. 1298-1312, 1312. PMLR, 2022.
Tadas Baltrusaitiss, Chaitanya Ahuja. and Louis- Louis-Phillpps Philippe Morency. Multimodal machine learning: A survey and taxonomy. IEEE Transactions on Pattern Analysis and Machine Intelligence, 41:423 443, 2017. URL
https: api. www.ntttscchheooooooo......
org/ CorpusID:1011/44
Hangbo Bao, Dong, Songhao Piao, and Furu Wei. BETT: BERT pre-training image transformers. In
International Conference on Learning Representations, 2022.
Jonathan T. Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin Baaiin-,rualla,. and Pratul P. Srinivasan. Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), pp.
October 2021.
Gilad Baruch, Zhuoyuan Chen, Afshin Dehghan, Tal Dimry, Yuri Feigin, Peter Fu, Thomas Gebauer, Brandon Joffe, Daniel Kurz, Arik Schwartz, Elad Shulman. Arkitscenes a diverse real-world dataset for 3d indoor scene understanding using mobile rgb-d data. In NeurlPS, 2021. URL http:: /tw.xi.vvvoggg////
pdf/2111 03897. pdf
David Berthelot, Nicholas Carlini, Ian J. Goodfellow, Nicolas Papernot, Avital Oliver, and Colin Raffel. Mixmatch: A holistic approach semi-super learning. ArXiv, 10s/0995.00249, 2019. URI.https! https
//api semant icschc wwwpsschhaaarrrr//ooooooo wwggCorpus ID: 46808485)
Malik Boudiaf, Romain Mueller, Ismail Ben Ayed, and Luca Bertinetto. Parameter-free online test-time adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8344 8353, 2022.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Vooe Gretchen Krueger, T. J. Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeff Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alee Radford, Ilya Sutskever, and Dario Amodei. Language models few-shot learners. ArXiv, abs/2005. 14165, 2020.
Cristian Buciluă, Rich Caruana, and Alexandru Niculescu-Mizil. Model compression. Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 535-541, 2006.
Neil Burgess, Jelena Milanovie, Nigel Stephens, Konstantinos Monachopoulos, and David Mansell. Bíloat processing for neural networks. In 2019 IEEE 26th Symposium on Computer Arithmetic (ARITH), pp. 88-91, 2019. doi: 1119//RTTH.2019.00
11

---

Published as a conference paper at ICLR 2026
Minwao Byeon, Beomhee Park, Haecheon Kim, Sungjun Lee, Woonhyuk Back, and Sachoon Kim. COYO- 700M: Image-text pair dataset. https://sithub www/ waw.aaabbbriiiooooddaaaoaaa
2022.
Zhaowei Cai and Nuno Vasconcelos. Cascade R-CNN: Delving into high quality object detection. 2018
IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 6154-6162, 2017.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin.
Emerging properties self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 9650-9660, 2021.
Soravit Changpinyo, Piyush Kumar Sharma, Nan Ding, and Radu Soricut. Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. 2021 IEEE/CVE Conference on Computer Vision and Puttern Recognition (CVPR), pp. 3557-3567, 2021.
Devendra Singh Chaplot, Murtaza Dalal, Saurabh Gupta, Jitendra Malik, and Ruslan Salakhutdinov. Seal: Self-supervised emhodied active learning. In NeurIPS, 2021.
Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever. Generative pretraining from pixels. In International Conference on Machine Learning, 2020a.
Ting Chen, Simon Korbllihh, Kevin Swersky, Mohammad Norouzi, and Geoffrey Hinton. Big supe-supe models are strong semi-supervised learners. arXiv preprini (tt:::2000.100223 2020b.
Xiaokang Chen, Yuhui Yuan, Gang Zeng, and Jingdong Wang. Semi-supervised semantic segmentation with cross pseudo supervision. 2021 IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pp. 201226132222 2622, 2021. URL Colnatts::: /api. http:/apppaaaaaaaa....aaaaaaaaat www.nntcccccc..oooooo.....
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recogmition (CVPR), pp. 15745-15753 2020. URL htttps:/aapi bemanticschne
Bowen Cheng. Ishan Misra, Alexander G. Schwing, Alexander Kirillov, and Rohit Girdhar. Masked-attention mask transformer for universal image segmentation. CVPR.
Kevin Clark, Minh- Thang Luong, Quoe V. Le, and Christopher D. Manning. Electra: Pre-training text encoders discriminators rather than generators. In International Conference Learning Representations, 2020.
Virginia R. de Sa. Sensory modality segregation. In Neural Information Processing Systems, 2003.
Matt Deitke, Eli VanderBilt, Alvaro Herrasti, Luca Weihs, Jordi Salvador, Kiana Ehsani, Winson Han, Eric Kolve, Ali Farhadi, Aniruddha Kembhavi, and Roozbeh Mottaghi. ProcTHOR: Large-Scale Embodied AI Using Procedural Generation. In Neur/PS, 2022. Outstanding Paper Award.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, K. Li, and Li Fei-Fei. ImageNet: A large- scale hierarchical image database. 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248-255, 2009.
Jacob Devlin, Ming-Wes Wei Chang, Kenton Loc, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding, In North American Chapter of the Association for Computational Linguistics, 2019.
Jiahua Dong, Yang Cong, Gan Sun, Zhen Fang, and Zhengming Ding. Where and how to transfer: Knowledge aggregation-indueed transferability perception for unsupervised domain adaptation. IEEE Transactions on Puttern Analysis and Muchine Intelligence, 2021.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenbom, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021.
Ainaz Eftekhar, Alexander Sax, Roman Bachmann, Jitendra Malik, and Amir Roshan Zamir. Omnidata: scalable pipeline for making multi-task mid- level vision datasets from 3d scans. 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 10766-10776, 2021.
Alaacldin El-Nouby, Gautier Izacard, Hugo Touvron, Ivan Laptev, Hervé Jégou, and Edouard Grave. Are large-scale datasets necessary for self-supervised pre training?? ArXiv, abs/2112.107400, 2021a. URL 24534477
hittps://apl. stt.aap...mmaaniccoooolaaaaaa htw.:aacee.ccctooaoooooooooooraa htt.op///aa...caaoooeooooooooorr
Alaaeldin Fl-Nouby, Gautier Izacard, Hugo Touvron, Ivan Laptev, Hervé Jegou, and Edouard Grave. Are large-scale datasets necessary for self-supervised pre-training? ArXiv, abs/21 12, 2021b.
12

---

Published as a conference paper at ICLR 2026
Chen al. A simple framework for contrastive learning of visual representations. In ICML, 2020.
Schuhmann et al. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint arXiv:2111.02114. 2021.
Zhou et al. Scene parsing through ADE20K dataset. CVPR, 2017.
Alex Fang, Albin Madappally Jose, Amit Jain, Ludwig Schmidt, Alexander Toshev, and Vaishaal Shankar. Data filtering networks. ArXiv, abs/2309.17425, 2023, URL htt:: happ semant icscholar. org/
CorD311D:263310455
Christoph Feichtenhofer, Yanghao Li, Kaiming He, et al. Masked autoencoders as spatiotemporal learners. Advances in neural information processing systems, 35:35946 35958,, 2022.
Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jicyu Zhang, al. Datacomp: In search of the next generation of multimodal datasets. Advances in Neural Information Processing Systems, 36, 2024.
Yossi Gandelsman, Yu Sun, Xinlei Chen, and Alexei Efros. Test-time training with masked autoencoders. Advances Neural Information Processing Systems, 35:29374-29385, 2022a.
Yossi Gandelsman, Yu Sun, Xinlei Chen, and Alexei A. Efros. Test-time training with masked autoen- coders. ArXiv, abs/2209.07522, 2022b. URL URL hwt.h:htp////.m /api www.stiicccooolar............aa. Crrppoopa//CaaaDa TD3
252283956
Yaroslav Ganin and Victor S. Lempitsky. Unsupervised domain adaptation by backpropagation. In International Conference on Muchine Learning, 2014.
Jin Gao, Jialing Zhang, Xihui Liu, Trevor Darrell, Evan Shelhamer, and Dequan Wang. Back to the Diffusion-driin driven adaptation to test-time coruptiinn. Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 11786-11788 11796, 2023.
Rohit Girdhar, Mannat Singh, Nikhil Ravi, Laurens van der Maaten, Armand Joulin, and Ishan Misra. Om- nivore: A single model for many visual modalities. 2022 JEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pp.
2022. URL https:://apa www.nttcccccnohoooococo...occ org/
Corpus 24606385
Rohit Girdhar, Alaacidin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala. Armand Joulin, and Ishan Misra. ImageBind: One embedding space bind them all. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15180-15190, 2023.
Priya Goyal, Piotr Dollár, Ross B. Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He.. Accurate, large minibatch sgd: Training imagenet in I hour. ArXiv, abs/1706.02677, 2017.
Lan-Zhe Guo, Yi-Ge Zhang, Zhi-Fan Wu, Jiejing Shao, and Yu-Feng Li. Robust semi-supervised learning when not classes have lahels. In Neural Information Processing Systems, 2022. URL httts:/tps
semant i wwweicisschhaacc
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross B. Girshick. Mask r-cnn. IEEE Transactions on Pattern Analysis and Machine Intelligence, 42:386-397, 2017.
Kaiming He. Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patter Recognition (CVPR), June 2020.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. CVPR, 2021.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint arXiv:1503.02255 2015.
Gao Huang. Yu Sun. Zhuang Liu, Daniel Sedra, and Kilian Q Weinberger. Deep networks with stochastic depth. In European conference on computer vision, pp. 646-661. Springer, 2016.
Andrew Jaegle, Sebastian Borgeaud, Jean- Baptiste Alayrac, Carl Doersch, Catalin lonescu, David Ding. Skanda Koppula, Daniel Zoran, Andrew Brock, Evan Shelhamer, Olivier J Henaff, Matthew Botvinick. Andrew Zisserman, Oriol Vinyals, and Joao Carriia. Perceiver 10: A general architecture for structured inputs & outputs. In International Conference Learning Representations, 2022.
13

---

Published as a conference paper at ICLR 2026
Swapnaa Jayaraman, Caitlin M Fausey, and Linda B Smith. The faces in infant-eerspecttive scenes change over the first year of life. PloS one, 10(5)x00133780, 2015.
Guoliang Kang, Lu Jiang. Yi Yang, and Alexander Hauptmann. Contrastive adaptation network for unsupervised domain adaptation. CVPR, 2019.
Bernhard Kerbl, Georgios Kopanas, Thomas Leimkuehler, and George Drettakis. 3d gaussian splatting for
real-time radiance field rendering. ACM Transactions on Graphics (TOG), 42:1 - 14, 2023. URL https
//api. semant icscholar wrtp/oorrssID::5522222222
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehcad, Alexander C. Berg, Wan Yen Lo, Piotr Dollár, and Ross B. Girshick. Segment anything. ArXiv, abs/2304.02643, 2023. URL https: //appppp.pp api. semant weschcccclhaaccc erg/ ::: 257952310
Eric Kolve, Roozbeh Mottaghi, Winson Han, Eli VanderBilt, Luca Weihs, Alvaro Herrasti, Matt Deitke, Kiana Ehsani, Daniel Gordon, Yuke Zhu, et al. Ai2-thor: An interactive 3d environment for visual ai. arXiv preprint
172.55474 2017.
Klemen Kotar and Roozbeh Mottaghi. Interactron: Embodied adaptive object detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 14860- 1186091 2022.
Alexander C Li. Yuandong Tian, Beidi Chen, Deepak Pathak, and Xinlei Chen. On the surprising effectiveness of attention transfer for vision transformers. Advances in Neural Information Processing Systems, 37: 113963-113990 2024a.
Da Li, Yongxin Yang. Yi-Zhe Song, and Timothy M. Hospedales. Deeper, broader and artier domain gener- alization. 2017 IEEE International Conference on Computer Vision (ICCV), pp. 5543 5551, 2017. URL https: / /api. semant ierrsaattiscchlaas erg/ Corppsrippppooeeeee 6697
Jeffrey Li, Alex Fang, Georgios Smyrnis. Maor Ivgi, Matt Jordan, Samir Gadre, Hritik Bansal, Etash Guha, Sedrick Keh, Kushal Arora, Saurabh Garg, Rui Xin, Niklas Muennighoff, Reinhard Heckel. Jean Mercat, Mayee Chen, Suchin Gururangan, Mitchell Wortsman, Alon Albalak, Yonatan Bitton, Marianna Nezhurina, Amro Abbas, Cheng- Yu Hsich, Dhruba Ghosh, Josh Gardner, Maciej Kilian. Hanlin Zhang, Rulin Shao, Sarah Pratt, Sunny Sanyal, Gabriel Ilharco, Giannis Daras, Kalyani Marathe, Aaron Gokaslan, Jieyu Zhang, Khyathi Chandu, Thao Nguyen, Igor Vasiljevic, Sham Kakade. Shuran Song, Sujay Sanghavi, Fartash Faghri, Sewoong Oh, Luke Zettlemoyer. Kyle Lo, Alaacldin El-Nouby, Hadi Pouransari, Alexander Toshev, Stephanie Wang, Dirk Groeneveld, Luca Soldaini, Pang Wei Koh, Jenia Jitsev, Thomas Kollar, Alexandros G. Dimakis, Yair Carmon, Achal Dave, Ludwig Schmidt, and Vaishaal Shankar. Datacomp-In In search of the next. generation of training sets for language models. arXiv preprint arXiv:2406. 11794, 2024b.
Yanghao Li, Hanzi Mao, Ross Girshick, and Kaiming He. Exploring plain vision transformer backbones for object detection. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviy, Israel, October 23-27, 2022, Proceedings, Part IX, pp. 280-296. Springer, 2022.
Tsung-Y Yi Lin, Michael Maire, Serge J. Belongic, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: Common objects in context. In European Conference Computer Vision, 2014.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. arXiv preprint arXiv:2304.0844. 2023.
Xuan Liu, Ying Huang. Hao Wang, Zheng Xiao, and Shigeng Zhang, Universal and scalable weakly-supervised domain adaptation. IEEE Transactions on Image Processing, 2024.
Yen-Cheng Liu. Chih-Yar Ma, Zijian He. Chia- Wen Kuo, Kan Chen, Peizhao Zhang, Bichen Wu, Zsolt Kira, and Péter Vajda. Unbiased teacher for superviie object detection. ArXiv, 0000000 2021a. URL https:: :/appl semant icscholar. org/ /ctpppss::2211111550
Yuejiang Liu, Parth Kothari, Bastien Van Delft, Baptiste Bellot-Gurlet, Taylor Mordan, and Alexandre Alahi. Tit+: When does self-supervised test-time training fail or thrive? Advances in Neural Information Processing Systems, 34:21808-22 21820, 2021b.
Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 11976-11986. 2022.
Tlya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on I Learning Representations, 2019.
14

---

Published as a conference paper at ICLR 2026
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining lask-agnostic visiolinguistic represen- tations for rision-and-a and--anddddaa language tasks. Advances in neural information processing systems, 32, 2019.
Jiasen Lu, Christopher Clark, Sangho Lee, Zichen Zhang, Savya Khosla, Ryan Marten, Derek Hoiem, and Aniruddha Kembhavi. Unified-IO 2: Scaling autoregressive multimodal models with vision, language, audio, and action, ArXiv, abs/2312.17172, 2023a. URL httpss:/app. /t semant
Corpus ID:26 ID:266573555
Jiasen Lu, Christopher Clark, Rowan Zellers, Roozbeh Mottaghi, and Aniruddha Kembhavi. Unified-IO: unified model for vision, language, and multi-modal tasks. In The Eleventh International Conference on Learning Representations, 2023b.
Xuan Luo, Jia-Bin Bin Huang, Richard Szeliski, Kevin Matzen, and Johannes Kopt. Consistent video depth estimation. ACM Transactions on Graphics (ToG), 39(4):71-1, 2020.
Zhaoyang Lv, Nickolas Charron, Pierre Moulon, Alexander Gamino, Cheng Peng, Chris Sweeney, Edward Miller, Huixuan Tang, Jeff Meissner, Jing Dong, Kiran Somasundaram, Luis Pesqueira, Mark Schwesinger, Omkar M. Parkhi, Qiao Gu, Renzo De Nardi, Shangyi Cheng, Steve Saarinen, Vijay Baiyya, Yuyang Zou, Richard A. Newcombe, Jakob Julian Engel, Xiaqing Pan, and Carl Ren. Aria everyday activilies dataset. ArXiv, abs/2402.13349, 2024. (Rttt:t::///app
semant isstiicssshi ceschhlarssr wttpa//ooo/oooooo....20.....
Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf. Communications of the ACM, 2020.
David Mizrahi, Roman Bachmann, Oguzhan Fatih Kar, Teresa Yco, Mingfei Gao, Afshin Dehghan, and Amir Zamir, 4M: Massively multimodal masked modeling. Advances Neural Information Processing Systems, 2023.
OpenAl. GPT-4 technical report, 2023.
OpenAT. Hello gpt-4o. https: http:::://ppeaaaa httpa////./e////eeeeeeee/eeeeeee
2024. Accessed: 2024-09-23.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Mahmoud Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Po-Yao Huang. Shang-Wea Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Xu, Hervé Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. DINOv2: Learning robust visual features without supervision. ArXiv, 2023.
Alec Radford, Prafulla Dhariwal, and llya Sutskever. Learning transfcrable visual models from natural language supervision. In ICML, 2021.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Tyya Sutskever. Zero-shot text-to-image generation. In International Conference on Machine Learning, 2021.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael S. Bernstein, Alexander C. Berg, and Li Fei-Fei. Image large scale visual recognition challenge. International Journal of Computer Vision, 11-22,, 2014.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev. LAION-5B: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 35:25278 25294, 2022.
Noam Shazeer. GLU variants improve transformer. ArXiv, abs/2002.05202, 2020.
Yang Shu, Zhangjie Cao, Mingsheng Long, and Jianmin Wang. Transferable curriculum for weakly-supervised domain adaptation. In AAAI Conference Artificial Intelligence, 2019.
Lauren K Slone, Linda B Smith, and Chen Yu. Self-generated variability in object images predicts vocabulary growth. Developmental science, 2(6))128866 2019.
Linda B. Smith and Michael Gasser. The development of embodied cognition: Six lessons from babies. Artificial Life. 11:13-29, 2005.
Kihyuk Sohn, David Berthelot, Chun-Liang Li, Zizhao Zhang. Nicholas Carlini, Ekin Dogus Cubuk. Alexey Kurakin, Han Zhang, and Colin Raffel. Fixmatch: Simplifying semi- supervised learning with consistency and confidence. ArXiv, abs/2001 abs/2001.07685, 2020. httpst //api..... semant cacho la ar www.rrorrooooooaaoee
210839228
15

---

Published as a conference paper at ICLR 2026
Julian Straub, Thomas Whelan, Lingni Ma, Yufan Chen, Erik Wijmans, Simon Green, Jakob Engel. Raul Mur-Artal, Carl Ren, Shobhit Verma, Anton Clarkson, Mingfei Yan, Brian Budge, Yajic Yan, Xiaqing Pan, June Yon, Yuyang Zou, Kimberly Leon, Nigel Carter, Jesus Briales, Tyler Gillingham, Elias Mueggler, Luis Pesqueira, Manolis Savva, Dhruv Batra, Hauke M. Strasdat, Renzo De Nardi, Michael Goesele, Steven Lovegrove, and Richard Newcombe. The replica dataset: A digital replica of indoor spaces. arXiv, 2019.
Yu Sun, Xiaolong Wang, Liu Zhuang, John Miller, Moritz Hardt, and Alexci A. Efros. Test-time training with self-supervision for generalization under distribution shifts. In JCML, 2020.
Matthew Tancik. Ethan Weber. Evonne Ng. Ruilong Li, Brent Yi. Justin Kerr, Terrance Wang, Alexander Kristoffersen, Jake Austin, Kamyar Salahi, Abhik Ahuja, David McAllister, and Angjoo Kanazawa. Nerfstu- dio: A modular framework for neural radiance field development. In ACM SIGGRAPH 2023 Conference Proceedings, SIGGRAPH 223, 2023.
Jesper E. van Engelen and Holger H. Hoos. A survey semi-supervised learning. Machine Learning, 109: 373-440, 2019.
Dequan Wang, Evan Shelhamer, Shaoteng Liu, Brumo Olshausen, and Trevor Darrell. Tent: Fully test-time adaptation by entropy minimization. In International Conference on Learning Representations, 2021. URL
attpps:/aape semant wwwcchcaa....o.ooo ozg/ 2ttp:/r......rr.rraaa02222222000
Jimmy Wu, Rika Antonova, Adam Kan, Marion Lepert, Andy Zeng, Shuran Song. Jeannette Bohg. Szymon Rusinkiewicz, and Thomas A. Funkhouser. Tidyhot: Personalized robot assistance with large language models. 2023 JEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pp. 3546-3553, 2023. URLpttts::// /api www.nteiasttttteeeeeeee wewantisccoomoaarrcccc.......rr. org Corpus : 258564887
Yuxin Wu, Alexander Kirillov, Francisco Massa, Wan-Yen Lo, and Ross Girshick. Detectron2. https
2019.
Zehao Xiao and Cees G. M. Snoek. Beyond model adaptation at test time: A survey. 2024. URL attps
//api. semant cccchllcccrr...cccc htt.rpaa.....oor/oooooso//22333
Qizhe Xie, Zihang Dai, Eduard H. Hovy, Minh-Thang Luong, and Quoe V. Le. Unsupervised data augmentation for consistency training. arXiv: Learning, 2019. URL htttt::////// :/apsi//pp..../ www.nticsccoooooo...oo...o ar, org/
[oorpuID:19587733
Tongkun Xu, Weihua Chen, Pichao Wang, Fan Wang, Hao Li, and Rong Jin. Cdtrans: Cross-domain transformer for unsupervised domain adaptation, ArXiv, 2021.
Chandan Yeshwanth, Yueh-Cheng Liu, Matthias Nießner, and Angela Dai. Scannet++: A high-fidelity dataset of 3d indoor scenes. In ICCV, 2023.
Sharare Zehtabian, Siavash Khodadadeh, Ladislau Bölöni, and Damla Turgut. Modeling an intelligent controller for predictive caching in ar/vr-enabled home scenarios. Pervasive and Mobile Computing. 71:101334, 2021.
Bowen Zhang, Yidong Wang, Wenxin Hou, Hao Wu, Jindong Wang, Manabu Okumura, and Takahiro Shinozaki. Flexmatch: Boosting semi-supervised learning with curriculum pseudo labeling. In Neural Information Processing Systems, 2021. URL
org/ Corpus ID :23901 6453
Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. iBoT: Image BERT pre-training with online tokenizer. In International Conference on Learning Representations, 2022.
Kaiyang Zhou, Ziwei Liu, Qiao, Tao Xiang, and Chen Change Loy. Domain generalization: survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45:4396-4415, 2021. URL httpa. / api.. semant icscholar. Ct:C/oooou:::2211100
16

---

Published as a conference paper at ICLR 2026
APPENDIX
Overview video
18
Additional related work.
18
Broader impact of TST
18
D Dataset Details.
19
E Additional downstream evaluations: Zero-shot cross-modal retrieval task
20
MST in the wild.
20
TST with the DINOv2 objective.
22
Benchmarking different self-supervised objectives under TST.
23
Using multimodal data during transfer: are additional modalities all you need?
23
Do we need trained networks outputs for modalities?
23
K What is the smallest unit of space specialize on?
25
TST with synthetic data.
25
M The role of the transfer dataset mix-in during pre-training,
25
TST with off-the-shelf transfer set.
26
Pscudo-labeler baselines
26
TST with semantic modalities
28
Do other self-supervised objectives benefit from specialized pre-training?
28
R Is distilling in the test space beneficial?
28
S Application for hardware data collection
29
Test time training with TST
30
Continual learning with TST.
31
Sampling ratio between test space and transfer data during TST pre-training
31
W Privacy aspect of using test-space data.
32
17

---

Published as a conference paper at ICLR 2026
X
Experimental Setup details
32
Pre-training details
32
2
Transfer details
32
Computational Resources.
35
A
OVERVIEW VIDEO
We provide a video with narration that gives a high-level summary of our paper. The video can be found our webpage
B ADDITIONAL RELATED WORK.
Domain Adaptation in vision (Li et al. 2017 Zhou et al. 2021) addresses the gap between source domain, where abundant data is available, and the target domain, where limited (Shu al. 2019) eiu al. 2024) or no data (Dong et al. 2021 Ganin & Lempitsky (014)) are available. TST, when initialized from an Internet-based model, as presented in Tab. [ can seen as an instantiation of adapting generalist model to the test space. However, TST differs by learning task-agnostic representations by self-supervised pre-training the test space, as opposed domain adaptation, which generally adapts pre-trained task-specific network (Xu et al. 2021 Kang et al. 2019).
Semi-Supervised Learning refers to line of work that attempts to learn task from a limited labeled dataset and massive unlabeled data (van Engelen & Hoos 2019)) Clearly. it involves consistency
regularization (Berthelot et al. 2019;Soon et al. 2020]]Xix et al. 2019) and pscudo-labeling (Guo)
al. 2022; Chen al. 2021 Zhang al.) 2021 Liu et al. 2021a) to generate supervision of unlabeled data, followed by joint training. Our framework, TST is closer in spirit to self-supervised learning, it tries to learn task-agnostic representation for the space, that we transfer for various downstream tasks like segmentation, detection and image captioning. Under semi-supervised learning, specialization with TST can be posed as using unlabelled data from the test space, as opposed to other sources like Internet or similar spaces.
Embodied Active learning. In another line of work, SEAL (Chaplot et al. 2021). Interactron (Kotar & Mottaghi| 2022) learn reinforcement learning based policy to collect supervision in a house to finetune an ohf-the-shelf MaskRCNN (He al. (017). or observe additional frames for multi-frame inference for object detection. As opposed to focusing on adapting task task-specific models, we focus on learning task-agnostic pre-trained representations over a test space.
C BROADER IMPACT OF TST
Privacy Aspect of Test-Space Training. This paper sheds light the critical role of data, and data sources in AI. The setting studied in TST suggests that it is possible to achieve competitive results without relying on large diverse internet-based datasets that essentially require the data of different users to be harvested and mixed. The paper shows training on only the deployment space data is an alternative worth considering and investigate the requirements of making that viable (e.g. utilizing multimodality being critical for achieving good results). This setting enables putting divider between the data of different users and can trigger intriguing questions on the role of data in AI, which is commonly assumed to be large and diverse datasets are essential for strong results. The studied setting requires training only on deployment space data, which can be done completely in-house and under the control of the user for privacy critical scenarios, to avoid any data contact with the external world. We provide an extended discussion on this in App.
TST as sandbox for Self-Supervised Learning. As mentioned in Sec. 3.4 all results presented so far use external transfer images - i.e., the annotated images used for training the transfers are from other spaces, rather than the test space. This choice matches real-world setup, it is infeasible for a user to spend time annotating images from their space. However, this introduces a minor suboptimality in quantifications, since the transfer set may exhibit distribution shifts from the test.
18

---

Published as a conference paper at ICLR 2026
Muttinodal Scratch
(Sensors only)
TST-MM
51.25
Upper Bound
31.05
37.
Fully supervised Large Data
Figure 12: Label Propagation with TST. We find that the trends in performance under the Label Propagation, with no transfer set distribution shift, are similar to what we observe in Tab. ] and Fig. thereby reassuring the efficacy of pre-training in the test space. The lower bound here is learning from scratch (i.e., no pre-training and learning using the small transfer set only). The upper bound approximated by a fully supervised model trained with a large number of annotated segmentation images from the test space. All methods share the same model architecture. The results here correspond to semantic segmentation on Scannet++ (Yeshwanth et al. 2023).
space (e.g., some test objects may appear in the transfer set have vastly different appearances). This mismatch makes it difficult to attribute poor results primarily to ineffective pre-training.
To study the effect of pre-training, a small set (+ 20) from the same test space. This is akin to having a few annotated images from the test space and expecting the model propagate the labels to the of the images of the test space. Therefore, we name this setup Label Propagation. This allows us to leverage this setup as sandbox for studying various self-supervised pre-training methods in the test space. We present the results of various TST variants and relevant baselines under this setup in Fig. 12
experiment with controlled setup where transfer images
D DATASET DETAILS.
1. Scannet++ (Yeshwanth et al. 2023) is large dataset of real-world indoor spaces containing sub- millimeter resolution laser scans, paired with DSLR and iPhone RGB images.
Pre-training dataset. We use Scannet++ Yeshwanth et al. 2023) scenes test space. We use a mix iPhone and DSLR images for pre-training. with the iPhone containing 19165 samples and the DSLR dataset containing 15000 samples.
Transfer dataset. We use non-test space buildings for creating a transfer set of 40000 RGB, segmentation pairs. Note that Scannet++ (Yeshwanth et al. (023) only provides 3D instance annotations, which we project to 2D to create a semantic segmentation dataset.
Evaluation. We evaluate on semantic segmentation in the test space. The test dataset for evaluation contains 3000 RGB image samples. Note that we collect separate held out set from the test space for this stage.
Replica (Straub et al. 2019 provides high quality 3D reconstructions of real indoor spaces.
Pre-training dataset. We use Omnidata (Eftekhar et al., 2021), to densely sample Replica meshes corresponding to the scenes to build our pre-training dataset, Drr, containing 84889 samples. We defer the details of the sampling procedure to Omnidata (Eftekhar et al.
2021
Transfer dataset. Similar to Scannet++ (Yeshwanth et al. 2023), we collect transfer set from another set of Replica scenes that are different than the scenes used during pre-training. We collect 20000 RGB images and semantic segmentation masks, and use it as our transfer dataset, D,.
We evaluate on semantic segmentation in the test space. We collect 5000 images and semantic segmentation annotations from the same test space we pre-train on, and report performance on it. We leverage Omnidata annotation pipeline to extract the segmentation labels.
Evaluation.
of
ProcTHOR DDetkee et al. 2022) It includes procedurally generated house-like like- environments. We use 5 procedurally generated houses test space.
19

---

Published as a conference paper at ICLR 2026
Pre-training dataset. We randomly sample various agent x, y, 2 positions and orientations along its axis in the test space, and collect RGB-D images at these points. This sampling process yields a total of 163767 samples. We collect data by sampling densely across the test space and use it as our pre-training dataset Drr-
Transfer dataset. For the transfer data D,. we collect a small dataset of 20000 RGB and task annotation pairs, from 800 houses generation using different asset and layout distritbiittion to it.
distribution than the pre-training test space, thereby making them
Evaluation. We evaluate TST and present results on three tasks, namely semantic seg- mentation, object detection and image captioning. We collect a test set with 5000 samples from the same test space, where we performed pre-training, and report performance on We use the A12-THOR (Kolve et al.) 2017) metadata to extract semantic segmentation and object detection labels for evaluations. For captioning, we generate ground truth captions by prompting GPT-4o (OpenAl 2024) with privileged information, e.g. class names and bounding boxes. Finally, we additionally evaluate our model on cross-modal retrieval (in Sec.
E ADDITIONAL DOWNSTREAM EVALUATIONS, ZERO-SHOT CROSS-MODAL RETRIEVAL TASK
Image to Depth
Depth to Image
Method
@
R R@S
R @ 10
R@D 1
R@5
R@ 10
4M-21 (Bachmann et al. 2024) TST MM
1.06
2.18
3.08
1.0
2.76
3.66
25.48
37.00
41.58
24.32
36.46
40.82
Table 2: Zero-shot Cross-modal retrieval. When performing the image-to-depth and depth-to-image cross-modal retrievals on the test space data using the predicted CLJIP embeddings, we observe that the TST-MM method constantly outperforms the Internet-based 4M-21 (Bachmann et al.) 2024).
As mentioned in Section 4.1 we present results on zero-shot cross-modal retrieval to further support our framework TST. Specifically, we evaluate the performance of models pre-trained with TST-MM on RGB-to-Depth and Depth-to-RGB retrieval. To perform retrieval using Internet-based model, 4M-21 [Bachmann et al. 2024) and TST-MM, MM, we utilize their cross-modal generation capabilities by transforming depth and RGB images into CLIP embeddings, and then apply retrieval directly on the CLIP embeddings. Since 4M-21 (Bachmann et al.) (024) and TST-MM generate feature maps for CLIP as the target modality from RGB and Depth images, maps to obtain global CLIP embeddings. Cross-Modal retrieval evaluates TST-MM on two fronts: i) How well test-space paired modality inputs are aligned in the model representations internally, and ii) How effectively TST-MM can perform cross-modal generalization. For the evaluation, we report zero-shot recall at various thresholds on test set of 5000 samples from ProcTHOR (Deitke al. 2022) test space. The results are presented in Tab. 2 We also present qualitative examples in Fig. 13 Note that given our method TST-MM has access the test space, can retrieve RGB to Depth and Depth to RGB much more effectively than models based on external data like the Internet.
apply mean peoaniggoling the feature
We find that TST-MM substantially outperforms 4M-21 (Bachmann et al. 2024). The recall perfor- mance of TST-MM further increases when evaluated on R@5 and R@ TO, whereas Internet- based 4M-21 (Bachmann et al. 2024) shows diminishing returns. This underscores the effectiveness of test- space training, where specialization itself is crucial for learning test-space-aligned representations.
F TST-MM DEPLOYMENT IN THE WILD.
In addition to real-world results Scannet++ 4.3 we also experiment with the deployment of TST, in a custom space. We collect a 15 15-minute video of a meeting room and used the resulting frames for pre-training described in Sec. 4.1 followed by transfer
the ScanNet++ (Yeshwanth et al.) 2023) transfer set (Sec. D). We evaluated TMT-MM and the baselines on the semantic segmentation task. We evaluate TST MM and the baselines on the semantic segmentation task. Tab. shows that
20

---

Published as a conference paper at ICLR 2026
RGB Depth
Query input (RGB)
GT rotrioval (Depth)
Intemet 4M-21
17
TST (Ours)
Depth to RGB
Quary input
(Depth)
retrieval (RGB)
Interet 4M-21
TST (Ours)
Figure 13: TST-MM cross-modal retrieval predictions. TST-MM retrieves corresponding RGB images from query Depth input and Depth images from RGB input more accurately than the Internet- based 4M-21 (Bachmann et al.) 2024) model.
21

---

Published as a conference paper at ICLR 2026
Input
Scratch
4M-21
TST-MAE
TST-MM (Sensors)
TST-MM
Figure 14: TST-MM predictions deployment in the wild. We showcase the qualitative results for on the semantic segmentation task against the Internet-based pre-trained model 4M- 21 Bachmann et al.) 2024) and scratch (to-pretraining.. TST-MM predictions are notably better across object categories, showing the value of access to test space and the deployment potential of TST-MM.
for this custom scene deployment, pre-training on the test-space through TST-MM outperforms the Internet-based baseline 4M-21 (Bachmann al. 2024). The qualitative comparison in Fig. shows that TST-MM's predictions are notably better than those of the Internet- 4M-21 (Bachmann
al. 2024).
G TST WITH THE DINOV2 OBJECTIVE.
In this section, explore how TST trained with DINOY2 objective, TST-DINO from scratch, compares with its Internet counterpart trained on 142M images from the Internet (Oquab et al. 2023). Fig. shows that pre-training only data from the test space substitute large-scale Internet pre-traming. This further underscores that TST framework extends to other self-supervised objectives (Oquab et al. 2023) beyond masked modeling for specialization. However, we find that
22

---

Published as a conference paper at ICLR 2026
Scratch
Internet
TST
Segmentation
40
16
27.17
25.42
3.78
Method
mloU
Scratch
21.82
Scratch 4M-21
CLIP
DINOV2
SAM
TST-MM
4M-21
54.58
TST-MM
59.11
Figure 15: TST works with off-the-shelf trans- fer set. For Replica (Straub et al. 2019), we find that even when we use ADE20k (et al.) 2017) transfer set, TST-MM outperforms Internet- based generalist models, showcasing the impor- tance of having access to the test space, agnostic to the transfer set.
Table Semantic segmentation performance. Comparison of mloU scores across different training methods.
TST-MM, which uses multimodal masked modeling outperforms other unimodal self-supervised objectives like DINOv2 (Oquab al. 2023) and MAE (He al. 2021).
H BENCHMARKING DIFFERENT SELF-SUPERVISED OBJECTIVES UNDER TST.
We compare the performance TST with different pre-training objectives such as multimodal masked modeling (Bachmann al., 2024). DINOv2 (Oquab al., 2023) and MAE (He al. 2021). As shown in Fig. 21, we find that multimodal masked modeling (TST-MM) to be the most performant among the self-supervis objectives we explored. However, note that all objectives show specialization trends as presented in Fig. and Fig.
USING MULTIMODAL DATA DURING TRANSFER: ARE ADDITIONAL MODALITIES ALL YOU NEED?
In Sec. discuss that pre-training multimodal data with multimodal masked modeling objective in our TST-MM method leads to a specialist model more performant than other baselines. Here check if this superior performance is solely due access to the additional modalities besides RGB that simplify the task, rather than representation learning value through multimodal pre-training?
Figure 16 shows the performance of model trained from scratch, only on the semantic segmentation task using multimodal data, ie, no supervieed pre-trainin traininig.. It receives all of the modalities as input and predicts the corresponding segmentation map during both training and testing. We find that this model performs poorly compared to TST-MM, which leverages multimodal data during pre-training and only RGB input during transfer and test. This experiment signifies the importance of pre-traine pre-training using multimodal data from the test space.
J Do WE NEED trained NETWORKS OUTPUTS FOR MODALITIES?
As described in Sec. TST-MM leverages off-the-shelf pseudolabel networks to create additional modalities. This is akin distilling their feature space test space data, without directly accessing the external data that these networks were trained on.
This raises intriguing question, is it the trained feature space of network Radford et al. (202) Girdhar et al. (202)), that makes it a good modality, or simply the architecture inductive bias that creates different transformation of the input image is sufficient to drive performance. To study this we present an analysis in Fig. [77] which shows that indeed, the learned feature space of pseudolabel network is crucial in making it as useful modality.
23

---

Published as a conference paper at ICLR 2026
Unimodal Scratch
Multimodal Scratch
TST-MM
Semantic Segmentation
Object Detection
60.85
60
50
49.38
50
40
28.62
24.59
26.29
20
19.19
20
10
10
Figure Multimodal pre-training crucial in TST-MM. We compare our method to the model that also has access multimodal data during supervised training transfer data. "Multimodal Scratch" uses all modalities, including RGB, as imput and predicts the corresponding semantic segmentation map during both training and testing. TST-MM, which uses only RGB as input during transfer, significantly outperforms the multimodal scratch model, signifying the importance of multimodal pre-training.
Scratch
TST
Semantic Segmentation
35
30
28.0
25
20
18.79
13.1
10
5
Scratch
TST-MAE
TST-MM
TST-MM
(Corrupt)
Figure 17: TST with untrained pseudolabel modalities. We train variants of TST, with RGB-only MMST-MAE), with, RGB and 4 pscudolabel modalities (Depth, Surface Normal, CLIP Radford et al.) (202I), Imagebind Girdhar et al. (2023)). ) with trained networks (IST-MM), and with untrained or call them corrupt networks (TST-MM (Corrupt)). We find that using untrained pseudolabels as modalities drops performance below the RGB-only baseline, suggesting that the model capacity spent learning those modalities, does lead useful representation. All results ViT-S backbone model, on Scannet++ Yeshwanth et al. (003))
24

---

Published as a conference paper at ICLR 2026
Ring 1
Ring 2
Ring 3
21
18
mlou
15
TST-MM
TSEMM
TST-MM
Scratch Baseline
Scratch Baseline
Scratch Baseline
9
Transfer-only Baseline
Transfer-only Baseline
Transfer-only Baseline
3-
Ring
Ring 2
Ring
Ring 1
Ring 2 Test Spaces
Ring 3 Ring 1
Ring 2
Ring 3
Figure 18: Smallest unit of space specialize on. We reduce the test space size, that can special- ize and pre-train models with TST-MA We compare it with an Internet pre-trained model (Bachmann al. 2024). and a baseline that pre-trains only on the transfer set. We also find that training on a ring smaller than the test ring, leads to diminished performance.
K WHAT IS THE SMALLEST UNIT OF SPACE WE CAN SPECIALIZE ON?
In the results presented so far, we have shown that TST can specialize on test spaces at the size from 1-8 houses. However, this raises question, what is the smallest unit of space we specialize on? To probe this, we reduce the size of the test space and evaluate if TST can specialize to it. We consider model trained via TST specialized, if generalist, when evaluated on that test space. We reduce the test space, in the form of concentric rectangles, starting with a room, and then reducing the size of the rectangle. For each rectangle, we pre-train a specialist model via TST. We compare this against 4M-21 (Bachmann et al. 2024), on the task of semantic segmentation. As shown in Fig. [8 we find that we can specialize on single room (ring 3) that has an area of 20 square metres, and this trend continues as we reduce it down to ring 2, which 11 square metres and ring which is square metres. Reducing the test space, below square metres results in failed specialization, where the pre-training on just the transfer pre-training performs the best.
outperform an off-the-shelf Internet-based
L TST WITH SYNTHETIC DATA.
Recent advances in novel view synthesis (Mildenhall et al. 2020 Barron al. 2021 Kerbl et al. 2023) have enabled realistic renderings of indoor spaces, opening up the potential for generating synthetic training data. In TST, we leverage existing indoor scene datasets (Yeshwanth al. 2023 Straub et al.) 2019). which include real RGB images captured with DSLR/iPhone cameras or rendered from 3D meshes, to develop specialized models for specific test spaces. This leads to a key question: if novel view synthesis model can generate images from arbitrary viewpoints in test space, can serve as a controllable data generator- can its outputs match real images in utility?
To explore this, we train NeRF model (Tancik et al. 2023) using DSLR images from Scan- Net++ (YeshwYys et al. 2023), and render images from the same camera poses. We then pre-train two models-on one using real DSLR images and the other using NeRF-rendered views-to assess the performance gap. As shown in Fig 19, NeRF-generated images result in negligible performance loss compared to real images. This suggests interesting future direction: if high-fidelln NeRF models can be trained with fewer input images, they could act as steerable data generators, reducing the need for extensive real-world data collection in test environments.
M THE ROLE OF THE TRANSFER DATASET MIX-IN DURING PRE-TRAINING.
We study the role of mixing images from the test space and transfer datasets during pre-training, as mentioned in Sec. 4.1 Tab, shows that using only test-space data outperforms both pre-training on large-scale Internet data and using only transfer images, but mixing test space and transfer data achieves the best performance. We hypothesize that sceing transfer images during pre-training helps the model better align with the fine-tuning stage the transfer dataset. Note that
25

---

Published as a conference paper at ICLR 2026
Semantic Segmentation
40
35
33.51
32.73
25
mlou
15
0
DSLR
NeRF
Figure 19: TST with synthetic data. We replace real DSLR images in ScanNet+ Yeshwanth et al. 2023) with NeRF(Miidenhall al. 2020 rendered images from the same training viewpoints. We find that this results in only negligible performance, hence demonstrating that NeRF"s output quality known poses is sufficient to substitute high-quality DSLR images.
Table 4: Ablating the use of transfer RGB frames during pre-training. As noted in Sec. 4.1 additionally use RGB images from the transfer set during pre-training. We ablate this choice by comparing all three dataset configurations. We use the ViT-S backbone for all models.
Test Space
Transfer
Segmentation (mloU 1)
42.01
X
50.21
TST
/
56.96
4M-21
46.12
cannot be explained by more diverse data in the transfer set, adding non-test spaces decreases the specialization performance, as observed in Fig.
N TST WITH OFF-THE-SHELF TRANSFER SET.
As noted in Sec. 4.1 for each dataset (Deitke al. 2022 Straub al. 2019 Yeshwanth al. 2023 we explore, the transfer set comes from similar domain, as the pre-training dataset, albeit from non-test spaces. naturally raises the question, what we use an existing off-the-shelf semantic segmentation dataset like ADE20K fet al.) (2017) as transfer set. Does TST generalize and result in performant specialist models, or an in-domain transfer set necessary? To probe this, for the Replica (Straub al. 2019) dataset, from Replica as the transfer set, we use ADE20k (et al. 2017). Fig. [5 shows TST-MM outperforms various generalist models (Bachmann et al. 2024 Oquab et al. 2023 Radford et al.) 2021), even when using ADE20k (et al. (017) as the transfer set. All models are evaluated the test space from Replica (Straub al. 2019) semantic segmentation, with ViT-B backbone.
pre-train TST-MM, but instead of using non-test spaces
O PSEUDD-LLABELER BASELINES
various off-the-shelf networks to pseudolabel RGB data, and create additional (optional) modalities for TST-MM. We present a comparison for TST-MM against these pseudolabel baselines in Tab. 5 TST-MM and TST-MM (adapted) outperform all pseudolabel baselines, suggesting the benefit of pre- pre-training the test space with them, via multimodal masked modeling.
As mentioned in Sec. 4.3
26

---

Published as a conference paper at ICLR 2026
Semantic Segmentation
62
60.85
Internet
60
TST
58
56
n
55.1
54
53.74
52
50
DINOV2
TST-DINO
TST-MM
Figure 20: TST with DINOv2 objective outperforms its Internet counterpart. We compare the performance of DINOv2 pre-training in the test space, ISI-DINO, with DINOv2 pre-trained on the large-scale Internet dataset of 142M images (Oquab al. 2023). ISI-DINO outperforms its Internet counterpart, showing the value specialization. Yet, TST TST-MM with multimodal masked modeling achieves the best performance.
Semantic Segmentation
65
TST
60.85
60
mloU (t)
55.1
55
50
46.29
45
40
TST-MAE
TST-DINO
TST-MM
Figure 21: Comparison between different pre-training objectives under the TST framework. We compare
the performance of different pre-training objectives using IST on semantic segmentation task. We find that multimodal masked modeling (ISI-MD) achieves the best performance followed by ISI-DINO. All the three objectives were trained using the ViT-B model size on the ProcTHOR (Deitke et al. 2022) datasct.
27

---

Published as a conference paper at ICLR 2026
Semantic Segmentation
Object Dotection
Captioning
Method
Scannet++ mioL
PmeTTHOR
Replica mloU
Scunnet++
PraCTHOR mAP1
PraCTTOR
mloU
mAP
CIDEr SPICE
ImageBind Girdhar al. 2023
25.40
44.54
12.78
6.78
32.54
CLIP (Radiord aaa 2021
23.02
48.66
20.92
19.75
38.47
1844
16.2
Pscudo-labelers
Mask mai
29.42
50.28
22.68
(li
SAM Kirtlov oral M025
34.75
56.72
28.51
Specialist
IST-X8
32.87
Pre-training
TST XM (adapted)
36.44
60.59
34.53
35.83
51.25
39.9
20.5
Table 5: Comparing TST-MM against pseudolabels. We find that TST MM outperforms all pseu-
dolabels underscoring the value of pre-training on them via multimodal masked modelling in the test space.
P IST WITH NO SEMANTIC MODALITIES
TST-MM includes modalities obtained as outputs from different off-the-shelf models. Tab. T shows that TST-MM outperforms each individual model used modality. Since transfer tasks are semantic segmentation and object detection, we further study if having off-the-shelf models trained on related tasks modalities is crucial for final performance.
We present three experiments using the ViT-B backbone on ProOTHOR (Deitke et al. 2022). For each experiment, we drop one of the following modalities: Semantic segmentation, Object detection, iii) Semantic segmentation, Object detection, and SAM edges. Tab. [ shows the results for each model when transferred semantic segmentation and object detection. We find that even though the performance drops if we remove all three modalities, TST-MM still outperforms the Internet- based 4M-21 (Bachmann et al. 2024) model.
Modalities
Task
Method
Semantic segmentation
Object detection
SAM
Segmentation (mloU1)
Detection (mAPf)
Others
edge
60.85
49.38
59.43
49.58
TST-MM
/
59.38
49.34
X
X
x
55.39
45.97
4M-21 Bachmann al., 2024]
53.24
41.43
Table 6: The effect of semantic modalities in TST-MM. As the results demonstrate, removing the semantic segmentation and object detection modalities obtained from off- off-the-shelf networks does not significantly hurt the TST-MM's performance on the downstream semantic segmentation and object detection tasks. When all three semantic modalities are removed, we observe drop in performance, but TST-MM still outperforms the Internet-based 4M-21 (Bachmann et al.) 2024) model, demonstrating the value specialization.
Q DO OTHER SELF-SUPERVISED OBJECTIVES BENEFIT FROM SPECIALIZED PRE-TRAINING?
In Sec. 4.3 present results with TST-MM, which employs multimodal masked modeling. However, as mentioned in Sec. 3.3 TST also supports other self-supervised objectives. Fig. 22 shows that pre-training objectives, DINOv2 (Oquab al. 2023), and RGB-only MAE (He al. 2021 exhibit similar specialization trends.
R IS DISTILLING IN THE TEST SPACE BENEFICIAL?
As discussed in Sec. 4.3 we scale modalities by
RGB data with various Internet-
based models (Oquab al. 2023 Radford al. 2021 Cheng al.). This process of creating
28

---

Published as a conference paper at ICLR 2026
TST DINO
Test Space
42.6
40.1
53.3
46.8
36.3
41.7
46.0
47.7
II
Pre-training Space
Figure 22: Specialization using other objectives. We further demonstrate the specialization using other pre-training objectives including MAE and DINOv2. The results shows similar specialization trend considering the other pre-training objectives.
Test space
Non-test Space
Semantic Segmentation
55
54.26
47.12
45
39.19
35
29.96
28.43
25
CLIP
DINOV2
Mask2former
Figure 23: Distillation in test space. We find distilling over data from the test space, from various off-the-shelf models, results in more performant models in the test space. All results here with the ViT-B backbone, on ProcTHOR (Deitke et al. 2022).
additional modalities, and pre-training on them has enabled powerful multimodal foundation mod- els (Mizrahi et al. 2023 [Bachmann et al. 2024 2022). This form of pre-training can also be seen as distilling the knowledge from these powerful off-the-shelf networks into a single unified model.
With TMT-MM, we also distill from various off off-the-sshe the-shelf networks like CLIP (Radford et al. 2021), DINOv2 (Oquab et al.) 2023) with masked modelling (He et al. 2021| Mizrahi et al.) 2023). Results in Tab. suggest that distilling with multiple modalities on the test space, results performant specialist models. However, to disentangle the effect of multimodality and distillation,
take it one
step further to probe whether just distilling in the test space, provides some additional benefit, over non-test spaces? Therefore, we distill, CLIP RRdffrrd al.) (021), DINOv2 (Oquab et al. 2023) and Mask2former (Cheng et al.) in the test space, and compare it with distilling in an IID, but non test space, and report the results in Fig. 23 semantic segmentation in ProcTHOR (Deitke et al., 2022). We find that distilling over data from the test space is more performant than the data from non-test spaces, underscoring the importance access to the test space for specialization.
APPLICATION FOR HARDWARE DATA COLLECTION
As discussed in Sec. 3.2 TST be extended to leverage more hardware- based modalities. such as IMU, GPS, Audio, which can be found on most common user devices. such as iPhone. To facilitate future research in this area, release iOS application that enables anyone to collect aligned
29

---

Published as a conference paper at ICLR 2026
Multi-modal Multi-sensory Data
Custom Space
Oven LoMe
Figure 24: iOS application for custom data collection. It leverages the open source ARKit API to stream outputs of various sensors such as RGB, LiDAR, IMU, magnetometer, and ambient lighting, and supports paired data collection.
MAE (He al. 2021
Before TTT
After TTT
pre-trane veettraning
(mloU 1)
(mloU t)
Internet TST
34.54
39.28
35.48
42.41
Table 7: TST with Test-Time Training. Before TTT corresponds to the performance of the models directly after the transfer training without any test-time training, whereas after TTT shows the results when test-time training the test samples performed. Both models ViT-B backbone and evaluated on the semantic segmentation on ProcTHOR (Deitke et al. 2022
multimodal data from RGB-D and additional hardware sensors present on an iPhone. Fig. 24 shows overview of our application.
T TEST TIME TRAINING WITH TST
As noted in Sec. 2 we share a similar goal with Test-time Training (TTT) (Sun et al. 2020 in bridging the train-test divide. TTT does it from the lens of inference time optimization specialize to particular test instance, whereas TST attempts to specialize to given test space by pre-training in it.
However, in practice, these strategies can be orthogonal and complement each other. We can potentially apply TTT to model pre-trained with TST, to improve performance. To benchmark how this combination works, we conduct an analysis where apply Test-time training with masked autoencoders (TTT-MAE) (Gandelsman et al. 2022a), with two pre-trained methods, MAE (He et al.. 2021) pre-trained on Internet data and TST MAE pre-trained on the test space.
In TTT-MAF, we first start with a pre-trained MAE ViT-B encoder as the backbone, and train task-specific head on the transfer set. During the test phase, the backbone is further tuned using the masked modeling objective for each test sample individually. This adaptive tuning enhances the model's performance the downstream task for the given test samples.
As presented in Tab. TTT-MAE improves the mloU results for both the Internet pre-trained backbone and TST-Ma MAE. However, find that TST-MAE gets significantly more improvement than internet-based MAE (He et al. (02T). Both models use a ViT-B backbone and are tested on the semantic segmentation the TrocTHOR (Deitke al. 2022) dataset.
30

---

Published as a conference paper at ICLR 2026
U CONTINUAL LEARNING WITH TST.
As discussed before, when pre-training is performed on the exact same test space we deploy TST results in the most performant models. However, TST specializes the test space, and all its characteristics, at the state when the pre-training data collected. Therefore, natural question to ask is, what happens the test space undergoes some changes after data collection? This could include changes in the lighting of the space minor object placements. We begin by investigating these changes lead to a drop in performance for the TST model trained on the original test space. Thereafter, we leverage the ability of ProcTHOR create perturbed version of the test space. Note that the overall layout and assets remain exactly the same, only the lighting and placement of small objects are varied.
randomize object placements and lighting to
We first evaluate the performance of TST-MM pre-trained on the unperturbed test space, on the perturbed test space (Fig. 25] right), and we find that it experiences drop as compared to its performance in the original test space, (Fig. 25 left). However, as we continually pre-train the model by collecting data in the updated test space (TST-MM (CL)), quickly recovers the loss in performance, and is still highly performant as compared to Internet-basee generalists (Bachmann iet al. 2024). This suggests that even under the condition that the test space undergoes changes, by simply continuing data collection in the test space, TST can continually improve its performance, without any access to external data.
Scratch
4M-21
TST-MM
TST-MM (CL)
Original
Randomized
63
63
52
52
mlou
42
ml hou
42
31
31
21
10
10
Figure 25: Continual Learning with TST. We study the performance of TST-MM, the test space, undergoes lighting and minor object placement changes. The plot on the left, shows the result of the baselines on the original test space, without any changes. On the right, we present results after the test space has undergone lighting and object displacements. As expected, the TST-M, trained in the original test space, looses some performance, however as we continually train by collecting pre-training in the perturbed test space, we find that TST-MM (CL) quickly recovers performance.
V SAMPLING RATIO BETWEEN TEST SPACE AND TRANSFER DATA DURING TST PRE-TRAINING
As mentioned in Section 4.1 found mixing RGB images from the transfer our pre-training data beneficial for performance. To study the interplay of this dataset mix further, we analyze the effect of the sampling frequency of the samples from the transfer set and the test space data during tre-train A ratio of 1/1 implies that half the samples in pre-training come from the test space data and the other half from the transfer dataset. We pre-train the TST-MM model using both small and base sizes on the same test space as in Tab. T in the ProcTHOR (Deitke et al. (022) dataset under different ratios. The models are then transferred and evaluated on the semantic segmentation task. Tab. demonstrates the results for various ratio configurations and their effect different model sizes. First, we find that in all cases, TST-MM consistently outperforms et al. 2024) models of the same size. Secondly, note that the performance of the bigger ViT-B based models is not sensitive to the ratio sampling transfer and test space data, whereas for smaller ViT-S based models, ratio of 1/1 seems to be reasonable default choice.
snernet-bbseed 4M-21 (Bachmann
31

---

Published as a conference paper at ICLR 2026
Transfer set Test space set sampling ratio
Model Size
4M-21
1/1
1/4
1/8
1/16
Small
61.01
59.03
56.96
57.01
46.12 12
Base
60.36
60.65
60.85
60.36
53.24
Table The effect of the sampling ratio between the test space and transfer data during pre- training, We report transfer performance on semantic segmentation as we vary the sampling ratio between transfer and test space data during pre-training. There is no significant difference in results across different ratios for the base model, and for the small model, the best result is obtained with one-to-one sampling ratio between the transfer set and the test space set. Irrespective of the sampling ratio observe TST models always outperform Interettbaseed 4M-21 pre-training (Bachmann al. 2024)
W PRIVACY ASPECT OF USING TEST-SPACE DATA.
TST presents risk and an opportunity, in terms of data privacy, and we believe the opportunity could be more consequential. The risk stems from the data collected in the test space, which may need to be sent out for pre-training, as on-device compute may not be sufficient. While true, this does not seem insurmountable, as TST can ultimately enable training compact, dedicated model for the user space, which may become possible with some in-house compute for sensitive domains.
On the other hand, continued commitment to the dominant paradigm of generalist models, trained on massive data, requires harvesting and pooling the data across a large number users to create training datasets. This poses significant privacy risk just not for the specific user that deploys the model, but for all the other users whose data had to be harvested to enable the creation generalist.
Taking further steps toward developing an effective test-space training paradigm with reduced need for external data, can decrease the necessity to pool diverse users' data to make large training datasets, thereby reducing the potential privacy abuse well unwanted commodification of data by corporate entities.
Besides the ethical aspect. it is interesting performant models (how much data, what kind of data, for pre-training or distillation, are all data equal, etc). The common assumption that large, diverse data in raw form (images and annotations) is the way to go does not have to be the full story, as several papers, including Attention Transfer al. 2024a) and TST allude
concretely understand the role of data in developing
X EXPERIMENTAL SETUP DETAILS
X.1 PRE-TRAINING DETAILS
Initialization. For TST-MM, we use two initializations for pre-training. Unless stated otherwise.
pre-train model from scratch, following the hyperparameters in Tab. 9 Additionally, for
adaptation results in Tab. D we start from a pre-trained 4M-21 (Bachmann et al. 2024) model and
finetune it with the hyperparameterss in Tab. 10
we use the implementation from the original DINOv2 r We use the default provided training configuration files and train model with the ViT-B/14 backbone for 300,000 steps with a batch size of 1024.
DINO Pre-training, For the DINO TST pre-training in Sec. Q
X.2 TRANSFER DETAILS
Semantic segmentation: For semantic segmentation РrocTHOR (Deitke et al. 2022).
Replica (Straub 2019) and Scannet++ (Yeshwanth et al. 2023) datasets, we use the ViT
32

---

Published as a conference paper at ICLR 2026
Configuration
Small
Base
Training length (n tokens)
100B
500B
Warmup length (n tokens)
10B
Optimizer
Adam W Loshchilov & Hutter 2019
Opt. momentum
31, P32
0.9,
Base learning rate (Goyal et al. 2017)
1e-4
Batch size
4096
Weight decay
0.05
Learning rate schedule
Cosine decay
Feedforward activation
SwiGLU (Shazeer
2020
Input token budget Target token budget
128
256
128
256
Input and target a
Mixture (Bachmann al.
2024
Masking strategy
Mixture Bachmann et al.
2024
Image resolution
2242
Augmentation
Random Crop
Repeated sampling (Feichtenhofer et Data type
2022)
bfloat 16 BBurgess et al 2019)
Table 9: Pre-training settings for scratch initialization. initialized from scratch.
Training configuration for TST-MM
Configuration
Small
Base
Training length (n tokens)
100B
Warmup length (n tokens)
10B
Optimizer
Adam W (Loshchllov & Hutter 2019
Opt. momentum
F1, 13 0.9,
Base learning rate Goyal et al. 2017
5e-5
Batch size
4096
Weight decay
0.05
Learning rate schedule
Cosine decay SwiGLU (Shzzeer
Feedforward activation
2020
Input token budget
128
256
Target token budget
128
256
Input and target a Masking strategy
Mixture [Bachmann al.
2024
Mixture Bachmann et al.
2024
Image resolution
224"
Augmentation
Random Crop
Repeated sampling (Feichtenhofer et 2022
Data type
bfloat 1 Burgess et al
2019
Table 10: Pre-training settings for Internet initialization. Pre-training configuration for TST starting from the the pre-trained 4M (Bachmann al. 2024) model weights.
33

---

Published as a conference paper at ICLR 2026
encoder from the pre-trained models with decoder head, based on the ConvNext (Liu al. 2022 network with a depth of 4. This decoder head is initialized from scratch. Training details are provided in Tab. [T] On Replica (Straub et al. 2019), and ProcTHOR (Deitke et al. 2022), pre-trained models are transferred and evaluated using transfer training dataset of 20,000 images and evaluated on 5000 images sampled from the test space. On Scannet++ (Yeshwanth et al.
2023). we transfer
dataset of 40,000 images and evaluated on 3000 images from the test space.
Configuration
Small
Basc
Fine-tuning epochs
64
Warmup epochs
Optimizer
Adam W Loshchilov & Hutter 2019
Opt. momentum
31, 132
0.9, 0.999
Learning rate
le-4
2e-4
Batch size
32 (16 for Scannet++)
Weight decay
0.05
Learning rate schedule
Cosine decay
Layer-wise Ir decay (Clark al. 2020
0.75
Drop path (Huang et al. 2016)
0.1
Input resolution
224
Augmentation
RandomFlip + RandomCrop
Table 11: Semantic segmentation settings. Configuration used for fine-tuning the pre-trained models on the semantic segmentation task.
Object detection. For object detection, we evaluate pre-trained models by using the ViT-based pre-trained encoder the feature extractor in the detection framework. We use Cascade Mask- RCNN (He et al., 2017 Cai & Vasconcelos/ 2017) as our primary object detection model. Besides the feature extractor, the other learnable components including the detector's neck and head initialized from scratch. All training and evaluations are performed using the Detectron2 (Wu et al., 2019) framework. Exact training settings are provided in Tab. 12 We evaluate object detection in the test spaces from the ProcTHOR (Deitke et al. (2022) dataset as described in Section 4.1 For transfer, dataset of 20,000 images from external space, that is different from the test space. We
evaluate the transferred model on 5000 images from the test space.
Configuration
Small
Base
Fine-tuning epochs
150
Optimizer
Adam W (oshchilov Hutter 2019
Opt. momentum
11,
0.9, 0.999
Weight decay
0.1
Learning rate
0.0001 Multi-step decay [Epoch 133, Epoch 144]
Learning rate schedule
LA schedule milestones
Lr schedule decay values
[1.0, 0.1, 0.01] 0.01
Warmup epochs
Batch size
128
Layer-wise Ir decay (Clark et al. 2020 Drop path (Huang et al. 2016)
0.7
0.1
Input resolution
224
Augmentation
RandomFlip + RandomCrop
Table 12: Object detection settings. Configuration used for fine-tuning the pre-trained models on the object detection task.
Image Captioning. For image captioning, we evaluate the pre-trained models obtained from various methods including TST, 4M-21 (Bachmann al. 2024) (Internet), and also include randomly-
34

---

Published as a conference paper at ICLR 2026
hase dataser inages captored Indoor ser-ings thoweseing ny ferrrre comaon honsehata onjnots. 1 sant to cneate coco acyde conclaw and plotel captsons for tasau inapez. Finos qusarasee a single cuption for each Large, udiasing so collowing guidelion::
1. thatebal Coxtect bet Conclaate: The capcion shoald be abfectivs, descrihing the preainent abfecta and thair spatist calationatips within the zcene. capsion
globsi soone
prosinent aojects.
2. 1ruee of Ground-treth Classeate;
Alony vita each iaigh, greure-truth cleases and boinding box inforration are poovidel, hounding box inforeation is forezt ooper laft ccordinate, ipper
osordinate, visth, teight bounding inferaation balos,
selatiosatipa
3. *Sounding Besas and Class labela Visuaisted in Imagate;
The bornding by toses and clase tames are overt sid on the laage, stoutng eash deterned clsos for better tootissstion.
stipatial Racitionnng
Dusteibe objects" posiiiuts and apecial celecionuhipy visibie is the image and ground tzath information to selp locate thes sccurubaly. it mustiple dejects ure poesent is the itage (wr incicabed is groind truth intormationl, explicitly amttion sheir soun and opiaii tauiz positionsl recatiomiiia with other objaess is the ieage.
5. +*g iallicinatione
Each genarated concita captios suet agres sics the actual contente show in the peoctded haags, strictly avese adding information about abfects usiase you are carrain, only utilica the inforassion viatble in the image and the provides ground-treth clase Information.
: -illl provide bach the original Image and the image with over boves and labels, the both taaghs provide scounded globs and asyoe concioe capcion.
ceapsare*:;
soturn Python contstaing tonciie global captions. outgor any other
Ground fhrech isfomatiasi an clasee and bbox, information
Caupu Antosacieuu Image Jnnodated
Tmage
Figure 26: LLM Prompt instruction for ProcTHOR caption generation transfer task. We generate ground-trute truth captions by providing multimodal information to GPT-4o (OpenAl 2023) including annotated image, class and instance-wise bounding-box information. For each image, we generate 5 COCO-style captions.
initialized baselines (training from scratch). We adopt standard transformer-based encoder-decoder architecture for image captioning and employ cross-entropy loss for next-token prediction during training. Images are input to the encoder which serves as the context for the decoder network. The encoder network is initialized from the respective method's 's encoder while the decoder is initialized randomly. Training and hyperparameter details are listed in Tab. [3] We also train LLaVA style (Liu baseline. We first train the connector
et al. 2023) model that serves as a Large- laaee-lannnggemdellleeee module (MIP layer) using LLa'A's first-stage pretraining data consisting of 558K image-text pairs subset of the LAION-CC-SBU dataset (et al. 2021). For second-stage, we re-format our Proc THOR captioning dataset into instruction-tuniig format and jointly finetune both the connector and LLM.
Captioning data generation. To train models on the captioning task, we create a transfer dataset on a set of external spaces by generating captions using GPT-4o OOeeAA 2023) for the transfer dataset. We follow similar procedure for the evaluation set from the test space. We ensure the quality of generated captions by providing GPT-4o with multimodal inputs that include i) original RGB image ii) RGB image with instance-win wise detection boxes and class names overlayed iii) Class names and bounding box coordinates in text format. We design an input prompt that instructs GPT-4o to leverage the multimodal inputs and generate COCO-style (Lin et al., 2014) 5 concise captions with global context per image. For a sanity check, we randomly sampled 500 generated samples from the transfer set and found all captions to be consistent with the visual contents present in their respective images. The prompt message used for generating captions from GPT-4o is shown in Fig. 26
Y COMPUTATIONAL RESOURCES.
All model pre- training and adaptations were done on 64 H100 GPUs, with the base and small models taking approximately 12 hours and 7 hours to train, respectively. For the semantic segmentation transfer runs, we fine-tuned the models on 4 H100 GPUs, resulting in approximately 3 hours of training for the base model and 1.5 hours for the small model. For the detection task, we only fine-tuned the base model on 8 A100 GPUs, training for approximately 6 hours. Similar to detection, for captioning we only fine-tuned the base model training on 8 H100 GPUs for approximately hours.
35

---

Published as a conference paper at ICLR 2026
Configuration
ProcTHOR Captioning
Fine-tuning epochs
1400
Warmup epochs
600
Optimizer
AdamW (oshchilov & Hutter 2019
Opt. momentum
31, B2 0.9,
Base learning rate (Goyal et al. 2017)
Ic-5
Batch size
2048
Weight decay
0.05
Learning rate schedule
Cosine decay
EMA decay
SwiGLU (Shazeer 2020
Eval. freq (epochs)
50
Input resolution
224
Table 13: Training details: Image Captioning. Configuration used for transfer training for image captioning.
Input
Scratch
4M-21
DINOv2
TST-MM (Orss)
Ground Truth
Semantic Segmentation
Object Detection
Figure 27: Additional qualitative results. As demonstrated here TST performs better compared to the other models for all tasks.
36

---

Published as a conference paper at ICLR 2026
Replica
ProcTHOR
Semantic segmentation
Semantic segmentation
Object detection
Capttoning
Thuss
Scartch
CUP
Scattch
CUP
Dhand
Figure 28: Additional qualitative results. As demonstrated here TST performs better compared to the other models for all tasks.
37


Reviews:
[{"review_id": "AaF5axs4DR", "reviewer": "Reviewer_NjHj", "summary": "This paper proposes Test-Space Training (TST), a self-supervised framework that learns visual representations directly from multimodal data within the deployment environment, without using any external datasets. By leveraging cross-modal masked modeling among locally available sensors (e.g., RGB, depth), TST treats multimodality as supervision to pre-train models specialized for the target test space. Experiments across different datasets and various downstream tasks show that TST consistently matches or surpasses large-scale Internet-pretrained models (e.g., CLIP, DINOv2, 4M-21).", "questions": "1. The proposed TST framework mainly combines existing components with limited methodological detail. Could the authors clarify what the core technical novelty of TST.\n2. During inference, different modalities may introduce cross-modal bias or inconsistency. How does the proposed method effectively handle or align such discrepancies.\n3. In Figure 3, TST-MM still performs worse than large-scale Internet-pretrained models. Could the authors provide an analysis or discussion explaining the potential causes of this performance gap.\n4. In Figure 8, the meaning of different colors is not clearly explained. Could the authors specify what each color represents.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper explores an interesting and meaningful problem. It is clearly written and well-structured, making the methodology and insights easy to follow.", "weaknesses": "1. The paper raises an interesting problem but lacks methodological innovation — the proposed TST framework mainly integrates existing components, with less than one page describing the method in detail.\n2. Inference relies on multimodal inputs, yet potential modality bias or inconsistency is not addressed; the paper should clarify how multimodal information is effectively fused for downstream tasks.\n3. In Figure 3, TST-MM still performs noticeably worse than large-scale Internet-based pre-training, and the paper should analyze the reasons behind this gap.\n4. There are several formatting issues: for instance, the description of Table 1 should appear on the same page, and figure order needs adjustment (e.g., Figures 6–8 appear out of sequence).\n5. The meaning of different colors in Figure 8 is not explained and should be clarified.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2ThdXqlMWe", "reviewer": "Reviewer_ygUd", "summary": "The paper tackles an important problem, how to use data from a given deployment scenario effectively in the pre-train and post-train phase to maximally improve performance in that specific scenario we care about -- this would be the central challenge in deploying robotics applications in the real world. \n\nThe proposed method is straightforward, they use a self-supervised multi-modality objective to pre-train the model on the test space only. Additionally, they use features from web-scale models as additional self-supervisory alignment targets for improving performance, showing that direct access to web-scale data while training is unnecessary and methods like Attention Transfer are enough. The authors further show more analysis of the specialization-generalization tradeoffs, paving the way for personalization of models for each deployment site through their simple training framework.", "questions": "Please clarify the points made in weaknesses. I like this paper overall and would be open in increasing my rating further, however, the writing in a few places feels misleading to me and paints a picture that would border on over-claiming in my view (maybe the terminology needs to be refined in this case) -- please list down steps that would make the claims in the abstract and introduction much more tighter and easier to digest.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 1, "contribution": 4, "strengths": "In general, a well-executed paper with thoughtful experiments targeting a real problem.\n\n1. The idea of employing test-space data to personalize the model to a given deployment scenario is feasible and very useful in many real world situations.\n2. TST-Adaptation and TST-Sensory methods make sense to me, glad to know self-supervised cross modality helps as a self-supervised objective.\n2. Very useful ablations targeting the real deployment problems that would plague perception and robotics in the coming years. Namely, \"How many spaces is one test space worth?\" and the \"specialization-generalization tradeoffs\" are valuable signals and will become increasingly critical for robotics deployment in the coming years.", "weaknesses": "1. Parts of the paper over claims w.r.t the TST-MM method. \n\n(a) TST-MM is trained on additional pseudo modalities obtained by executing generalist models on the test space data. If we employ pseudo labelling from generalist models like CLIP, ImageBind and SAM, this is akin to distilling features (i.e. some spiritual variant of Attention Transfer [a]) of the generalist models, thus utilizing the pre-trained nature of the generalist model trained on internet data. This is framed as \"psuedo-modality\" instead of calling the spade what a spade is -- i.e. leveraging web-scale model's generalist pre-trained feature space.\n\n(b) The central claim that \"access to\" generalist data is unnecessary, while true, is riddled with caveats. The specific writing felt misleading as it seemed to imply that web-scale data/models are not needed and the caveat should be clarified at the outset in the abstract, teaser figure and introduction. As the paper itself shows, It is definitely necessary to have access to the generalist models (and indirectly their data and learned manifolds) to obtain the pseudo modalities for training. \n\n(c) The phrasing of the abstract and introduction seem to paint a picture that web scale datasets (and models) are not needed and generalist pre-training is not needed -- this is not true as it's indirectly used by this method. Please clarify what am I missing here, maybe the key claim should be rephrased to say \"one does not need to 'directly' access and train on web-scale data while deploying models to a specialized test space, feature alignment from such generalist models is enough\".\n\n2. TST-Sensory: How does it compare with TST-MM (i.e. with the web scale targets as modality sources)?\n\n3. When the generalist models (DINOv2, CLIP etc) are trained on the small external dataset, is the generalist backbone frozen? Many vision works have shown that -- surprisingly -- finetuning the DINOv2 backbone is counter-productive.\n\n4. No discussion of privacy concerns when using test space data. Is it possible to do this form of (pre/post-)training on-premise/on-device efficiently. For example, How do we ensure that data from deployment is minimally accessed by a robotics developer due to heightened privacy concerns of accessing such test space data. In the considered scenario, house hold robots would collect data of end-users engaging in day to day activities, I'm not sure how all the stakeholders would react to their private data being accessed over the internet, and what ethical, technical and sociological frameworks are necessary to address issues arising from the concerns of such stakeholders. While I don't expect this specific paper to solve this problem completely, a discussion or feasibility study keeping these concerns in mind would significantly strengthen this paper given it's applicability to real world deployment.\n\n[a] On the Surprising Effectiveness of Attention Transfer for Vision Transformers, NeurIPS 2024", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0CjrlBLtPd", "reviewer": "Reviewer_r7Ks", "summary": "This paper explores how rich multimodal data collected directly from the test environment can be used to pre-train visual representations in a self-supervised manner, without relying on any external data. To assess its effectiveness, the authors evaluate the proposed approach across multiple datasets and downstream tasks, showing that it can even outperform generalist models pre-trained on large-scale Internet datasets.", "questions": "Could the authors provide more explanation on why TST-MM performs less effectively on the captioning task?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper proposes training a specialized model using rich multimodal data collected from the test space without relying on external data sources, which is highly practical and closely reflects real-world scenarios. Moreover, they provide an insightful analysis of the trade-off between specialization and generalization, as well as the effectiveness of specialization, demonstrating its practical value.\n\n2. The viewpoint presented in lines 172–174 and fig. 4 is quite insightful. Nowadays, most research focuses on training large-scale generalist foundation models with massive datasets, but this paper highlights the often-overlooked importance of multimodality.\n\n3. The paper is clearly written and easy to follow.", "weaknesses": "1. The authors pre-train their model on the entire dataset but evaluate it on only a small portion of it. For instance, in the Replica dataset, the model is pre-trained on 84,889 samples but evaluated on only 5,000 images. Could the authors clarify the rationale behind this evaluation setting and discuss whether it might affect the reported results?\n\n2. In Table 2, the superscript “²” attached to Task-Specific Methods is referenced on page 8, but its explanatory note appears on page 7.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)