TEOChat: A Large Vision-Language Assistant for Temporal Earth Observation Data
Authors: Jeremy Andrew Irvin
Emily Ruoyu Liu
Joyce C. Chen
Ines Dormoy
Jinyoung Kim
Samar Khanna
Zhuo Zheng
Stefano Ermon
Venue: ICLR 2025 Poster (2025)
Keywords: vision-language model
large multimodal model
satellite imagery
earth observation
change detection
Primary area: foundation or frontier models, including LLMs
TL;DR: We develop a new language and vision assistant that can engage in conversation about sequences of temporal earth observation imagery.

Abstract: Large vision and language assistants have enabled new capabilities for interpreting natural images. These approaches have recently been adapted to earth observation data, but they are only able to handle single image inputs, limiting their use for many real-world tasks. In this work, we develop a new vision and language assistant called TEOChat that can engage in conversations about temporal sequences of earth observation data. To train TEOChat, we curate an instruction-following dataset composed of many single image and temporal tasks including building change and damage assessment, semantic change detection, and temporal scene classification. We show that TEOChat can perform a wide variety of spatial and temporal reasoning tasks, substantially outperforming previous vision and language assistants, and even achieving comparable or better performance than several specialist models trained to perform specific tasks. Furthermore, TEOChat achieves impressive zero-shot performance on a change detection and change question answering dataset, outperforms GPT-4o and Gemini 1.5 Pro on multiple temporal tasks, and exhibits stronger single image capabilities than a comparable single image instruction-following model on scene classification, visual question answering, and captioning. We publicly release our data, models, and code at https://github.com/ermongroup/TEOChat .

Published as a conference paper at ICLR 2025
TEOCHAT: A LARGE FOR TEMPORAL EARTH OBSERVATION DATA
ASSISTANT
Jeremy Andrew Irvin Emily Ruoyu Liu, Joyce Chuyi Chen, Ines Dormoy, Jinyoung Kim, Samar Khanna, Zhuo Zheng, Stefano Ermon Stanford University "Correspondence to: jirvinl 60cs stanford.ede
ABSTRACT
Large vision and language assistants have enabled new capabilities for interpreting natural images. These approaches have recently been adapted to earth observation data, but they are only able to handle single image inputs, limiting their use for many real-world tasks. In this work, we develop a new vision and language assis- tant called TEOChat that can engage in conversations about temporal sequences of earth observation data. To train TEOChat, we curate an instruction-followinng dataset composed of many single image and temporal tasks including building change and damage assessment, semantic change detection, and temporal scene classificatton. We show that TEOChat can perform wide variety of spatial and temporal reasoning tasks, substantially outperforming previous vision and lan- guage assistants, and even achieving comparable better performance than sev- eral specialist models trained to perform specific tasks. Furthermore, TEOChat achieves impressive zero-shot performance on a change detection and change question answering dataset, outperforms GPT-4o and Gemini 1.5 Pro multiple temporal tasks, and exhibits stronger single image capabilities than a comparable single image mnstruction-folloiin model on scene classification, visual question answering, and captioning. We publicly release our data, model, and code at https: //tgt//gtttttt/t com/ http//erooooooooeooeeeeoo TEOChat
INTRODUCTION
Many earth observation (EO) tasks require reasoning over time. For example, change detection is a widely studied task with the goal of identifying salient changes a region using EO images at different times (Chughtai et al. 202T). Automated methods to perform change detection can greatly aid humanitarian and sustainability efforts, including planning disaster relief (Khan et al. 2023),
tracking urban change (Reba & Seto 2020),, and monitoring deforestation (Mitchell et al. 2017)
crops (Karmakar et al., 2024). and ecological conditions (Besson et al. (022).. Prior automated methods to detect change in EO imagery have been specialist models, constraining their to single tasks they were explicitly trained to perform (Bai et al. 2023 Cheng et al. 2024).
Advancements the modeling of multimodal data have enabled generalist vision-language models (VLMs) that can perform a variety of natural image interpretation tasks specified through natu- ral language (Liu et al. 2023a). These models combine the capabilities of large language models and vision encoders, aligning visual and textual representations to enable natural, conversational interfaces for diverse tasks. This flexible interface presents major opportunity for developing mul- timodal conversational agents for a variety of real-world applications, such as emergency response, where rapid and adaptable information processing is crucial.
New VLMs have gained capabilities to model temporal sequences of natural images (video) TTang ctt al. 2023) and single BO images (Li et al., 2024c). However, no prior VLMs can model temporal EO data (left of Figure []. We investigate the performance of Video-LLaVA (Lin et al.] 2023). strong natural image VLM that can receive images and videos as input, and GeoChat (Kuckreja et al. 2023), , strong VLM fine-tuned on single EO image tasks (right of Figure I). We find Video- LLa VA generates inaccurate information, likely due to being trained on natural images and videos, whereas GeoChat only inputs single images and cannot process information across time.

---

Published as a conference paper at ICLR 2025
Model
Temporal FO
LLaVA (Lu et al. 2024a
X
LLaVA-1.5 Luu et
202.a
X
VideoChatGPT Maaz
2023
X
Video-LLaVA Lin et al. 2023
X
Chat-UniVi (Jin et al. 202.3
X
LLaMA-VID el
2024
X
Image
Tange
Video-L.aaMA (Zhang ci al.. 20223b
X
User
loune of the form (arda, yain, 2002, yeae) in your
ST-LLM (Liu et al. 2024b
X
RSGPT (uu
20261
area been trees sepetatton.
GeoChat (Kuckreja cl al. 2023
X
Video-LLaVA
Jeawingg lendscane toureiig
sudared onee are W.D.
SkyEyeGPT (Zhan cf al. 2024
X
X
GeoChat
LHRS-BOT (Muhtar al. 2024
X
VHM (Pang al. 20244
TEOChal (Ours)
TEOChat
Cectsally, Ennes buzitiigs,
Lhen saee boen controsed. the loce uue of thow will
178,
Figure I: Left: VLM Capabilities. TEOChat is the first VLM for temporal EO data. Right: Example outputs of prior VLMs. We compare to a temporal VLM (Video-LLat LLaVA) and a single EO image VLM (GeoChat).
We address these limitations by introducing TEOChat, the first VLM for temporal sequences of EO images. TEOChat adapts the Video-LLaVa VA architecture, designed originally for natural images and videos, to single and temporal sequences of EO images. To train TEOChat, we curate TEOChatlas, the first instruction-tuning dataset with instruction-folloiing examples for temporal EO data. We construct a variety tasks which require spatial and temporal reasoning capabilities using four EO datasets, namely fMoW (Christic et al. 2018). xBD (Gupta et al., 2019), S2Looking (Shen et al., 2021], and QFabric (Verma et al., 2021). TEOChatlas is large and diverse in task composition, sensors, and geography, making it strong dataset for training generalizable EO VLMs.
In summary, our contributions are:
1. We develop the first vision-language model for temporal EO data called TEOChat. When prompted with temporal images and natural language instructions, TEOChat can perform EO tasks that require temporal reasoning, e.g. change detection. TEOChat outperforms previous VLM for single EO images (GeoChat) and VLM for temporal sequences of natural images (Video-LLaVA), and also rivals specialist models on multiple tasks.
2. We introduce the first temporal EO dataset for multimodal instruction-tuning called TEOChatlas to train TEOChat. TEOChatlas contains 245,210 temporal EO examples spanning dozens of temporal instruction instluttion-ollloinng tasks as well as 308,86 single EO ex- amples from GeoChat Instruct, totalling 554.071 examples and 1,244.393 images.
3. We conduct a thorough set of experimental ablations to motivate TEOChat's design. We find the key design decisions include Video-LLaVi initialization, vision-language connec- tor fine-tuning, and the inclusion of image references between visual tokens in the prompt.
4. We demonstrate that (i) TEOChat achieves impressive zero-shot performance on an EO change detection and a change QA datasct, (ii) TEOChat outperforms two strong propri- etary foundation models for modeling sequences of images (GPT-4o and Gemini-1.5 Pro), and (iii) TEOChat possesses strong single image capabilities, outperforming GeoChat on multiple zero-shot scene classification and visual question answering (VQA) tasks.
2 DEVELOPING A VLM FOR TEMPORAL EO DATA
We aim to develop a VLM that can input natural language instruction and a temporal sequence of EO images, and output a natural language response. For example, given satellite images capturing an urban area before and after a hurricane, the model should respond to questions about damaged build- ings or flooded regions. The vast majority of natural image VLMs for this task use a LLa VA- -ike architecture (Liu et al. (024a) consisting of (i) a large language model (LLM; commonly LLaMA 2 (Touvron et al. 2023) initialized with Vicuna weights (Chiang et al. 2023)) (ii) a vision encoder (commonly a VIT (Dosovitskiy et al. 2020) pre-trdined with CLIP (Radford al. (021 to gen- erate visual representations, and (ii) a vision- -anounlanguage connector (commonly an MLP (Liu et al, 2023ab) align the visual representations with the language embeddings input to the L.M.M
2

---

Published as a conference paper at ICLR 2025
Training the VLM requires aligning language representations with visual concepts in EO images, but these concepts are not well- well-representee natural image data. EO imagery is taken from a birds- eye view and captures land use and land cover changes like flooding road construction. To align EO data with language and allow the model to respond to instructions about it, we require a dataset of instruction, EO image, and ground truth response triplets (an cestruction-oolloiing dataset). The responses are used to supervise the VLM using a next-token cross-entropy loss on the response to- kens (visual instruction tuning (Liu al., 2024al). ). We note that existing instruction-folloving EO datasets are insufficient for training temporal EO VLM, as the tasks do not require temporal ca- pabilities and therefore do not promote their development. We confirm this empirically by showing that GeoChat substantially underperforms our approach across all temporal reasoning tasks we test.
3 TEOCHATLAS: A TEMPORAL EO INSTRUCTION-FOLLOON
DATASET
The main obstacle to developing a temporal EO VLM is the absence of a suitable instruction- following dataset. To overcome this, we curate instraction-followe followwng dataset of temporal EO imagery called THOChatlas. A few key desired capabilities of the VLM induce important dataset design decisions. First, the model should follow instructions about a wide variety of tasks requiring spatial and temporal reasoning capabilities, so we construct diverse set of cntrructionfollloviinn tasks (Section B.I]. Second, the model must reason over variable-length temporal sequences of EO data from different sensors, so we ensure many sensors and sequence lengths are represented (Sec- tion 3.2). Third, the model should be able reference specific images in its input and output. and support usee-friendly conversation, we design the prompts support these features (Section 3.3
3.1 TEOCHATLAS: TASK COMPOSITION
We curate instruction-following tasks in TEOChatlas to develop a range of model capabilities. Im- portantly, because our goal is to add temporal capabilities without compromising single image pabilities, we include both single image and temporal tasks in the dataset, described below.
3.1.1 SINGLE IMAGE TASKS
To support the development of single image capabilities like object recognition and spatial reason- ing, we include many single image instruction foolloiinnnfloowingg tasks in TEOChatlas. Specifically, we use GeoChat Instruct (Kuckreja et al. 2023). a composition of single image cdaructtio-followiin tasks including scene classification, visual question answering, referring expression, region caption- ing, detailed description, grounded description, and multi-turn conversation. ScoCChat_Intrue was used to train the GeoChat model, which exhibits strong performance on these single image tasks. Appendix B has more detail about GeoChat Instruct and carruction-follownng tasks.
3.1.2 TEMPORAL TASKS
We construct temporal instruction-folloiing tasks using four EO datasets. We group tasks into seven categories spanning two common real-world applications. namely disaster response and urban development monitoring. Figure 2 shows example of tasks and Table x enumerates all of the tasks and prompts. TEOChatlas is the first citrrcctionfollowinng EO dataset with temporal tasks (Table
Temporal Scene Classification (TSC) We construct an instruction following task from a standard EO task called temporal scene classification, where the goal is to classify a sequence of satellite images into a set of predefined categories. For this task, we use the Functional Map of the World (fMoW) dataset (Christie et al.) 20188), consisting of satellite image time series classified as one of 62 categories. To convert fMoW to an instructioo-followeee instructs the model to classify the image by selecting of the 62 classes provided in the prompt.
task, we use a standard prompt which
Change Detection (CD) We curate instruction followiig tasks for change detection, a widely stud- ied earth vision task to identify changes in images of area over time. We include two canonical change detection tasks, namely building damage assessment and building change detection.
For building damage assessment,
xBD (Gupta et al. 2019), building damage detection dataset consisting of bitemporal pre- and post-disaster images, where every building is localized and classified into four damage severity categories. Following standard practice on this dataset,
3

---

Published as a conference paper at ICLR 2025
Temporal Seene Classification
Change Detection
Bu lingg Looal eifice and Derrage Chassifcation
Stemass bakking Change Detection
Iratrucioon
Ware ane the tallecin
Intruction
Label
Sumege [ibbrl Bestroved
Label
(boos mere convir siced
Spatial Change Referning Expression
Change Answering
Ragion-based Change Question Answering
Instrutiie
Tieectiy anverdle denaget
Instuction
Butt foaetrr reeered haver
Intrectie
reggan
Label
[lowa), [kowe), (kaaa), (maes)
Label
Giodd
Label
caneercal..
Temporal Referring Expression
Region-besed besed Temporal Quection Answering
Instruction
In wisin stages aettractiio Foidbed in Dux region B4aejts
Instruciion
#hat arban dtelument
taspeeed [iboa] batnees 27age
20age
Label
Image
Label
constraccion Firsshes
Figure 2: Examples of instruction- instruction-folloiing tasks in TEOChatlas. We curate many instruction-folloot following tasks for temporal EO data and group them into categories. The tasks require spatial and temporal reason- ing capabilities, span real-world applications including disaster relief and urban development monitoring.
we divide this task into two subtasks, namely building localization (Loc.) and building damage classification (Dmg Cls.). Building localization prompts the model to predict bounding boxes around all buildings the area, representing each box as a sequence of natural language tokens of the form
[mmin Vmin maxx ymax] following Kuckreja al. (0233. Building damage classification requires the model to classify each bounding box input in the instruction as one of the four damage categories provided in the prompt. These tasks together form canonical semantic change detection task.
For building change detection (Det.), we use S2Looking (Shen et al. 2021), a building change detec- tion dataset consisting of bitemporal, side-looking satellite images labeled with polygons indicating that a building has been constructed or demolished. We follow standard practice and task the model with outputting all changed buildings in the images. We the type of instruction for the building localization task. This is canonical binary change detection task.
used
Spatial Change Referring Expression (SRE) To facilitate better spatial reasoning capabilities, create instruction-folloiing tasks to identify the locations of specific changes which are referred create tasks from xBD where the model is
by natural language expressions. Specifically, instructed to identify the locations of only the buildings which experienced a specific damage type. We also construct tasks from S2Looking where the model is required detect only the constructed buildings or only the demolished buildings. These temporal tasks are analogous to the referring expressions for single images used in the GeoChat dataset (Kuckreja et al. 2023).
Change Question Answering (QA) To develop the model's ability to recognize change and reply to a diverse array of user questions, we construct multiple visual question answering tasks about changes between temporal images. From xBD, create tasks such as instructing the model to output the type of disaster from the provided labels in the dataset, the location in the region (a cell of 3 x grid specified with top left, top center, etc.) most affected by the disaster, indicating if there are any destroyed buildings in the area. Similarly, we add tasks from S2Looking to identify whether any buildings have been constructed demolished in the region.
binary label

---

Published as a conference paper at ICLR 2025
Region-based Change Question Answering (RQA) We create question answering tasks based on regions to improve reasoning about spatial inputs. We generate the same change question answering tasks from xBD and S2Looking, but constrain the question to bounding box imput representing specific area within the image. We also create tasks using QFabric (Verma et al. (021), a multi-task urban change detection dataset consisting of five image sequences each labeled with polygons repre- senting urban change. Each polygon has a sequence-level label indicating the broad type of change (change type classification) and an image-level label indicating the specific change that occurred between adjacent timesteps (change status classification). We construct eestructionfoolloinng tasks by mirroring building damage classification, instructing the model classify the change type or change status of an input bounding box into categories included in the prompt.
Temporal Referring Expression (TRE) To enable the model to temporally localize change, we in- clude tasks instructing the model to identify the specific image(s) in which given change occurred. We use the change type labels from QFabric which indicate the type of urban development that oc- curred in each image. Because these labels are tied to specific regions of the images, we include a bounding box in the prompt and task the model with identifying the image(s) where the change occurred. To perform this task, the model has model spatial inputs and generate temporal outputs.
Region-based Temporal Question Answering (RTQA) In order to support the development of sophisticated spatial and temporal reasoning capabilities together, the final tasks we curate for TEOChatlas instruct the model to answer questions about a given region, inputting or outputting references to images in the sequence. Specifically, we create questions about the presence of any urban development across two images within a box specified in the prompt and we generate tasks instructing the model to identify all images in the sequence where given change type occurred.
3.2 TEOCHATLAS: EO IMAGE SEQUENCES
To allow the model to reason about varying input sequence lengths and sensor types, we ensure that TEOChatlas includes diverse set of image sequence sizes and image providers. For sequence lengths, the temporal EO datasets we include consist of bitemporal (xBD and S2Looking), pen- tatemporal (QFabric), and multitemporal (variable length) sequences (fMoW). Due to memory straints, we limit the sequences to maximum of 8 images, and randomly sample 8 images without replacement from longer sequences. To increase the representation of other sequence lengths in the dataset, we also randomly sample shorter sequences from QFabric. For sensor types, the EO datasets in TEOChatlas span 8 sensors (Table 77.. To increase the representation of Sentinel-2, publicly available sensor with global coverage that is commonly used for EO tasks, we include se- quences from -MoW-Sentinel (Cong et al., 2022). More detail about how we process the images in Appendix B and we also note that TEOChatlas has considerable geographic diversity (Figure 5
3.3 TEOCHATLAS: PROMPT DESIGN
We design the prompts enable the model reference images and converse in a user-friendly man- ner. As the TRE and RTQA tasks require the model to input and output references to specific images in the sequence, we design the prompts to facilitate this. To do so, we interleave image references between the visual tokens input to the LLM (see Figure 3). This differs from prior methods for temporal sequences of natural images which simply concatenate the representations (Maaz et al.) 2023
Lin et al., 2023 Zhang et al., 2023b Jin et al. 2023 Li et al. 2023, Touvron et al. 2023).
GeoChat Instruct depends on task tokens to help the model differentiate between tasks, for example
to determine whether to include boxes in its response. We remove this requirement, and replace the tokens with short natural language specifications of the task. Examples are provided in Table
Finally, we modify the system prompt to (1) specify that the input contains sequence of satellite images and (2) randomly include the image resolution (high or low) and sensor name in the prompt to allow TEOChat to leverage that information when provided (see Appendix).
4 TEOCHAT
4.1 TEOCHAT ARCHITECTURE
We adopt a LLaVA-1.5-like architecture (Liu et al. 2023a) consisting of (i) a cemporall--share image encoder (CLIP VIT-L/14 (Radford et al. 2021 obtain representations of each image in
5

---

Published as a conference paper at ICLR 2025
tonstruction
piteo]
daappcor
Image
The last nos constructson
Large Language Model
[T]
T
M
T IT.
M
V
This
outell swegro
Lazer Instraction]
Whics class does this
T
LORA Fine-Tuning
Shared Vision-Language Connector
zdentsfy Jocation
constructton
Shared Vision Encoder
grase riele the atatt
Full Fine- Tuning
What disascer
tas cccurred here?
Ioentaly the type urban
Weights Frozen
Sten cte stadium
leppored betweer and
Figure 3: Overview of TEOChat. TEOChat inputs a temporal sequence of EO images and a user instruction, and outputs natural language response. It can input and output regions specified through bounding boxes, and also input and output references to specific timesteps using image identifiers. We show one example from each task category, including temporal scene classification (peach)), ), change detection (yellow), ) spatial change referring expression (green), (, change question answering (cyan), region-based change question answer- ing (blue), temporal referring expression (purple), and region-batt based temporal question answering (pink).
the sequence, (ii) 2-layer MIP to project the visual representations to the input of the LLM, and (iii) an LLM decoder (Llama (Touvron et al., 2023)) which inputs the instruction and temporal sequence of projected visual representations to generate the response (Figure 3 A recent extension of LLaVA-1.5 to handle videos called Madeo-LLaVa. (Lin et al. 2023) uses joint natural image-video instruction-tune by using the frozen CLIP VIT image encoder for images and a frozen Language- Bind (CLIP-aligned) video encoder for videos (Zhu et al.] (023a)), while keeping the rest of the architecture the same as VL. We test whether these projector and LLM weights learned on video data are better for temporal EO data by comparing Video-LLaV initialization to LLaVA initialization, as well freezing vs. fine-tuning the projector, our experimental ablations.
Another key architectural decision is whether to use the image encoder or video encoder from Video- LLaVA (Lin et al., 2023) generate visual representations for the sequence. We choose use the image encoder for two reasons: (1) prior change detection approaches have demonstrated that are highly effective for iden- tifying changes in sequences of EO data (Zheng et al.] 2021a (022a),, and (2) the video encoder was designed input fixed sequences of images. While is possible to remove this second constraint, observed performance degradation when using shorter sequences in preliminary periments, likely because the vidco encoder was trained strictly on image sequences (Zhu et al.) 2023ab. Replicating images in the sequence may address this but makes training and inference much slower. Altogether, our vision encoder fast, memory efficient, and achieves strong performance.
Siamese encoders with weights shared across time
4.2 TEOCHAT TRAINING DETAILS
Following prior work (Kuckreja et al. 2023 to retain the strong capabilities of the pre- mr-ttriiee image encoder and LLM, and minimize memory usage during training, we freeze the vision encoder then fine-tune the LLM using Low-Rank Adaptation (LoRA) A) (Hu et al. 2021). To further reduce memory footprint, we use 8-bit quantization of the base LLM weights. These strategies allow us train the large multimodal model on temporal sequences of up to 8 images using an NVIDIA A4000 GPU (16GB VRAM). We provide additional training details in Appendix Section
EXPERIMENTAL RESULTS
We evaluate TEOChat in a variety of settings. First, we find TEOChat demonstrates impressive tem- poral reasoning capabilities, outperforming two strong VLMs, one for videos (Video-LLaVA) and one for single EO images (GeoChat), and even rivals or outperforms several specialist approaches (Section 5.1 Second, through experimental ablations show TEOChat's design is superior to

---

Published as a conference paper at ICLR 2025
Task
Dataset/Subtask
Specialist
Video-LLaVA
GcoChat
TEOChat-T
TEOChat
IMoW RGB (Acc.)
65.9
16.6
59.2
73.5
75.1
TSC
IMoW Sentinel (Acc.)
4.9
26.3
45.5
45.5
xBD Loc. (Fi)
85.9
7.0
7.4
32.9
38.9
CD
xBD Dmg Cls. (Fi)
26.5
7.0
11.8
49
50.0
$21 Lookin Det. (F))
26.5
8.0
7.7
30.6
34.5
xBD (Fi)
3.7
8.7
18.3
25.1
SRE
S2Looking (Fi)
1.7
4.5
31.0
32,9
xBD (Acc.) S2Looking (Acc.)
30.8
34.0
89.8
89.9
QA
41.5
51.5
75.4
73.4
xBD (Acc.)
57.3
63.6
93.7
94.0
S2Looking (Acc.)
47.8
86.7
90.0
RQA
QFubric (2 images) (Fr)
77.0
22.6
63.9
66.7
QFabric images] (F))
81.6
26.5
72.8
74.3
TRE
QFabric (Acc.)
1.9
73.1
74.9
QFabric (Acc.)
26.5
71.4
71.7
RTQA
QFubric [5 images] (F:)
54.5
10.5
12.5
65.2
66.4
Table 1: Temporal task performance of TEOChat compared to other VLMSs and specialists. We bold the best generalist result on each task. TEOChat outperforms VLM instruction-tune on natural images and videos (Video-LLaVA) as well as a VLM instruction-ture on single EO images (GeoChat). We note Video- LLaVA and GeoChat are not trained on these tasks. TEOChat also performs comparably to or outperforms several specialist models trained to perform specific tasks. TEOChat-T T was only trained on temporal tasks.
many others (Section 5.2g Third, TEOChat achieves impressive zero-shot generalization new temporal datasets not in TEOChatlas (Section 5.3). Fourth, we find that TEOChat outperforms two strong proprictary VLMs trained on sequences of images (Section (5.4). Fifth, show TEOChat possesses strong single image capabilities, outperforming GeoChat on average across zero-shot scene classification, VQA, and captioning (Section (.5). Sixth, we find TEOChat exhibits inter- esting instances of generalization to tasks not explicitly represented in the training set (Section 5.6
5.1 COMPARISON TO OTHER VLMS AND SPECIALIST APPROACHES
Baselines and Evaluation Metrics We compare TEOChat Video-LLaVA (Lnn al. 2023 GeoChat (Kuckreja et al., 2023), and specialist models. To obtain GeoChat's predictions on the temporal tasks, we perform temporal post-processing of its single image outputs (Appendix Sec- tion G For the specialists, we use strong self-supervised EO model adapted to fMoW (SatMAE (Cong et al. )022)) and strong models trained on each individual dataset for the change detection tasks (a modilied UNet (Ronneberger et al., 2015) from Gupta et al. (2019). FC-Siam-Dint (Daudt ct al. 2018) from Shen et (2021), and a modified UNet from Verma et al. 2021) ). We evaluate using spse-ipec metries described in detail in Appendix Section
Temporal Scene Classification TEOChat achieves impressive performance on TSC with the fMoW RGB (75.1%) and Sentinel (45.5%) validation sets, outperforming both Video-LLaVA (16.6%, 4.9%) and GeoChat (59.2%, 26.3%) (Table []. We also compare TEOChat to linear probing a specialist encoder that was pre-trained with self-supervision
IMoW images (SaMMAE). TEOChat obtains considerably higher accuracy than SatMAE on fMoW RGB (+9.2%), and emphasize the TEOChat is generalist which is pre-trained on EO imagery. We also note that, impres- sively, TEOChat performs comparably to SatMAE+Stack (-0.8%) and is approaching state-of-the. art (SOTA) specialist performance on fMoW RGB (-4.1%) (Khanna et al., 2024a).
Change Detection and Spatial Referring Expression TEOChat outperforms Video-LLaVA and GeoChat by a substantial margin on all CD tasks (Table M). However, on xBD localization, it un- derperforms the specialist by large margin. TEOChat's CD performance on S2Looking is higher than the specialist model's performance (+8.0 F,). However, we note that SOTA performance on S2Looking CD (Li et al. 2024b) and xBD localization Chen al. (002a)) is much higher than TEOChat (+31.3 F1, +48.5 which we believe is largely because these bitemporal change detec- tion models output segmentation masks, whereas TEOChat is constrained output boxes. Qualita- tively, TEOChat does moderately well at identifying building locations but struggles to precisely lo- calize boundaries and localize large numbers of buildings. This may be partially due representing

---

Published as a conference paper at ICLR 2025
M/rjjccee
Initialization
Image Encoder Initialization
Projector
Image References
Fine-tuned on TEOChataas
Train Ieers
fMoW RGB
xHD Dong Cls.
S21 noking Det.
QFabric RQA-2
Q
RQA-5
Vadoo-LLaVA
CLIP
Frozen
7k
16.6
8.3
6.2
22.6
Video-LLaVA
CLIP
Frozen
7k
63.2
22.2
9.9
59.0
65.7
LLaVA
CLIP
Frozen
7k
66.8
8.9
4.8
58.7
64.6
Video-LLaVA
SkySeript
Finzen
3.7
29.9
Vidco-LLVVA
SkySeript
Fine-tuned
7k
68.3
40.8
12.5
59.0
66.6
Video-LLaVA
CLIP
Frozen
7k
69.2
40.6
18.1
61.4
67.9
Vidoo-LLVA
CLJP
Fine-tuned
7k
71.7
47.2
61.7
Voko-LLaVA
CLIP
Frozen
14K
71.5
45.8
30.4
66.6
74.3
Vidoo-LLVA
CLIP
Fine tuned
14k
75.1
50.0
33.6
66.7
74.3
Table 2: Model ablations testing the impact of different design decisions in TEOChat, Our design (bottom row) outperforms all other tested designs on the five canonical temporal tasks.
object locations using natural language tokens, and partially due to using axis-aligned boxes rather than general polygons. On damage classification, TEOChat outperforms the specialist substantially (+23.5 Ft). On SRE, TEOChat achieves much higher performance than Video-LLaVA (+21.4 4 F1 on xBD, +31.2 F1 on S2Looking) and GcoChat (+16.4 F1 on xBD, +28.4 F1 on S2Looking).
Change Question Answering and Region-Based Change Question Answering On QA and RQA, TEOChat achieves much higher performance than the baseline VLMs on all tasks [ On QA, TEOChat achieves 89.9% and 73.4% accuracy on xBD and S2Looking respectively, compared to GeoChat which attains 34.0% and 51.5% the respective datasets. Similarly, on RQA. TEOChat outperforms GeoChat by 30.4% accuracy on xBD and 22.8% on S2Looking, and drastically out- performs both QFabric tasks. TEOChat underperforms both specialist models on QFabric but achieves a much more similar F1 to the specialists compared to both GeoChat and Video-LLaVA.
Temporal Referring Expression and Region-baseds Temporal Question Answering We find TEOChat obtains much higher accuracy than Video-LLaVA on TRE (74.9% vs. 1.9%) and RTQA (71.7% vs. 26.5%). On 5 image RTQA in QFabrie, TEOChat outperforms the specialist by 11.9 F1.
5.2 MODEL ABLATIONS
Temporal vs. joint single-temporal training. We investigate the effect of joint single image and temporal training by training model on only the temporal tasks (TEOChat-T). TEOChat-T under- performs TEOChat on all but two tasks, matching it on fMoW Sentinel and slightly outperforming it on S2Looking QA (Table I]. This suggests joint training also largely benefits temporal capabilities.
Model design We test various design decisions of TEOChat. Specifically, we measure the impact of using LLaVA initialization instead of Video-LLaVA, using an image encoder initialization from strong remote sensing pre-trained weights (SkyScript (Wang et al., 2024d)) instead of CLIP, fine- tuning versus freezing the projector layer, and including image references in the prompt. We note that we couple the LLM and projector initialization they were pretrained together. Furthermore, to fairly compare to a new image encoder initialization, we also test the impact of using a fine-tuned projector with the alternate initialization to allow it to adapt to the new image encoder outputs. Finally, we measure the performance improvement from training on TEOChatlas (without using image references) and the effect of training for longer (7k iters [1 epoch) vs 14k [2 epochs)).
Our design of TEOChat outperforms all tested designs on the five canonical temporal tasks (Ta- ble D. Fine-tuning on TEOChatlas leads to large performance improvements across the tasks and including image references leads to further improvements. Interestingly, image references help even in the bitemporal case, which is likely because the 'Image 2' reference helps delineate the visual tokens between the two images, as it explains most of the improvement from including references on the canonical bitemporal tasks (Table 9.. The benefits of image references hold at test-time as well, as their exclusion from the test-time prompt leads to substantial performance drops on both tasks (Table 10p. Using LLaVA weights for the LLM and projector initialization underperforms VideoL VA across all tasks which we attribute to the fact that Video-LLLaV is stronger model and was trained with natural video data. The use of remote sensing pre-trained image encoder also underperforms a CLIP-pretraing is partially because the LLM weights were pre-traine pre-trained with a frozen CLIP-initializer image encoder during multi modill instruction sittucttiottuiing -tuiigg natural video. We also find that fine-tuning outperforms freezing the projector, and training the model for longer leads to further performance improvements on all tasks, with especially large improvements fMoW RGB and xBD damage classification.
image encoder even with projector fine-tuning, which we believe
8

---

Published as a conference paper at ICLR 2025
Model
ABCD
CDVQA
Model
xBD Dan Cls.
S2Looking Det.
Specialist
95.2
63.4
Intern VL2 CChen et al. Qwen2VL Wang et
2024b
11.8
7.8
Video-LLaVA
50.0
29.8
2024b
17.3
9.9
Gemmi Pm
35.8
16.5
GeoChat
GPT-4o
38,3
21.5
TEOChat
85.6
47.2
TEOChat
50.0
33.6
Table Zero-shot ABCD (building CD) and CDVQA (change QA) results.
Table Comparison other multi-image VLMss. GPT-4o
and Gemini 1.5 Pro use three in-context examples.
LRBEN
HRBEN
Model
AID
UCMereed
Average
Pres.
Comp.
Rural/Urbun
Pres.
Comp.
Vidco-LLaVA (Lin all.22000 2023
52.4
46.5
55.8
65.1
61.0
64.5
66.8
58.5
RSVQA (Lobry
87.5
81.5
88.2
RSGPT (uu 2025
91.2
91.7
94.0
90.9
90.0
LHRS-Bol (Muhtar el
2024
91.3
88.5
90.0
89.1
92.6
92.5
VHM (Pang al. 2024
91.7
91.1
92.0
95.0
64.0
83,5
GeoChal (Kuckreaa et
2023
72.0
84.4
91.1
90.3
58.5
83.2
79.9
TEOChat (Ours)
91.7
92.7
81.1
83.4
Table Single image accuracies for scene classification (AID, UCMerced) and VQA (LRBEN, HRBEN). italicize results from fine-tuning on the corresponding training set. other results are zero-shot. We bold the highest zero-shot score, except LRBEN which all models but Video-LLaVA are fine-tuned on.
5.3 ZERO-SHOT PERFORMANCE ON CHANGE DETECTION TASKS
We assess TEOChat two new temporal remote sensing datasets, namely change detection using ABCD (Fujita et al. 2017) and change visual question answering using CDVQA Yuaan et al. (022) (Table 3). TEOChat demonstrates impressive generalization these datasets, outperforming Video- LLaVA (+35.6 on ABCD, +17.4 on CDVQA) and approaching specialist performance.
5.4
COMPARISON TO OTHER MULTI-IMAGE VLMS
We compare TEOChat to open source multi-image VLMs Intern (Chen et al. 2024b) and Qwen2VL-7B-Instre (Wang et al.) 2024b) as well GPT-4o and Gemini 1.5 Pro with three demonstrating examples on xBD damage classification and S2Looking change detection (Table|4) We focus this analysis on these datasets as we could not evaluate the proprietary models on fMoW or QFabric (larger both in size and sequence length) due to cost. TEOChat outperforms both Qwen2- VL and VL2 by a large margin on xBD damage classification (+38.2 Fl, +32.7 Fl respec- tively) and S2Looking change detection (+25.8 F1, +23.7 Fl respectively). TEOChat also performs better than both proprietary models on xBD damage classification (+11.7 F1 compared to GPT- 4o, +12.8 Fl compared to Gemini 1.5 Pro) and S2Looking change detection (+14.2 F1 compared to GPT-40, + +17.1 F1 compared to Gemini 1.5 Pro). Qualitatively, the proprietary models tend to under- predict change in S2Looking and underestimate building damage in xBI,, whereas the open-source models produce highly inaccurate boxes and classify almost all examples as 'No Damage'.
5.5 SINGLE IMAGE PERFORMANCE
We measure TEOChat's single EO image capabilities by evaluating it on (1) scene classification tasks (AID (Xia al. 2017) and UCMerced (Yang & Newsam, 2010)) and (2) visual question answering tasks (LRBEN and HRBEN (Lobry et al., 20200). We note that evaluation on AID, UCMereed, and HRBEN zero-shot, as they are not TEOChat's training set.
TЕOChat matches exceeds the performance of all single image models on six of the seven tasks (Table 5). It outperforms GeoChat on both LULC classification tasks, notably by large margin on AID (+8.9), much larger-scale dataset than UCMerced. It underperforms LHRS-Bot and VHM on AID, likely due to the inclusion of several more LULC classification datasets their training data. TEOChat also outperforms all models on low resolution VQA presence and comparison tasks, and attains close to the best performance on rural/urban classification. It achieves the highest zero-shot performance high resolution (HR) presence, while slightly underperforming GeoChat and VHM on HR comparison. Notably, GeoChat and VHM use higher resolution images than TEOChat (504 and 336 vs. 224). TEOChat attains higher average performance than GeoChat by considerable

---

Published as a conference paper at ICLR 2025
bappoed aips
Teage
Teage ?
this
ThI objectt
zage??
ic siate
azert new expore wired leug??
vocar taay ween all strpher Sertnn strpleee
Deaeccyy the Sonetoon the siilier sirstere.
Image
Image
ieroraaioa enstreneen i bulll to relee decelaac
with lmage cap eurting
koy durang 1em The Saace rastured of 103 1er2ne
tree: greet
Figure 4: Qualitative examples of task generalization exhibited by TEOChat on temporal EO images.
margin (+3.5), suggesting it has stronger single image capabilities on top of the new temporal ones. Finally, we measure TEOChat's performance on three EO caption datasets. We find it outperforms GeoChat on all datasets and matches other single image EO generalists on the larger two datasets in the fine-tuning setting (Tables
and matches or outperforms GeoChat on average in the zero-shot setting (Tables 15 to 18 Figures and g show example TEOChat single image responses.
5.6 TEOCHAT GENERALIZATION
TEOChat also exhibits interesting cases of generalization to tasks not explicitly represented in the training data (Figure For example, TEOChat can compose its spatial and temporal reasoning abilitics, such as identifying objects that are only regionally annotated in the single image training tasks (like airplanes and cars) within different images in the temporal sequence. It can also recognize differences between images (like seasonality) which not explicitly reflected the training data.
6 CONCLUSION
In this work, we propose TEOChat, the first vision and language assistant that can engage in con- versation about temporal EO imagery. We introduce novel dataset called TEOChatlas to train TEOChat on a diverse set of spatial and temporal instructionnfollooinng tasks. TEOChat exhibits strong performance, outperforming previous VLMs substantially, and even competing with or out- performing several specialist models. TEOChat also demonstrates impressive zero-shot generaliza- tion to two change detection datasets and outperforms strong multi-image proprietary models.
There are several interesting directions for future work. First, it would be interesting to explore other architectures to process the visual data across time. The LLM is responsible for performing the temporal information aggregation in TEOChat, but it may be helpful to use a dedicated part of the architecture for this. Second, it would be useful to enable the model to process multi-spectral bands which is common in EO data. Third, exploring methods to improve object localization is worthwhile, including regressing the coordinates instead of representing them with natural language tokens (Zhang et al. 2023a), using oriented bounding boxes to allow for tighter localization, or increasing memory efficiency of the architecture enable inputting higher resolution images.
10

---

Published as a conference paper at ICLR 2025
7 REPRODUCIBILITY STATEMENT
Training data, model weights, training code, and evaluation code are hosted at https:// github. com/ erwongroup TEECChat All major experimental results can be reproduced by following the steps described there.
REFERENCES
Kumar Ayush, Burak Uzkent, Chenlin Meng, Kumar Tanmay, Marshall Burke, David Lobell, and Stefano mon, Goography-aware self-supervised learning. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pp. 10181-10190, 2021.
Ting Bai, Le Wang, Dameng Yin, Kaimin Sun, Yepei Chen, Wenzhuo Li, and Deren Li. Deep learning for change detection in remote sensing: a review. Geo-spatial Information Science, 26(3):262-288, 2023.
Yifang Ban, Puzhao Zhang, Andrea Nascetti, Alexandre R Bevington, and Michael A Wulder. Near real-time wildfire progression monitoring with sentinel-1 sar time series and deep learning. Scientific reports, 10(1): 1322, 2020.
Mare Besson, Jamie Alison, Kim Bjerge, Thomas E Gorochowski, Toke Høye, Tommaso Jucker, Hjalte MR Mann, and Christopher Clements. Towards fully automated monitoring of ecological communities. Ecology Letters, 25(12))2753-2775 2022.
Hongruixuan Chen, Jian Song, Chengxi Han, Junshi Xia, and Naoto Yokoya. Changemamba: Remote sensing change detection with tempora state space model. arXiv preprint (xXix:2444.03225, 2024a.
Jun Chen, Deyao Zhu, Xiaogian Shen, Xiang Li, Zechun Liu, Pengchuan Zhang, Raghuraman Krishnamoorthi, Vikas Chandra, Yunyang Xiong, and Mohamed Elhosseiyy. Minigpt-v2: large language model as a unified interface for vision-language multi-task learning, arXiv preprint arXi:23100009478, 2023a.
Keqin Chen, Zhao Zhang. Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. Shikra: Unleashing multi- modal Ilm's referential dialogue magic. arXiv preprint 2.Xi:2306.11195 2023b.
Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong. Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. Interv12:: Better than the best expanding performance boundaries of open- source multimodal models with the progressive scaling strategy, 2024b.
Gong Cheng, Junwei Han, and Xiaoqiang Lu. Remote sensing image scene classification: Benchmark and state of the art. Proceedings of the IEEE, 105110)1865-1888 2017.
Guangliang Cheng. Yunmeng Huang, Xiangtai Li, Shuchang Lyu, Zhaoyang Xu, Hongbo Zhao, Qi Zhao, and Shiming Xiang. Change detection methods for remote sensing in the last decade: A comprehensive review. Remote Sensing, 16(13):2355, 2024.
Wei-Lin Chiang, Zhuohan Li. Zi Lin, Ying Sheng. Zhanghao Wu, Hao Zhang. Lianmin Zheng, Siyuan Zhuang. Yonghao Zhuang, Joseph E Gonzalez, al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See https://vicuna. Imsys. org (accessed 14 April 2023), 2(3):6, 2023.
Gordon Christie, Neil Fendley, James Wilson, and Ryan Mukherjee. Functional map of the world. In Proceed- ings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6172-6180, 2018.
Ali Hassan Chughtai, Habibullah Abbasi, and Ismail Rakip Karas. A review change detection method and accuracy assessment for land use land cover. Remote Sensing Applications: Society and Environment, 100482, 2021.
Yezhen Cong, Samar Khanna, Chenlin Meng, Patrick Liu, Erik Rozi, Yutong He, Marshall Burke, David Lobell, and Stefano Ermon. Satmae: Pre-training transformers for temporal and multi-spectral satellite imagery. Advances in Neural Information Processing Systems, 35:197-211, 2022.
Rodrigo Caye Daudt, Bertr Le Saux, and Alexandre Boulch. Fully convolutional siamese networks for change detection. 2018 25th IEEE international conference on image processing (ICIP), pp. 2018.
IEEE,
Rodrigo Caye Daudt, Bertrand Le Saux, Alexandre Boulch, and Yann Gousseau. Multitask learning for large- scale semantic change detection. Computer Vision and Image Understanding, 187:102783, 2019.
11

---

Published as a conference paper at ICLR 2025
Pablo Pozzobon De Bem, Osmar Abílio de Carvalho Junior, Renato Fontes Guimarães, and Roberto Amaldo Trancoso Gomes. Change detection of deforestation the brazilian amazon using landsat data and convo- lutional neural networks. Remote Sensing, 12(6):901, 2020.
Mathieu Decuyper, Roherto O Chávez, Madelon Lohbeck, José Lastra, Nandika Tsendbazar, Julia Hackländer, Martin Herold, and Tor-G Vågen. Continuous monitoring of forest change dynamics with satellite time series. Remote Sensing of Environment,
2022.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Un- terthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words:
Transformers for image recognition at scale. arXiv preprint arXiv:2010. /929, 2020.
Aito Fujita, Ken Sakurada, Tomoyuki Imaizumi, Riho Ito, Shuhei Hikosaka, and Ryosuke Nakamura. Dam- age detection from aerial images via convolutional neural networks. In 2017 Fifteenth IAPR international conference on machine vision applications (MVA), pp. 5-8. IEEE, 2017.
Tautvydas Fyleris, Andrius Kriščiūnas, Valentas Cružauskas, Dalia Calneryté, and Rimantas Barauskas. Urban change detection from acrial images using convolutional neural networks and transfer learning. ISPRS International Journal of Geo-Information, 1144::246, 2022.
Hyeon-Ju Gim, Chang-Hoi Ho, Sujong Jeong, Jinwon Kim, Song Feng, and Michael I Hayes. Improved mapping and change detection of the start af the crop growing season in the corn belt from long-term 2020.
avhrr ndvi. Agricultural and forest meteorology.
Ritwik Gupta, Bryce Goodman, Nirav Patel, Ricky Hosfelt, Sandra Sajeev, Eric Heim, Jigar Doshi, Keane Lucas, Howie Choset, Matthew Gaston. Creating xbd: dataset for assessing building damage from satellite imagery. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition workshops, pp. 10-17,2019
Gene Hoxha, Giacomo Scuccato, and Farid Melgani. Improving image captioning systems with postprocessing strategics. IEEE Transactions on Geoscience and Remote Sensing, 61:1-13 2023.
Edward Hu, Yclong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv: 09685, 2021.
Yuan Hu, Jianlong Yuan, Congcong Wen, Xiaonan Lu, and Xiang Li. Rsgpt: remote sensing vision language model and benchmark. arXiv preprint arXiv:2307.1 1526666....... 2023.
Xiaowei Huang, Wenjic Ruan, Wei Huang, Gaojic Jin, Yi Dong, Changshun Wu, Saddek Bensalem, Ronghui Mu. Qi. Xingyu Zhao, et al. survey of safety and trustworthiness of large language models through the lens verification and validation, Artificial Intelligence Review, 57(7):175, 2024.
Peng Jin, Ryuichi Takanobu, Caiwan Zhang, Xiaochun Cao, and Li Yuan. Chat-univi: Unified visual representation empowers large language models with image and video understanding. arXiv preprint arXiv:2311.08044 2023.
Priyabrata Karmakar, Shyh Wei Teng, Manzur Murshed, Shaoning Pang, Yanyu Li, and Hao Lin. Crop mon- itoring by multimodal remote sensing: A review. Remote Sensing Applications: Society and Environment, 33:101093, 2024.
Ravneet Kaur, Reet Kamal Tiwari, Raman Maini, and Sartajvir Singh. framework for crop yield estimation and change detection using image fusion of microwave and optical satellite dataset. Quaternary 6(2):28, 2023.
Saad Mazhar Khan, Imran Shafi, Wasi Haider Butt, Isabel de Torre Diez, Miguel Angel López Flores, Juan Castanedo Galán, and Imran Ashraf. A systematic review of disaster management systems: approaches, challenges, and future directions. Land, 1218):1514, 2023.
Samar Khanna, Medhanic Irgau, David B Lobell, and Stefano Ernon. Explora: Parameter-effiiien extended pre-training to adapt vision transformers under domain shifts. arXiv preprint arXiv:2406. 10970,,,,,,, 2024a.
Samar Khanna, Patrick Liu, Linqi Zhou, Chenlin Meng, Robin Rombach, Marshall Burke, David B. Lobell, and Stefano Ermon. Diffusionsat: A generative foundation model for satellite imagery. In The Twelfth Inter- national Conference on Learning Representations, 2024b. URL htttps://openeeiieeeeee/ net forum? Id=ISwebNNEgg)
Kartik Kuckreja, Muhammad Sohail Danish, Muzammal Naseer, Abhijit Das, Salman Khan, and Fa- had Shahbaz. Khan. Geochat: Grounded large vision-language model for remote sensing. arXiv preprint arXiv:2311.15822 2023.
12

---

Published as a conference paper at ICLR 2025
Bo Li. Yuanhan Zhang, Dong Guo, Renrui Zhang. Feng Li, Hao Zhang, Kaichen Zhang. Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408. 03326, 2024a.
Kaiyu Li, Xiangyong Cao, and Deyu Meng. A new learning paradigm for foundation model-based remote- sensing change detection, IEEE Transactions on Geoscience and Remote Sensing, 62:1-12, 2024b.
Ke Li, Gang Wan, Gong Cheng, Liqiu Meng, and Junwei Han. Object detection optical remote sensing images: A survey and a new benchmark. ISPRS journal of photogrummeiry and remote sensing, 159:296- 307, 2020.
Xiang Li, Congcong Wen, Yuan Hu, Zhenghang Yuan, and Xiao Xiang Zhu. Vision- languag models in remote sensing: Current progress and future trends. IEEE Geoscience and Remote Sensing Magazine, 2024c.
Yanwei Li, Chengyao Wang, and Jiaya Jia. Llama-vid: An image is worth 2 tokens in large language models. arXiv preprint arXiv:2311. 17043, 2023.
Bin Lin, Bin Zhu, Yang Ye, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual represen- tation by alignment before projection, arXiv preprint arXiv:2311.10122, 2023.
Bin Lin, Zhenyu Tang, Yang Ye, Jiaxi Cui, Bin Zhu, Peng Jin, Junwu Zhang, Munan Ning, and Yuan, Moc-llava: Mixture of experts for large lasion-language models. arXiv preprint arXiv:2401.159/ 2024.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning.
arXiv preprint 2310:231000,744, 2023a.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jac Lec. Visual instruction tuning. Advances in neural information processing systems, 36, 2024a.
Ruyang Liu, Chen Li, Haoran Tang, Yixiao Ge, Ying Shan, and Ge Li. St-llm: Large language models are effective temporal learners. arXiv preprint arXiv:2404.00308, 2024b.
Shilong Liu, Hao Cheng, Haotian Liu, Hao Zhang, Feng Li, Tianhe Ren, Xueyan Zou, Jianwei Yang. Hang Su, Jun Zhu, et al. Llava-plus: Learning to use tools for creating multimodal agents. arXiv preprint vrXiv:2311.05453 2023b.
Yinhe Liu, Sunan Shi, Zhuo Zheng, Jue Wang, Shiqi Tian, and Yanfei Zhong. Mapchange: Enhancing semantic change detection with temporal-invariant historical maps based on deep triplet network. arXiv preprint arXiv:2401.1 1489. 2024c.
Sylvain Lobry, Diego Marcos, Jesse Murray, and Devis Tuia. Rsvqa: Visual question answering for remote sensing data. IEEE Transactions on Geoscience and Remote Sensing, 58(12):8555 8566, 2020.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arX1v.1711.05101, 2017.
Xiaoqiang Lu, Binqiang Wang, Xiangtao Zheng, and Xuelong Exploring models and data for remote sensing image caption generation. IEEE Transactions on Geoscience and Remote Sensing. 56(4):2183-2192) 2017.
Xiaoqiang Lu, Binqiang Wang, and Xiangtao Zheng. Sound active attention framework for remote sensing image captioning. IEEE Transactions on Geoscience and Remote Sensing, 198- 2019.
Junwei Luo, Zhen Pang, Yongjun Zhang, Tingzhu Wang, Linlin Wang, Bo Dang. Jiangwei Lao, Jian Wang. Jingdong Chen, Yihua Tan, et al. Skysensegpt: A fine grained instruction tuning dataset and model for remote sensing vision- languae understanding. arXiv preprint arX.v:2406.10111 2024.
Muhammad Maaz, Hamoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Video-chatgpt: Towards de- tailed video understanding via large vision and language models. arXiv preprint vrXiv:2306.05422 2023.
Oscar Manas, Alexandre Lacoste, Xavier Giró-i Nieto, David Vazquez, and Pau Rodriguez. Seasonal contrast: Unsupervised pre-training from uncurated remote sensing data. Proceedings of the IEEF/CVF Interna- tional Conference on Computer Vision, pp. 9414-9423, 2021.
Lynn Miller, Charlotte Pelletier, and Geoffrey Webb. Deep learning for satellite image time series analysis: A 2024.
review. arXiv preprint arXiv:2404.
Anthea L Mitchell, Ake Rosenqvist, and Brice Mora. Current remote sensing approaches to monitoring forest degradation in support of countries measurement, reporting and verification (mrv) systems for redd+. Carbon balance and management, 12:1-22, 2017.
13

---

Published as a conference paper at ICLR 2025
Waytehad Rose Moskolai, Wahabou Abdou, Albert Dipanda, and Kolyang. Application of deep learning archi- tectures for satellite image time series prediction: review. Remote Sensing, 13(23):4822, 2021.
Dilxat Muhtar, Zhenshi Li, Feng Gu, Xucliang Zhang, and Pengfeng Xiao. Lhrs-bot: Empowering remote sensing with vgi-enhanced large multimodal language model. arXiv preprint arXi::2402.02544, 2024.
Mubashir Noman, Noor Ahsan, Muzammal Naseer, Hisham Cholakkal, Rao Muhammad Anwer, Salman Khan, and Fahad Shahbaz Khan. Cdchat: A large multimodal model for remote sensing change description. arXiv preprint arXiv:2409. 16261, 2024.
Chao Pang, Jiang Wu, Jiayu Li, Yi Liu, Jiaxing Sun, Weijia Li, Xingxing Weng, Shuai Wang. Litong Feng, Gui-Song Xia, et al. H2rsvlm: Towards helpful and honest remote sensing large vision language model. arXiv preprint arXiv:2403.2022
2024.
Bo Qu, Xuelong Li, Dacheng Tao, and Xiaoqiang Lu. Deep semantic understanding of high resolution remote sensing image. 2016 International conference on computer, information and telecommunication systems (Cits). pp. 1-5. IEEE, 2016.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763. PMLR, 2021.
Maryam Rahnemoonfar, Tashnim Chowdhury, Argho Sarkar, Debvrat Varshney, Masoud Yari, and Robin Roberson Murphy. Floodnet: A high resolution aerial imagery dataset for post flood scene under- 2021.
standing. IEEE Access,
Hanoona Rasheed, Muhammad Maaz, Sahal Shaji, AbdeIrahman Shaker, Salman Khan, Hisham Cholakkal, Rao M Anwer, Erix Xing, Ming- Hsuan Yang, and Fahad S Khan. Glamm: Pixel grounding large multimodal model. arXiv preprint 2rXiv:2311.0333 2023.
Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. In Proceedingy of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 3505-3506, 2020.
Meredith Reba and Karen C Seto. A systematic review and assessment of algorithms to detect, characterize, and monitor urban land change. Remote sensing of environment, 242: 11 1739, 2020.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical image computing and computer-assisted intervention-MICAA. 2015: 18th inter- national conference, Munich, Germany, October 5-9 2015, proceedings, part III 18, pp. 234 234-241. Springer, 2015.
Li Shen, Yao Lu, Hao Chen, Hao Wei, Donghai Xie, Jiabao Yue, Rui Chen, Shouye Lv, and Bitao Jiang. S2looking: A satellite side-looking dataset for building change detection. Remote Sensing, 13(24):5094, 2021.
Jacob Shermeyer and Barry Haack. Remote sensing change detection methods track deforestation and growth threatened rainforests madre de dios, peru. Journal of Applied Remote Sensing, 9(1,096040-090440 2015.
Quan Sun, Yufeng Cui, Xiaosong Zhang. Fan Zhang, Qiying Yu. Zhengxiong Luo, Yueze Wang. Yongming Rao, Jingjing Liu, Tiejun Huang, et al. Generative multimodal models are in-context learners. arXiv preprint a3 13286, 2023.
Xian Sun, Peijin Wang. Zhiyuan Yan, Feng Xu, Ruiping Wang, Wenhui Diao, Chen, Jihao Li, Yingchao Feng, Tao Xu, et al. Fairlm: A benchmark dataset for fine- object recognition in high-resolution remote sensing imagery. ISPRS Journal of Photogrammetry and Remote Sensing, 184: 13--30,, 2022.
Shanmugam Tamilenthi and Rajagopalan Baskaran. Urban change detection based on remote sensing and gis study of salem revenue division, salem district, tamil nadu, india. European Association of Geographers, 4 (3):50-59, 2013.
Yunlong Tang, Jing Bi, Siting Xu, Luchuan Song, Susan Liang, Teng Wang. Daoan Zhang, Jie An, Jingyang Lin. Rongyi Zhu, et al. Video understanding with large language models: A survey. arXiv preprint
1rX1v:2312.17432. 2023.
Shiqi Tian, Xicheng Tan, Ailong Ma. Zhuo Zheng. Liangpei Zhang, and Yanfei Zhong. Cemporal-agnosee change region proposal for semantic change detection. ISPRS Journal of Photogrammetry and Remote Sensing, 204:306-320, 2023.
14

---

Published as a conference paper at ICLR 2025
Aysim Toker, Lukas Kondmann, Mark Weber, Marvin Eisenberger, Andrés Camero, Jingliang Hu, Ari- adna Pregel Hoderlein, Çağlar Şenaras, Timothy Davis, Daniel Cremers, al. Dynamicearthnet: Daily multi--pectral satellite dataset for semantic change segmentation. In Proceedings the ence on Computer Vision and Pattern Recognition, pp. 21158-21167, 2022.
the IEEE/CVF Confer-
SM Tonmoy, SM Zaman, Vinija Jain, Anku Rani, Vipula Rawte, Aman Chadha, and Amitava Das. A comprehensive survey of hallucination mitigation techniques in large language models. arXiv preprint arXiv:2401.01313 2024.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Nikolay Bash- lykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.0922 2023.
Sagar Verma, Akash Panigrahi, and Siddharth Gupta. Qfabrie: Multi-task change detection dataset. In Pro- ceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pp. 1052-1061, 2021.
Jin Wang, Lifu Zhang, Ruoxi Song, Changping Huang, Donghui Zhang, Senhao Liu, and Yanwen Liu. A fast hyperspectral change detection algorithm for agricultural crops based low-rank matrix and morphological feature extraction. Frontiers in Sustainable Food Systems, 8:1363726, 2024a.
Peng Wang. Shuai Bai, Sinan Tan, Shijie Wang. Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-v1: Enhancing Ussio-llnguage model's perception of the world at any resolution. arXiv preprint arXiv:2409, 12/ 2024b.
Wenhai Wang. Zhe Chen, Xiaokang Chen, Jiannan Wu, Xizhou Zhu, Gang Zeng, Ping Luo, Tong Lu, Jie Zhou, Yu Qiao, Visionllm: Large language model is also an open-ended decoder for viston-centric tasks. Advances in Neural Information Processing Systems, 36, 2024c.
Zhecheng Wang. Rajanie Prabha, Tianyuan Huang. Jiajun Wu, and Ram Rajagopal. Skyscript: A large and semantically diverse vision-language dataset for remote sensing. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 3805-5813, 2024d.
Katherine S Willis. Remote sensing change detection for ecological monitoring in united states protected areas. Biological Conservation, 182:233 242, 2015.
Gui-Song Xia. Jingwen Hu, Fan Hu, Baoguang Shi, Xiang Bai, Yanfei Zhong, Liangpei Zhang, and Xiaoqiang Lu. Aid: A benchmark data set for performance evaluation of aerial scene classification. IEEE Transactions on Geoscience and Remote Sensing, 55(7):3965-35) 3981, 2017.
Gui-Song Xia, Xiang Bai, Jian Ding, Zhen Zhu, Serge Belongie, Jiebo Luo, Mihai Datcu, Marcello Pelillo, and Liangpei Zhang. Dota: large-scale dataset for object detection in aerial images. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3974 3983, 2018.
Hanqiu Xu, Yifan Wang, Huade Guan, Tingting Shi, and Xisheng Hu. Detecting ecological changes with a remote sensing based ecological index (rsei) produced time series and change vector analysis. Remote Sensing, 11(20):2345, 2019.
Linrui Xu, Ling Zhao, Wang Guo, Qiujun Li, Kewang Long, Kaiqi Zou, Yuhan Wang, and Haifeng Li. Rs- gpt4v: A unified multimodal instruction-follooiin dataset for remote sensing image understanding. arXiv preprint arXiv:2406. Att.22906.1249, 2024.
Kunping Yang, Gui-Song Xia, Zicheng Liu, Bo Du, Wen Yang. Marcello Pelillo, and Liangpei Zhang. Asym- metric siamese networks for semantic change detection in aerial images. IEEE Transactions on Geoscience and Remote Sensing, 60:1-18, 2021.
Yi Yang and Shawn Newsam. Bag-of-visual-wordd and spatial extensions for land-use classification. In Pro- ceedings of the 18th SIGSPATIAL international conference on advances in geographic information systems, pp. 270 20002279.200 2010.
Zhenghang Yuan, Lichao Mou, Zhitong Xiong, and Xiao Xiang 7hu. Change detection meets visual question answering. IEEE Trunsactions on Geoscience and Remote Sensing, 60: 100.1-13,22 2022.
Zhenghang Yuan, Zhitong Xiong, Lichao Mou, and Xiao Xiang Zhu. Chatearthnet: A global-scale image-text dataset empowering vision-language geo-foundation models. arXiv preprint 133. 2024.
Yang Zhan, Zhitong Xiong, and Yuan Yuan. Skyeyegpt: Unifying remote sensing vision-language tasks via instruction tuning with large language model. arXiv preprint arXiv:2401.09712, 2024.
15

---

Published as a conference paper at ICLR 2025
Ao Zhang, Liming Zhao, Chen-Wei Xie, Yun Zheng, Wei Ji, and Tat-Seng Chua. Next-chat: An Imm for chat, detection and segmentation. arXiv preprint arXiv:231 1.04498, 2023a.
Hang Zhang, Xin Li, and Lidong Bing. Video-Ilama: An instruction-ture connneeion-ttneddddeeeeee audio-visual language model for video understanding, arXiv preprint arXiv:2306.0288. 2023b.
Wei Zhang, Miaoxin Cai, Tong Zhang, Yin Zhuang, and Xuerui Mao. Earthgpt: A universal multi-modal large language model for menti-sensor image comprehension remote sensing domain. IEEE Transactions on Geoscience and Remote Sensing, 2024.
Zhuo Zheng, Ailong Ma, Liangpei Zhang, and Yanfei Zhong. Change everywhere: Single-temporal super- vised object change detection in remote sensing imagory. Proceedings of the IEEE/CVF international conference on computer vision, pp. 15193-15202, 2021a.
Zhuo Zheng, Ailong Ma, Liangpei Zhang, and Yanfei Zhong. Change everywhere: Single-temporal super- vised object change detection in remote sensing imagery. In Proceedings of the IEEE/CVF international conference computer vision, pp. 15193-15202, 2021b.
Zhuo Zheng. Yanfei Zhong, Junjue Wang, Ailong Ma, and Liangpei Zhang. Building damage assessment for rapid disaster response with deep object-based semantic change detection framework: From natural disasters man-made disasters. Remote Sensing of Enviromment, 265:1 1636., 2021c.
Zhuo Zheng, Yanfci Zhong, Junjue Wang, Ailong Ma, and Liangpei Zhang. Building damage assessment for rapid disaster response with deep object-based semantic change detection framework: From natural disasters to man mane disasters. Remote Sensing of Environment, 265:1 12636, 2021d.
Zhuo Zheng, Yanfei Zhong, Shiqi Tian, Ailong Ma, and Liangnei Zhang. Changemask: Deep multi-task mncoder-rransfommeedddddeerrrr architecture for semantic change detection. ISPRS Journal of Photogranmetrr and Remote Sensing, 183:228-2361 2299, 2022a.
Zhuo Zheng, Yanfci Zhong, Shiqi Tian, Ailong Ma, and Liangpei Zhang. Changemask: Deep multi-task encoddertranssforme--deecolll architecture for semantic change detection, ISPRS Journal of Photogranmeetry and Remote Sensing, 183:228-2333 2022b.
Weihong Zhong, Xiaocheng Feng, Liang Zhao, Qiming Li, Lei Huang, Yuxuan Gu, Weitao Ma, Yuan Xu, and Bing Qin. Investigating and mitigating the multimodal hallucination snowballing in large vision-language models. arXiv preprint arXiv:2407.008: 2024.
Baichuan Zhou, Ying Hu, Xi Weng, Junlong Jia, Jie Luo, Xien Liu, Ji Wu, and Lei Huang. Tinyllava: A framework of small-scale large multimodal models. arXiv preprint arXiv:2402. 14289, 2024.
Bin Zhu, Bin Lin, Munan Ning. Yang Yan, Jiaxi Cui, HongFa Wang. Yatian Pang, Wenhao Jiang, Junwu Zhang, Zongwei Li, et al. Languagebind: Extending video-language pretraining to n-modality by language-hased semantic alignment. arXiv preprint arXiv:2310.01852 2023a.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision- language understanding with advanced large language models. arXiv preprint arXiv:2304.10522 2023b.
Yichen Zhu, Minjie Zhu, Ning I Liu, Zhicai Ou, Xiaofeng Mou, and Jian Tang. Llava-phi: Efficient multi-modal assistant with small language model. arXiv preprint 2rXiv:2401.02333 2024.
16

---

Published as a conference paper at ICLR 2025
APPENDIX
A RELATED WORK
ML for Temporal Earth Observation ML methods have been commonly used to model tempo- ral sequences of earth observation data (Moskolai et al. 2021 Miller et al.] 2024 Khanna et al.) 2024b). Change detection is widely studied earth observation task where the goal is to identify change between pairs (bitemporal) or sequences (multitemporal) of images capturing the same loca- Bai et al. 2023 Cheng et al. 2024). Automated methods for
tion over time (Chughtai et al. 2021 |
change detection can aid important humanitarian and sustainability efforts, including planning dis- aster relief (Rahnemoonfar et al.
2021 Gupta et al.
2019; Ban et al. 2020), tracking urban changes
(Tamilenthi & Baskaran 2013] Fylerss et al. 2022 Verma et al. (2021). and monitoring ecological
conditions [Willis|[201) [2015] Xu et al. 2019). deforestation (Shermeyer & Haack, [005]] De Bem al.)
2020 Decuyper et al. 2022). and crops (Gim et al.
2020; Kaur et al.] [023] Wang et al. (0024a).
Commonly studied change detection MI. subtasks include building change detection (Shen et al,]
2021) and damage detection (Gupta al. 2019), semantic change detection including land cover changes (Daudt et al. 2019 Yang et al. 2021 Toker et al. (022) and land use changes (Verma et al.
2021 Many ML-based approaches have been developed to accomplish these tasks, and almost all
use Siamese architecture where the vision encoder is shared across time (Zheng et al., 2021d b
20226 Liu al. 2024c Tian al. (0023), which we adopt as well.
Beyond change detection, multitemporal scene classification using the Functional Map of the World (fMoW) is common benchmark (Christie al. 2018). Finally, while several works have designed self-supervised approaches to leverage temporal sequences of carth observation data (Ayush et al. 2021 Manas et al., 2021] Cong et al.) 2022), ) none have developed vision-language models that can tasks on earth observation imagery over time. VLMs for Natural Images There have been substantial recent advancements in the development of vision-language models (VLMs) for natural images, notably including LLaVA (Liu et al. (024a) and its successor VA-155 (Liu et al. 2023a). These seminal works have demonstrated the effec- tiveness of curating large multimodal instruction- followwig datasets to train high performing vision- language models that can respond to user instructions about image data using a simple model design consisting of vision encoder get visual representations, projector layer to convert visual tokens into language space, and an L.MM to decode the instructions and converted visual tokens into a re- sponse. Subsequent works have proposed approaches to build upon these instruction-tune from improving performance with mixture of experts (Lin et al. 2024), achieving similar perfor- mance with fewer parameters (Zhou et al., 2024 Zhu et al., 2024), J and improve task transfer (Li et al. 2024a) to adding new capabilities like object detection (Wang et al. (024c), semantic seg- mentation (Rasheed et al.. 2023] Zhang et al. (023a), image generation (Sun et al. 2023). and tool
perform wide variety of instruction-
VLMs,
use (Liu et al., (023b).
Temporal VLMs Several recent works have developed VLMs that can engage conversation about videos. VideoChatGPT curates a video instruction set to train LLaVA-style architecture but combining both spatially pooled temporal features and temporally pooled spatial features on CLIP-encoded frames (Maaz et al. 2023b. VideoLLaVA then demonstrates improved performance by training a LLaVA LLaVA-style architecture with separate, pre-aligned image and video encoders on a joint inslloction-folloiiing dataset of both images and videos (Lin et al. (023). Subsequent works have primarily adopted LLaVAestyle architectures as well, but include additional layers aimed to
process visual tokens across time before inputting them into the LLM (Jin et al. 2023 Li et al.
2023 Zhang et al. 2023b Liu et al. 2024b)).
VLMS for EO Efforts develop instruction-follwwiii
VLMs for EO data have been rapidly in-
creasing (Hu et al.) 2023 Kuckreja et al.] 2023 Muhtar et al.) 2024 Pang et al., 2024). These
methods commonly curate many different single image EO datasets and convert them to instruction- following tasks, then instruction-tum
LALaAA-ike model on the curated dataset. Importantly, no prior work has developed VLMs for temporal sequences for EO except for CDChat (Noman et al.) 2024). a concurrent work that fine-tunes LLaVA-1.5 1.5 on two bitemporal change detection datasets. In contrast, our work builds upon the more recent VideoLLaVA architecture, enabling the handling of sequences with up eight images. Additionally, including many different single image tasks as well as temporal tasks from four EO datasets, broad- ening the scope and capabilities of the model.
develop a public enstructionffolowinnn dataset
17

---

Published as a conference paper at ICLR 2025
Dataset
Examples (Images)
Temporal
Detection
GeoChat Instruct Kuckrejj et al.
2023)
309k
SkyEyeGPT (Zhan al.
2024
968k
MMRS1M M Zhang et al. 2024)
1M
LHRS-Instruct (Muhtar al.
(2024)
42k
ChatEarthNet (uuan et al.
2024)
173k
H2RSVLM -LS---sruuttttt ( al.
(2024)
180k
SkySenseGPT (Luo et al.
2024)
1.4M
RS-GPT4V (Xu et al. 2024)
991k
TEOChatlas (Ours)
554k (1.2M)
Table 6: Comparison of TEOChatlas to other mnstruction-folloviin EO datasets. TEOChatlas has large number of instruction-following examples, and the first EO dataset have temporal tasks.
Dataset
Temporal
Task
Sensor
# Examples
Object Detection, VQA
GaoFen, Jilin-1, Sentinel-2, DJI Mavic Pro
GeoChat
Single
308,861
Scene Classification
iMoW Multitemporal Sequence Classification
WorldView-2/33, Sentinel-2
83,412
xBD
Bitemporal
Building Damage Detection
WorldView-2
19,749
S2Looking
Bitemporal Building Change Detection GaoFen, SuperView, BeiJing-2
17,090
Multi-Task Urban
QFabric
Multitemporal
WorldView-2
124,959
Change Detection
Table 7: Datasets included in the TEOChatlas training set. TEOChatlas is composed of single image and temporal image datasets from a variety of sensors includes multiple instruction-following tasks both spatial and temporal reasoning.
TRAINING DATASETS
TEOChatlas consists of GeoChat.Instrus (Kuckreja al., 2023), an instruction-follooinng dataset with single earth observation images, as well as four temporal earth observation benchmarks (fMoW (Cristiie al., 2018), xBD (Gupta et al. 2019). S2Looking (Shen et al. [02T), and QFabric (Verma et al. (021)). We describe how we process each dataset in more detail below and summary of the dataset statistics is provided in Table 7 We note that for all datasets, we (1) resize the shorter side to 224 then crop to 224x224 before inputting the images into the network and (2) randomly sample between instruction-follon following tasks when creating examples, sometimes randomly sampling be- tween different phrasings of the same task to introduce diversity in the wording. A spatial map of the locations of the examples in TEOChatlas is shown in Figure 5 and a comparison of TEOChatlas other coiructtioo-folowiing EO datasets is shown in Table
GeoChat Instruct The GeoChat Instruct dataset (Kuckreja et al. 2023) an instruction- followiinnn dataset composed of multiple single image EO datasets spanning tasks including object detection derived from SAMRS (which is itself a composition of DOTA (Xia et al. 2018). DIOR (Li et al. 2020), and FAIRIM (Sun al., (022)). visual question answering from LRBEN (Lobry et al. 2020 and Floodnet (Rahnemoonfar et al. 202T), and scene classification from NWPU-RESISC-45
dataset include scene classification (classify the image into one of several classes provided in the prompt). visual question answering (answer a question about the image), referring expression (identify the location of an object given an expression describing its characteristics), region captioning (describe a location represented as bounding box in the prompt), detailed description (describing the whole image in detail), grounded description (describing the whole image in detail with bounding boxes included in the description), and multi-round conversation (multiple turns of conversation). In total, the BooChht_Instruee training dataset consists of 308,861 single instruction followiin examples 106,74 747 images.
(Cheng et al. 2017). The instruction-followe following tasks in the GeoChat._Istre
18

---

example for Published as a conference paper at ICLR 2025
[MoW Sentinel
MMoW RGB
XBD
Figure Locations the examples the xBD fMoW RGB, fMoW Sentinel subsets TEOChatlas dataset. GeoChat, S2Looking, and QFabric do not have geographic locations in the released data. However, the datasets GeoChat span many locations around the world, and both S2Looking and QFabric contain globally distributed locations well.
iMoW We use both the fMoW RGB dataset Christtie et al. 2018) and the fMoo Sentinel dataset (Cong et al. 2022), sampling randomly between the two for each example in the provided training set split. We follow standard practice for converting fMoW to classification dataset and crop each image to provided bounding box. create one instruction-follwwii example for every image in the training leading to 83,412 examples 83,412 images. We report model performance both the fMoW RGB and fMoW Sentinel validation sets.
xBD We include the xBD (Gupta al. 2019) training dataset in TEOChatlas. To both increase the representation of building change detection in TEOChatlas and maintain image resolution, we crop the 1024x1024 images into 16 256x256 images. When creating instruction-following tasks, we convert all building polygons to their minimal axis-aligned bounding boxes. We create one instruction-folloinng example for every image in the training set, leading 19,749 examples 19,749 images. We use the original polygons when creating the segmentation masks for evaluation (see below), and report model performance on the test set.
S2Looking We include the S2Looking (Shen et al. 2021) training dataset in TEOChatlas. Similarly xBD, we crop the 1024x 1024 images into 16 256x256 images and convert all building polygons to their minimal axis-aligned bounding boxes. We create one cdtructiionfollowiiii
every image the training set, leading to 17,090 examples on 17,090 images. We use the origi- nal polygons when creating the segmentation masks for evaluation (see below), and report model performance on test set.
We include the QFabric (Verma et al., 2021 training dataset in TEOChatlas. The images in Qabric cover large area with 10,000 x 10,000 pixels, which we crop into around 1600 256x256 images. We drop any cropped images that have polygons and convert all polygons to minimal axis-aligned bounding boxes. We create one instruction-followwe following example for every image in the training set, leading 124,959 examples 124,959 images. We the original polygons when creating the segmentation masks for evaluation (see below), and report performance on the test set.
QFabric
TEMPORAL INSTRUCTION-FOLLOWNN TASKS
We provide a summary of all temporal eestruction-follovinni tasks included in the TEOChatlas train- ing in Table g For description and captioning temporal tasks, use GPT-40 increase the diversity of the responses to avoid template fitting which we found plagued the model in early ex- periments.
19

---

Published as a conference paper at ICLR 2025
Dutaset
Task Calegory
Example
Sequence Classification
User; What does sequence images belong to?
AI: Lake or pond.
User; Idculily all the buildings the image.
Localization
Al: <bhor, <bbox>, <bbor>, <bboxs, <bborr
Classification
inage. Choose from: damage, Minor Damage, Major Damage, Destroed.
User: Clasdy the level of damage experienced by the builling at location <bbox-in the second
Al: Minor Damage.
[se:: What disaster has occurred here??
eartquuae..
User: Which part of the image was most affected by the disaster?
Change Question
Al: The couter left of the image was most affected by the disaster.
Answering
User: How many buildings the image have been destroyed?
Al: 3.
User: Are there any damaged buildings in the image?
Al: Yos.
User: Are there any damaged builddngg in this region <c <chouss?
XBD
Regien-bused Question Answering
User: How severe is the damage to this huilding <bbox>>?
Al: The given baikding has been destroyed.
User: Identify severely damaged destroyed buildings in image.
Al: <bbox>, <bbox,>
Spatial Change Referring
User: Identify the damaged buildings in the isection,, of the image.
Expression
Al: There arc no danagod buildings in the bottom right of the image.
User; Identify the destroyed buildings in this area <bboxt..
Al: There
destroyed baildings the given
Rogion Captioning
User: How has this building < <box changed!
Al: The given huikling has been destroyed.
User: Descrhee how buildiigs have changed.
Detailed Description
Al: There has been a flood that has damaged multiple buildings the aren.
ser: Desetbe how the buildings have changed. Include bounding boxes.
Grounded Description
Al: There has been a flood has damaged a building the location <hhox>.
User: Identity all changed buildings.
Localization
Al: cb--cboxss <bboxx, <bbox>
User: Have any buildings been constructed in the area' Please answer with Yes or No.
Change Quettion Answering
Al: Yes.
User: How many buildings the image 3.:
built destroyed?
Region based Temporal Question Answering
Lser: Has the area <bbox changed? Please with Yes No.
S2Looking
Spatial Change Referring Expression
Al: <hhoxs, <choxs,, <hhox>>
User: Identity the (constracted, destrocted] buildings in the image.
User: What is largest buikding that experienced change?
AI: <hbox>.
Description
Al: Five buildings have been constructed.
detaaled description of buildings
changed.
User: Provide a detalled description of The buildings that have changed. Include bounding boses
Description
your output.
Al: Three buildings have been destroyed <bbox>, <bboxt, <bbox>.
User: Descrthe how the buildings have changed in this <bbox..
Region Captioning
Al: Five baildings have been constructed.
User: Identily all images which land was cleared in this regicn <bbox:> from the previous image.
Al: Image Image
User; In which images is scelass> visible in this region <:bbor>?
Al: Image Image Image
Temporal Referring Expression
User: Identity images in which construction occurred in this region <bboxx> from the previous image.
Al: Image
User: Identify all images in which a construction project was hegun in this region, area vc xchoox>> from the previous image.
Image
User: Ideatify the image when <cehange phrase> in this region <bboxs.
Al: Image 2
QFabric
User; Has there been urban development in this area <bbox:?
Region-Bused Based Question
Al: Yes
Answering
User: Identify the type urhan development that has occurred in this area <choxx>.
Al: Residential
User: Has there been urhan development in this area <hhox>> between image 'N and image M?
Temporal Rogion-Based Question Answering
User: What is the development status in this region <bbox,> in image N?
Al: Finished Construction
Region Captioning
User: How has area <bbox' changed as a resalt urban development!
Al: This region was greenland at first, and then became finished construction.
image MP Al: Between the first and fourth images. this area was cleared land at first, and then became midway- finished construction and then became finished construction.
User: How has the area <<buxx>. changed due urbun development from
Temporal Region
Captioning
Table 8: Examples of the many temporal instruction-followe following tasks in TEOChatlas.
D SINGLE IMAGE DATASETS
We evaluate TEOChat the following datasets:
20

---

Published as a conference paper at ICLR 2025
1. AID (Xia et al. 2017), scene classification dataset composed of 10,000 images each classified into one 30 scene classes.
2. UCMerced (Yang & Newsam 2010), land use classification dataset consisting of 2100 images each classified into one of 21 land use classes.
3. LRBEN and HRBEN (Lobry et al., 2020), low resolution and high resolution datasets for visual question answering. We consider the presence task (answering yes or no for whether object is in the image) and comparison tasks (answering yes or no to comparison question, for example are there more buildings than roads in the image).
TEOCHAT PROMPTS
We use the following system prompt for the tasks in TEOChatlas.
A chat between curious user and an artificial intelligence assistant. The assistant gives help- ful, detailed, and polite answers to the user's questions. USER: This sequence of (high resolution, optical} satellite images (from <sensor>> J: <video>>.
where we randomly inject the information in curly brackets when metadata (resolution or sensor) is available and replace the "<video>> token with the "<image>" token N times, with N the number of images in the sequence. After this prompt we add the task instruction. Then, for tasks that require choice of options, we provide the options in the prompt. For tasks that require bounding boxes to be output, we include "Please include bounding boxes of the form [x_min, y y-min, x_max, y_max] in your response." While this does not guarantee correct output parsing in every case, fewer than 0. % errors occurred across all evaluations we conducted. Finally, we add "ASSISTANT.".
ADDITIONAL TRAINING DETAILS
We fine-tune the LLM in TEOChat using LoRA rank 128. Before inputting the images into the image encoder, resize the shorter dimension of each image to 224 pixels, and then apply center crop to obtain a 224x224 image. These design decisions allow us to train the model with sequences of up to 8 images increase the effective batch size, we use 8 steps of gradient accumulation and train the network on 10 GPUs using data parallelism together with optimizer state partitioning, gradient partitioning. parameter partitioning, and CPU offloading with DeepSpeed Zero-3 Offload (Rasley et al.) 2020). We optimize the network using AdamW (Loshchilov Hutter 2017) with a cosine learning rate scheduler, a peak learning rate of 2c-5, and warmup of 3% of the epoch. The model takes 125 hours train per epoch and we train for 2 epochs.
single NVIDIA A4000 GPU (16B of VRAM) with batch size of 1. To
G GEOCHAT TEMPORAL POST-PROCESSING
We describe how we obtain GeoChat predictions the temporal prediction tasks below.
G.1 TEMPORAL SCENE CLASSIFICATION
We run GeoChat on each image in the sequence to produce a per-image classification, then take the majority vote across the sequence as GeoChat's prediction over the whole sequence.
G.2 CHANGE DETECTION
xBD Building Localization We prompt GeoChat identify all buildings in the first image then the output bounding boxes to obtain the predicted segmentation mask.
xBD Damage Classification We input each ground truth box and the second image into GeoChat as separate examples, instructing to classify the given box into the four damage categories, and use the classification labels output by GeoChat with the boxes construct the segmentation mask.
21

---

Published as a conference paper at ICLR 2025
S2Looking Building Change Detection We prompt GeoChat to identify all buildings in first and second images individually. From the bounding box predictions on each image, we create binary segmentation masks then take the difference to obtain GeoChat's predicted change mask, masking out pixels from overlapping boxes across the two images to avoid small deviations in box predictions between the two images from harming performance.
G.3 SPATIAL REFERRING EXPRESSION
xBD We use the same procedure for building localization, but prompt GeoChat identify only the 'destroyed' buildings, for example.
the spatS2Lokking The spatial referring expression tasks for S2Looking to identify the structed or destructed buildings. For both, we prompt GeoChat to identify all buildings in the first image and in the second image, construct segmentation mask for each, then take difference. We use the pixels in the first image but not the second as destructed and the inverse as constructed.
G.4 CHANGE QUESTION ANSWERING
xBD As the xBD questions pertain to building damage levels and there is no damage to buildings in the pre-disaster images, we input the second image and the question (e.g. are there any severely damaged or destroyed buildings in this area?) to GeoChat to obtain the answer.
The change questions for o2ooon ask the model identify whether not buildings changed between the images. To obtain this binary prediction from GeoChat, we follow a similar approach and instruct it identify the buildings in the first image and the buildings the second, then consider the predicted building from each image as changed if it does not overlap a box in the other image. If any buildings changed, the binary prediction yes, otherwise it is no.
S2Iookingg
G.5 REGION-BASED CHANGE QUESTION ANSWERING
xBD As the input bounding box is the only difference between this task and change question an- swering, we use similar postprocessing as that task, inputting the second image and a question as well as a bounding box GeoChat to obtain the answer.
S2Looking Similarly, this task only differs from change question answering due to the bounding box input, so we perform the same postprocessing as for that task, but only taking into account the changes inside the input bounding box.
These tasks are standard classification tasks, so we follow similar procedure as for tem- poral scene classification and feed each image in the sequence (either or 5 images) along with the bounding box in the prompt into GeoChat, then take majority vote to get the final classification.
QFabric
H EVALUATION SETUP
For previously existing tasks, follow the metrics reported in the original works. For new tasks, we use accuracy. See further detail below.
Change Detection and Spatial Referring Expression To evaluate TEOChat on building change detection (CD) and spatial referring expression (SRE), we follow the standard performance metrics of the building localization (Loc.) and damage classification (Dmg Cls.) subtasks on xBD and building change detection (Det.) subtask on S2Looking, Specifically, for building localization and building change detection, we convert the bounding box predictions and ground truth polygons to segmentation masks and compute per-pixel F, between the predicted and ground truth segmentation masks. For building damage classification, we input the ground truth polygon to the model to obtain classification label, then create a segmentation mask where the pixels within the ground truth polygon are assigned the predicted label, and compute wlightwe F, between this mask and the ground truth mask. For SRE,
the same evaluation scheme building localization.
Change Question Answering and Region-Baas Based Change Question Answering We measure ac- curacy change question answering (QA) and region-based change question answering (RQA) for xBD and S2Looking. For QFabric, we use the proposed evaluation metrics from Verma et al. (202T) for comparability to the specialists, namely per-pixel F, after converting the classified polygons to
22

---

Published as a conference paper at ICLR 2025
segmentation masks as also done in xBD damage classification. We consider both image task and image task as done in Verma et al. (202I).
Temporal Referring Expression and Region-based Temporal Question Answering For tem- poral referring expression (TRE), we measure performance of the models to identify the correct image(s) using accuracy. For region-based temporal question answering (RTQA), we use F1 for the change status task (classifying the development status of each image in a sequence of 5 images) following Verma et al. (021), and use accuracy for the other tasks.
I ADDITIONAL RESULTS
1.1 IMAGE REFERENCE ABLATIONS
We run a ablation to investigate why image identifiers improve performance on the canonical bitem- poral tasks (Table ( and run another ablation measure the importance of including the image identifiers in the prompt on the temporal referring tasks (Table I0).
Image Reference Strategy
xBD Dmg Cls. (Fi) S2Looking Det. (Fi)
No References
44.7
25.5
Only 'Image 1'
47.5
25.8
Only 'Image 2'
49.2
34.5
Both 'Image 1 and 'Image 2'
50.0
34.5
Table 9: Impact of difffrent image reference strategies on the canonical bitemporal datasets. Including 'Image 2' leads to much larger performance gains than including 'Image 1', constituting most of the perfor- mance improvement from including both image references.
Image Identifiers
RTQA QFabric (Acc.)
TRE QFabric (Acc.)
59.2
58.6
71.7
74.9
Table 10: Impact of including image references in the test-time prompt TEOChat's performance region-based temporal question answering and temporal referring expression tasks, Including image references leads to substantial benefits on both tasks.
1.2 FINE-TUNED CAPTIONING RESULTS
We evaluate TEOChat on three BO captioning datasets used as benchmarks in prior work (Hu et al. 2023 Zhan et al. 2024). namely UCM-Caption (Qu al. 2016). Saannee-Captionss Qu al. (2016), and RSICD (Lu et al. 2017). Given these works include these datasets the training data, we follow Hu al. (203) and fine-tune TEOChat on each of the three datasets separately. We use the same training procedure as used while training on T'ECChatlas except we fine-tune until convergence (5, 20, and epochs respectively). For comparison we also fine-tune GeoChat as results on these datasets are not reported in Kuckreja et al. (2023), and use their same training procedure when fine- TEOChat. We evaluate using the same metrics reported in Hu et al.] (2023) and Zhan et al. 2024). Fine-tuning results are reported in Tables to 14 and zero-shot results are reported in Tables to
tuning on each dataset for the same number of epochs
In the fine-tuning setting, we find TEOChat exceeds the performance of specialists and matches gen- eralist single image EO models on the larger two of the three captioning datasets (UCM -captions and RSICD), and shows strong captioning abilities qualitatively on all three datasets (Figure B3). On Sydney-captions, TEOChat underperforms RSGPT, but qualitatively we find TEOChat's 's captions are high quality. Importantly, TEOChat outperforms GeoChat across all metrics all three cap- tioning datasets except METEOR on UCM-captions, for which slightly underperforms GeoChat (-0.4%). In the zero-shot setting. TEOChat performs similarly to outperforms GeoChat aver- age across all metrics, with notable improvements on the largest, more difficult captioning dataset (RSICD). Interestingly, MiniGPT-v2 achieves impressive zero-shot performance these datasets.
23

---

Published as a conference paper at ICLR 2025
We believe the explanation for strong fine-tuning performance of RSGPT and MiniGPT-v2's strong zero-shot and fine-tuning performance is twofold. First, we largely attribute this to the inclusion of higher proportion of captioning datasets in their training data, whereas GeoChat and TEOChat have more scene classification and question answering tasks. Second, these three captioning datasets have several known issues which make their use benchmarks suboptimal, including limited vocabulary and very short descriptions, and sometimes even misspelled words and grammatical errors. We caution researchers about using these benchmarks without careful analysis. We finally emphasize that in addition to these single image capabilities, TEOChat has temporal reasoning capabilities that none of these other models have.
Model
BLEU-1 BLEU-2
BLEU-3
BLEU-4
METEOR
ROGUE.L
CIDEr
Specialists
SAA al. 2019
79.6
74.0
69.1
64.8
38.6
69.4
294.5
Post processing Honsa et al. 2023
79.7
73.0
67.4
62.6
40.8
74.1
309.6
Generalists
MiniGPT-4 Zhu et
2023b
30.9
27.6
22.2
18.
33.4
41.4
0.0
Shikra (Che et 2023b
58.9
43.3
32.6
56.7
MiniGi
2023a
81.1
60.3
45.1
36.2
32.4
56.6
61..7
RSGPT (Hu cr 2023
86.1
79.1
72.3
65.7
42.2
78.3
333.2
GeoChat (Kickreja at.
82.9
74.9
68.5
44.4
78.1
299.1
78.2
72.4
43.9
316.5
Table 11:
Captioning performance on the UCM-captions dataset.
Model
BLEU-1
BLEU-2
BLEU-3
BLBU-
METEOR
ROGUE
CIDEr
Specialists
SAA et 2019
68.8
60.7
52.9
45.4
30.5
58.2
170.5
Post-processing (Hxhha
2023
78.4
69.9
63.2
57.2
39.5
71.1
255.5
Generalists
MiniGPT-4 Zhu
2023b
29.5
25.9
20.3
16.4
32.0
42.7
0.1
Shikra (Chen aral. 202th
77.5
53.2
37.0
27.8
29.4
53.3
26.8
Min Piz(Ci nea 2t
2023a
77.4
55.8
40.6
29.9
52.1
33.8
RSGPT (Hu ctal.
82.3
75.3
68.6
62.2
41.4
74.8
273.1
GeoChat (Kuckreja et al. 2023
74.1
61.2
53.1
47.0
34.2
67.8
184.3
TEOChat
75.1
64.6
56.9
50.5
36.2
69.5
204.6
Table 12: Captioning performance on the Sydney-captions dataset.
Model
BLEU-1
BLEU-2
BLEU-3
BLEU-
METHOR
ROGUEJI
CIDE
Specialists
SAA (Lu et 2019
2019
59.4
45.1
35.3
28.1
26.1
49.6
132.4
Post-processin Hoxbu et al.
2023
62.9
46.0
35.7
28.7
25.3
47.3
75.6
Generafists
MiniGPT-4 Zhu et 2023b
34.0
31.8
25.8
20.6
33.2
40.7
Shikra (Cheh 20250
82.6
62.5
45.2
34.6
30.3
$3.6
19.9
MiniGPT-vz (Chen
2023g
83.1
64.6
47.8
37.3
30.2
$4.0
52.4
RSGPT (Hu ata. 2015
70.3
54.2
44.0
36.8
30.1
53.3
102.9
GeoChat (Kuckrejj al.
40.1
95.2
TT:OChal
55.7
Table 13: Captioning performance the RSICD captions dataset.
24

---

Published as a conference paper at ICLR 2025
Model
BLEU-1
BLEU-2
BLEU-3
BLEU-4
METEOR
ROGUE.L
CIDEr
Specialists
SAA at a 2019
69.3
59.9
52.4
46.1
31.7
59.1
199.1
Post-procsssng (Huxha et al. 2023
55.4
35.2
Generalists
MiniGPT-4 Zhu 2023b
31.5
28.4
22.8
18.4
32,9
41.6
Shikra (Chen al. 20256
80.4
58.2
41.8
32.1
30.8
54.5
34.5
MiniGP Lx2 (Chen 2023a
80.5
60.2
44.5
35.3
30.8
54.2
49.0
RSGPT (Hu d
37.9
236..4
GeoChal (Kuckreja
2023
75.4
62.5
53.9
47.5
35.5
66.8
192.9
TEOChat
76.9
65.1
56.8
50.5
36.3
68.2
206.1
Table 14: Average performance across the three captioning datasets.
Model
BLEU-1
BLEU-2
BLEU-3
BLEU-4
METEOR
ROGUE.L
CIDEr
MiniGPT-v2 (Chen al. 2023a
17.6
9.1
14.8
23,5
GeoChat (Kuckreja 2023
22.8
10.3
1.7
10.9
21.5
13.9
TROChat
8.8
4.3
10.3
20.8
15.7
Table 15: Zero-shot captioning performance the UCM-captions dataset.
Model
BLEU-1
BLEU-2
BLEU-3
BLEU-4
METEOR
ROGUE L
CIDEr
MiniGPT-v2 (Chen et al. 2023a
18.6
8.1
2.8
1.0
13,5
24.4
10.0
GeoChat Kuckreja et al.
2023
19.8
6.9
0.0
0.0
8.5
19.1
4.3
TEOChat
28.4
8.4
2.1
0.0
8.8
21.8
4.1
Table 16: Zero-shot captioning performance on the Sydney-captions dataset.
Model
BLEU-1
BLEU-2
BLEU-3
BLEU-4
METEOR
ROGUE
CIDEr
MiniGPT-v2 (Chen
2023a
19.9
9.2
13.7
21.4
10.7
GeoChat Kuckreja cl
2023
22.3
8.8
3.2
1.2
10.5
18.8
7.1
TBOChat
29.2
12.9
5,5
2.5
10.4
19.5
10.1
Table 17: Zero-shot captioning performance the RSICD dataset.
Model
BLEU-1
BLEU-2
BLEU-3
BLEU-4
METEOR
ROGUE_L
CIDEr
MiniGPT-v2 (Chen et al 2023a
18.7
8.8
3.6
1.5
14.0
23.1
12.5
GeoChat Kučkreja al. 2023
21.6
8.7
10.0
19.8
TROChat
22.1
8.5
9.8
20.7
10.0
Table 18: Average zero-shot captioning performance across the three datasets.
I.3 SINGLE IMAGE GROUNDING RESULTS
We evaluate TEOChat's performance on single image grounding (Table 19). TEOChat outperforms MiniGPTv2 overall (+5.3%) and outperforms GeoChat in detecting large objects (+2.2%), but un- derperforms GeoChat across the other grounding tasks (-5.5% overall) which we suspect is due the lower spatial resolution (224 vs. 504) required to handle temporal inputs. Qualitatively, TEOChat tends to make similar correct predictions and errors as GeoChat, but is a little less precise. To in- vestigate the effect of lower spatial resolution GeoChat's performance, 224x224 images (after performing bicubic interpolation of the positional embeddings to account for the change in spatial resolution). This results in huge drops (20-40% absolute drop) in grounding performance across all tasks, potentially suggesting a high sensitivity to input positional information on the grounding tasks.
evaluate GeoChat on
25

---

Published as a conference paper at ICLR 2025
Large
Single object grounding
oulti-object grounding
[xadox]
[grounding]
MiniGPP--2
GeoChat Kuasre & 2024
26,4
50.4
63.1
50.6
22.8
45.1
27.5
43.8
GeoChal 201 X22
1.1
4.0
16.3
6.3
2.3
54
4.6
5.3
TEOChat
16.3
43.0
65.5
43.1
19.1
38.2
26.2
57.3
Table 19: Single EO image grounding performance (Aco@0.5). We evaluate all tasks using axis aligned bounding boxes. All metrics are defined as in Kuckreja et al. (2023).
1.4 TEMPORAL SEQUENCE SAMPLING ABLATION
We run an ablation to measure the impact of the random samples of 8 images taken when the image sequences longer than 8 images fMoW RGB and fMoW Sentinel (Table 20
Run
fMoW RGB
fMoW Sentinel
75.1
45.5
75.2
45.3
75.1
45.6
75.1
45.4
75.2
45,4
Table 20: Impact of random samples of 8 images from long image sequences IMoW RGB and fMoW Sentinel validation set performace. There minimal variance (0.1-0.2%) from different random samples.
1.5 ADDITIONAL TEMPORAL QUALITATIVE RESULTS
Examples of TEOChat's responses on canonical temporal tasks are shown in Figure
Elaaxify the iequence of teages of: flooded road, lake or ponz, aquacuiture, don, mcuntain treil.
Image
Irage
Image
Irage 4
disaster occurred
hurricane has ussurred in area.
Imsge
Irage 2
Slassify the level of darage expertenced the bust Iding
location (0, 53].
busidding
49, cestroyed.
Taage
Trage
Identify changed builddinga using bounding forr y.max].
Image
Iroge 2
which construstion finsshed?
Image
Image
Imace
Image 4
Image
Image
Figure 6: Qualitative examples of responses produced by TEOChat on temporal tasks.
26

---

Published as a conference paper at ICLR 2025
1.6 SINGLE IMAGE QUALITATIVE RESULTS
Examples of responses output by TEOChat on various single EO image tasks are shown in Figure and examples of TEOChat responses on single EO image captioning tasks are provided in Figure
Mhal is Uhis? [21, 3, 47, 19]
backetball court the top
Identafy the location of the builcing the right of the inage using bounding box of the form [x_min, y_min, x_mox, y_max]
[66, 87,
Classify this image Refinery, Compressor Station, Pipeline, Processing Plant, Well Pad.
0il Refinery.
Is there road next body of water?
Yes, there road present body
Figure 7: Examples of TEOChat's responses on single image tasks.
27

---

and Published as a conference paper at ICLR 2025
It river with many both banks of the river.
A river goee through dense forest.
Houte iut te
UCM Captions
Lots of care parked in 1ines in the parking lot.
Many cars densely arranged the parking lot.
Two houses with red roofs are surrounded by bushes and sands in the sparse residential area
house with sands and bushes surrounded has sone cars parked beside.
residential. area with many houses arranged lines and some roads through this area.
residential area with houses arranged in lines and separated straight roads.
meadow some green bushes on passed by.
highway
Sydney Captions
bushes and while bunkers the meadow while a highway passed by.
There
Some curving runways with some marking lines on while some lawns beside.
There white marking lines parallel straight runways.
tennis
surrounded by trees.
two tennis courts are next road and some green trees.
the sirport covers large area and has a lot planes.
RSICD
many planes parked large building airport.
a short, slender bridge connects two banks of the river.
bridge river with sone buildings side of
1t.
Figure 8: Examples of TEOChat's responses on the three single image EO captioning datasets after fine-
tuning following Hu et al. 2023)
J BROADER IMPACTS
We show that TEOChat is capable of temporal reasoning, achieving strong performance on several real world tasks like building damage detection. Automated, generalist methods for processing tem- poral EO data like TEOChat have the potential to lead to significant, positive societal impacts. For example, we believe that such methods could be deployed at scale sequences of Earth observa- tion data to automatically derive information, ranging from disaster response to urban development monitoring. This analysis is typically conducted by experts acquiring data the ground or by man-
28

---

Published as a conference paper at ICLR 2025
ually inspecting UAV/saeelitte imagery. which becomes impractical to do in large areas which may require several thousands of high resolution images to capture. Such technology could, for example, aid government agencies and humanitarian organizations acquire information much more rapidly, which is key for disaster response (Khan et al. 2023). We hope our work can inspire more work on developing vision and language assistants for important temporal EO tasks and lead to helpful new technologies that can benefit society.
We highlight a few potential negative societal impacts of TEOChat. First, TEOChat can automat- ically extract information from Earth observation imagery, which may lead to privacy concerns, especially as the imagery achieves higher and higher spatial resolution. Second, TEOChat, like other large multimodal models, does not always produce accurate outputs, sometimes hallucinat- ing information or objects the images, which has the potential to mislead users if misused. also possible but unlikely that fine-tuning on TEOChatlas may affect safeguards employed in Llama Ongoing work on mitigating hallucinations in L.MS (Tonmoy et al. 2024) and LMMs (Zhong et 2024), well as work addressing safety of LLMs (Huang et al. 2024). both have the potential to translate to LMMs for Earth observation, but while these approaches continue to mature, users should check outputs for important applications.
K AUTHOR CONTRIBUTIONS
1. Jeremy Andrew Irvin: Developed the initial research idea, formulated instruction- following tasks, prepared datasets, implemented training experiments and evaluation code, designed and conducted training experiments, drafted the paper.
2. Emily Ruoyu Liu, Joyce Chuyi Chen, Ines Dormoy: Prepared datasets, helped design ar- chitecture and formulate and edited the paper.
tasks, implemented evaluation code, drafted
3. Jinyoung Kim: Prepared data, edited the paper.
4. Samar Khanna: Helped design architecture, designed and conducted training experi- ments, edited the paper.
5. Zhuo Zheng
Supervised the project, helped design architecture and formulate instruction- following tasks, edited the paper.
6. Stefano Ermon: Co-developed the research idea, supervised the project, edited the paper.
29


Reviews:
[{"review_id": "dP63HiznTj", "reviewer": "Reviewer_7dgE", "summary": "The work attempts to develop the first vision-language model for temporal earth observation data (EOD). When an EOD is prompted with temporal images and NLP, the proposed model can perform tasks that need temporal reasoning,", "questions": "As mentioned in the weakness", "limitations": "", "rating": 1, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 1, "strengths": "This paper introduces TEOChat, which is the first unified model for temporal earth observation (EO) data\nThe authors create a novel EO dataset", "weaknesses": "It would be more convincing if the authors could provide more evidence on the diversity and representativeness of the dataset, as well as fine-tuning results on supervised datasets.\n\nWhy only a comparison with GeoChat is made, whereas many other similar works are available?\n\nWhat is the performance for other RS captioning datasets such as UCM-CAPTIONS and SYDNEY-CAPTIONS datasets which are mentioned in RSGPT[1].\n\nIt will also be important to showcase the performance w.r.t other performance measures such as recall etc\n\n[1] RSGPT: A Remote Sensing Vision Language Model and Benchmark\n\nIn addition, the architecture and concept are very similar to Video-ChatGPT which has never been discussed or compared. Also, there are RS-based VLM which target change detection (https://github.com/lzw-lzw/awesome-remote-sensing-vision-language-models) which will be important to compare.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FWv8Q5I0hj", "reviewer": "Reviewer_e7d5", "summary": "The paper introduces a instruction tuning dataset for temporal earth observation and a corresponding fine-tuned model trained on this data.", "questions": "Please address the weaknesses above.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "S1. Applications around geo-spatial data are important and have potential for great societal impact.\n\nS2. Applying VLMs to video data is a timely direction.\n\nS3. Additional evidence that state-of-the-art fine-tuning methods are broadly applicable to specific target domains.", "weaknesses": "W1. The introduction often reads as a related work section. Consider moving some of the contextualization of the field in the related work section and focus a bit more on the core story and contributions in the intro.\n\nW2. While there is a long history of geo-spatial data in computer vision, the paper contribution does strike me a little narrow in scope. \n\nW3. Methodological novelty. The paper applies a fairly straightforward fine-tuning recipe. While the paper does not oversell the method, this combined with the narrow scope (W2) do make me a bit concerned.\n\nW4. What are some broader applications of these agents? Is it possible to deploy these agents at scale to understand patterns in data that would be intractable for humans to otherwise deduce themselves?\n\nW5. The task presented seem like important capabilities but not particularly difficult for specialized models to solve. How do baseline classifiers, referring expression, change detection, etc. algorithms perform on the tasks. What generalization capabilities can be expected in the proposed model that are not present in these baselines?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "umaiELAgqx", "reviewer": "Reviewer_qA8u", "summary": "In this paper, the authors develop a novel vision and language assistant (TEOChat) to engage in conversations about temporal sequences of earth observation data. Their extensive experiments show that TEOChat can perform a variety of spatial and temporal reasoning tasks, achieving comparable or better performance than specialist models.", "questions": "My concerns are listed in the ''Weaknesses\".", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "+ The paper develops the first vision-language model (VLM) for temporal EO data and introduces the first temporal EO dataset for multimodal instruction-tuning, expanding the application scope for VLMs.\n\n+ The proposed method shows impressive zero-shot performance on several datasets, and outperforms foundation models (i.e., GPT-4o and Gemini-1.5 Pro) for modeling sequences of images.", "weaknesses": "- While contributing a novel application and a new dataset is encouraging, the technical contribution of this paper is insufficient.  This  paper primarily focuses on employing Low-Rank Adaptation (LoRA) to fine-tune a LLaVA-1.5-like architecture, which allows the model to handle both temporal and Earth Observation (EO) data. This paper could benefit from a more in-depth analysis of why such a design for the framework was chosen.\n\n-  Although the authors mention that their approach achieves impressive performance on various datasets (refer to Table 1), the analysis of these results lacks specificity. It is recommended to present relevant visualizations to demonstrate the hypothesis.\n\n- While the study provides a valuable analysis of the proposed approach against existing benchmarks, it would be enhanced by incorporating more recent specialist methods into the comparative analysis.\n\n- There are some typos in the paper. For example, Line 106: “To align the the EO imagery”, double “the”.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yr07eOowug", "reviewer": "Reviewer_3yvB", "summary": "The paper presents TEOchat, a vision-language model (VLM) explicitly designed for the temporal sequences of EO data. While existing temporal sequences models focus on natural images/videos and the EO VLMs focus mainly on single-images, TEOchat fills a gap by being the first to manage temporal EO data. Authors propose a new dataset called TEOchatlas with 554k examples of various single-image and temporal tasks instructions. TEOChat performs well in tasks such as change detection and temporal reasoning, outperforming both generalist and specialist VLMs designed for EO tasks.", "questions": "* A number of open VLMs now allow the processing of multiple images. Can the authors evaluate such options e.g., Qwen2VL, InternVL2 on temporal EO tasks explored in the paper?\n* Some recent parallel works (e.g., CDchat https://arxiv.org/html/2409.16261v1) have also explored extending EO VLMs for change detection. Authors are encouraged to explain differences with them (although this is an Arxiv work, and comparing them is not mandatory).\n* The VLM responses are textual, the metrics used for eval are accuracy, IoU and F1 measures. Its not clear how the textual outputs are parsed to obtain object coordinates, outputs classes etc. How robust is this process to avoid any errors due to incorrect parsing?\n* How much the choice of low-image resolution affect the performance on localization tasks like detection when the objects are quite small. Authors use all the GeoChat data for training, including referring and grounding instructions, but do not compare single-image grounding tasks. It will be good to understand how the choice of vision encoder affects the performance on these tasks. \n* Why GPT4 and Gemini models were evaluated/compared on only two tasks? Analysis on the kind of cases where these models fail will be important.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* A VLM tailored for temporal EO data addresses an important gap in current literature. Prior VLMs could only work with either natural video/multi-images or single EO images.\n* The dataset extends GeoChat datasets to the temporal domain. The temporal component of the TEOchatlas dataset is comprised of 4 datasets (fMoW, xBD, S2Looking, QFabric) that are converted to the following temporal tasks: Temporal Scene Classification, Change Detection, Spatial Change Referring Expression, Change QA, Region-based Change QA, Temporal Referring Expression and Region-based Temporal QA.\n* The model is comprehensively evaluated across multiple datasets and tasks. TEOChat consistently performs well on both generalist and specialist models on tasks such as change detection, temporal scene classification, and change question answering.  Its performance on zero-shot tasks is also significant. \n* Ablations done on the type of projector initialization, its finetuning startegy, and choice of image/video encoder are comprehensive and justify the design choices.", "weaknesses": "* Similar to video models like Video-LLaVa, the authors limit the sequences to a maximum of 8 images. For longer sequences, how this restriction can affect the model results?\n* TEOchat randomly sample 8 images without replacement from longer sequences, the choice of samples selected can significantly change the results. Can the authors report variance in results due to different random samplings?\n* The statement \"TEOChat also performs comparably to or outperforms a specialist model trained to perform specific tasks\" seems quite a strong claim given the proposed model performs quite low in certain cases: e.g., xBD Loc. (IoU) and QFabric [2 and 5 images].\n* Table 1, I believe the GeoChat model is evaluated zero-shot given it cannot be trained with multiple images? This should be clarified clearly in Table 1 that the comparison is between a zero-shot model evaluation (not built for multiple images) and TEOchat that is specifically trained with multiple images. Similarly, as Table 2 shows, the VideoLLava is also evaluated zero-shot in Table 1, and compared with the version that is finetuned on the TEOchatlas dataset, I do not think its a fair comparison and authors should clearly mention that they are comparing zero-shot models with a supervised model. \n* Contribution 2 gives the impression that the 554k instruction set is this paper's contribution while over 55% of it is included from previous work GeoChat. It would be reasonable to clarify this and change the claim accordingly. \n* The model uses interleaved image identifiers among the visual tokens in the sequence input to the LLM (Figure 3). This differs from prior methods for modeling temporal sequences of natural images which commonly concatenate the tokens and leads to significant gains as per ablation study. However, it is not clear why the image identifier would help in bi-temporal datasets like xBD and S2Looking, since tagging two images should not make much of a difference. Can the authors explain this?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)