TEOChat: A Large Vision-Language Assistant for Temporal Earth Observation Data

## Abstract

Abstract Large vision and language assistants have enabled new capabilities for interpreting natural images. These approaches have recently been adapted to earth observation data, but they are only able to handle single image inputs, limiting their use for many real-world tasks. In this work, we develop a new vision and language assistant called TEOChat that can engage in conversations about temporal sequences of earth observation data. To train TEOChat, we curate an instruction-following dataset composed of many single image and temporal tasks including building change and damage assessment, semantic change detection, and temporal scene classification. We show that TEOChat can perform a wide variety of spatial and temporal reasoning tasks, substantially outperforming previous vision and language assistants, and even achieving comparable or better performance than specialist models trained to perform these specific tasks. Furthermore, TEOChat achieves impressive zero-shot performance on a change detection and change question answering dataset, outperforms GPT-4o and Gemini 1.5 Pro on multiple temporal tasks, and exhibits stronger single image capabilities than a comparable single EO image instruction-following model. We publicly release our data, models, and code: https://github.com/ermongroup/TEOChat .

## 1 Introduction

Many earth observation (EO) tasks require the ability to reason over time. For example, change detection is a widely studied task where the goal is to identify salient changes in a region using multiple EO images capturing the region at different times (Chughtai et al., ; Bai et al., ; Cheng et al., ). Automated methods to effectively perform change detection can significantly aid humanitarian and sustainability efforts, including planning disaster relief (Rahnemoonfar et al., ; Gupta et al., ; Ban et al., ), tracking urban changes (Tamilenthi & Baskaran, ; Fyleris et al., ; Verma et al., ), and monitoring deforestation (Shermeyer & Haack, ; De Bem et al., ; Decuyper et al., ), crops, (Gim et al., ; Kaur et al., ; Wang et al., ), and ecological conditions (Willis, ; Xu et al., ). Previous methods to automatically detect change in EO imagery have been specialist models, constraining their use to a single task or small set of tasks that they were explicitly trained to perform (Bai et al., ; Cheng et al., ).

Advancements in the modeling of multimodal data have enabled generalist vision-language models (VLMs) that can perform a variety of natural image interpretation tasks specified flexibly through natural language
(Achiam et al., ; Team et al., ; Liu et al., ).
These models have recently gained capabilities to reason over temporal sequences of natural images (video) (Maaz et al., ; Lin et al., ; Jin et al., ; Li et al., ; Zhang et al., ; Liu et al., ) and single EO images (Hu et al., ; Kuckreja et al., ; Zhan et al., ; Muhtar et al., ; Pang et al., ).
However, no prior VLMs can model *temporal EO data* (left of Figure 1), notably including change detection tasks.
We investigate the performance of Video-LLaVA (Lin et al., ), a strong natural image pre-trained VLM that can receive images and videos as input, and GeoChat (Kuckreja et al., ), a strong VLM fine-tuned on *single* EO image tasks (right of Figure 1). We find that Video-LLaVA generates inaccurate information, likely because it has primarily been trained on natural images and videos, whereas GeoChat can only input single images and cannot process information across time.

Figure: Figure 1: Left: Capabilities of VLMs. TEOChat is the first VLM to model temporal earth observation (EO) data. Right: Example outputs of previous VLMs. We compare a temporal VLM (Video-LLaVA (Lin et al., )) and an EO VLM (GeoChat (Kuckreja et al., )) with TEOChat.
Refer to caption: extracted/5908013/images/Drawback_Figure.png

To address these limitations, we introduce TEOChat, the first large language and vision assistant that can handle temporal EO imagery.
Specifically, our contributions are:

- 1.
We develop the first vision-language model for temporal EO data called TEOChat. When prompted with temporal images and natural language instructions, TEOChat can perform EO tasks that require temporal reasoning, e.g. change detection. TEOChat outperforms a previous VLM for single EO images (GeoChat) and a VLM for temporal sequences of natural images (Video-LLaVA), and also rivals specialist models on multiple tasks.
- 2.
We introduce the first temporal EO dataset for multimodal instruction-tuning called TEOChatlas which we use to train TEOChat. TEOChatlas is composed of 554,071 examples spanning dozens of temporal instruction-following tasks.
- 3.
We demonstrate that (i) TEOChat achieves impressive zero-shot performance on an EO change detection and a change QA dataset, (ii) TEOChat outperforms two strong proprietary foundation models for modeling sequences of images (GPT-4o and Gemini-1.5 Pro), and (iii) TEOChat possesses strong single image capabilities, outperforming GeoChat on multiple zero-shot scene classification and visual question answering tasks.

## 2 Developing a VLM for Temporal EO Data

We aim to develop a VLM that can input a natural language instruction (prompt) and a temporal sequence of EO images, and output a natural language response. For example, given a sequence of satellite images capturing an urban area before and after a hurricane, the model should be able to respond to questions about damaged buildings or flooded regions. The vast majority of VLMs that can perform this task on natural images follow a LLaVA-like architecture (Liu et al., ) consisting of
(i) a large language model (LLM; commonly LLaMA 2 (Touvron et al., ) initialized with Vicuna weights (Chiang et al., ))
(ii) a vision encoder (commonly a ViT (Dosovitskiy et al., ) pre-trained with CLIP (Radford et al., )) to generate the visual representations, and
(iii) a vision-language connector (commonly an MLP (Liu et al., )) to align the visual representations with the natural language token embeddings input to the LLM.

Training the VLM requires aligning the language representations with the visual concepts in EO images, but these concepts are not well-represented in natural image data. EO imagery is taken from a birds-eye view and captures land use and land cover changes like flooding or road construction. To align the the EO imagery with natural language and allow the model to provide responses to instructions about the EO data, we require a dataset of triplets consisting of sample instructions, EO images, and ground truth responses, commonly referred to as an instruction-following dataset. The responses are used to supervise the VLM using a next-token cross-entropy loss on the response tokens (visual instruction tuning (Liu et al., )).
We note that existing instruction-following EO datasets are insufficient for training a temporal EO VLM, as the tasks do not require temporal capabilities and therefore do not promote their development. We confirm this empirically by showing that GeoChat substantially underperforms our approach across all temporal reasoning tasks we test.

## 3 TEOChatlas: A Temporal EO Instruction-Following Dataset

The main obstacle to developing a temporal EO VLM is the absence of a suitable instruction-following dataset.
To overcome this, we curate an instruction-following dataset of temporal EO imagery called TEOChatlas. A few key desired capabilities of the VLM induce important dataset design decisions. First, the model should be able to follow instructions about a wide variety of tasks requiring spatial and temporal reasoning capabilities, so we construct a diverse composition of instruction-following tasks (Section 3.1). Second, the model should be able to reason over variable-length temporal sequences of EO data from different sensors, so we ensure that many sensors and sequence lengths are represented (Section 3.2). Third, the model should be able to reference specific images in its input and output and support user-friendly conversation, so we design the dataset prompts to support these features (Section 3.3).

### 3.1 TEOChatlas: Task Composition

We curate various instruction-following tasks in TEOChatlas in order to develop a range of capabilities in the model. Importantly, because our goal is to add new temporal reasoning capabilities without compromising single image reasoning abilities, we include both single and temporal instruction-following tasks in the dataset, described in detail below.

#### 3.1.1 Single Image Tasks

To support the development of single image capabilities like object recognition and spatial reasoning, we include many single image instruction-following tasks in TEOChatlas. Specifically, we use the GeoChat dataset (Kuckreja et al., ), a composition of single image instruction-following tasks including scene classification, visual question answering, referring expression, region captioning, detailed description, grounded description, and multi-turn conversation. The GeoChat dataset was used to train the GeoChat model, which exhibits strong performance on these single image tasks. Appendix A has more detail about the GeoChat dataset and instruction-following tasks.

Figure: Figure 2: Examples of instruction-following tasks in TEOChatlas. We curate many instruction-following tasks for temporal EO data and group them into seven categories. The tasks require spatial and temporal reasoning capabilities, and span real-world applications including disaster relief and urban development monitoring.
Refer to caption: extracted/5908013/images/Task_Figure.png

#### 3.1.2 Temporal Tasks

We construct several temporal instruction-following tasks using four common temporal EO datasets. We group the tasks into seven categories spanning two common real-world applications, namely disaster response and urban development monitoring. Figure 2 shows examples of the temporal tasks and Table 8 in the Appendix provides the full set of tasks and associated prompts. TEOChatlas is the first instruction-following EO dataset to have temporal tasks (Table 6).

Temporal Scene Classification (TSC)  We construct an instruction-following task from a standard EO task called temporal scene classification, where the goal is to classify a sequence of satellite images into a set of predefined categories. For this task, we use the Functional Map of the World (fMoW) dataset (Christie et al., ), consisting of satellite image time series classified as one of 62 categories. To convert fMoW to an instruction-following task, we use a standard prompt which instructs the model to classify the image by selecting one of the 62 classes provided in the prompt.

Change Detection (CD)  We curate multiple instruction-following tasks for change detection, a widely studied earth vision task where the goal is to identify changes between images of the same region over time. We include two canonical change detection tasks in the dataset, namely building damage assessment and building change detection.

For building damage assessment, we use xBD (Gupta et al., ), a building damage detection dataset consisting of bitemporal pre- and post-disaster images, where every building is localized and classified into four damage severity categories. Following standard practice on this dataset, we divide this task into two subtasks, namely building localization (Loc.) and building damage classification (Dmg Cls.). Building localization prompts the model to predict bounding boxes around all buildings in the area, representing each box as a sequence of natural language tokens of the form $\left[x_{\text{min}},y_{\text{min}},x_{\text{max}},y_{\text{max}}\right]$ following Kuckreja et al. (). Building damage classification requires the model to classify each bounding box input in the instruction as one of the four damage categories provided in the prompt. These tasks together form a canonical semantic change detection task.

For building change detection (Det.), we use S2Looking (Shen et al., ), a building change detection dataset consisting of bitemporal, side-looking satellite images labeled with polygons indicating that a building has been constructed or demolished. We follow standard practice and task the model with outputting all changed buildings in the images. We use the same type of instruction as was used for the building localization task. This is a canonical binary change detection task.

Spatial Change Referring Expression (SRE)
To facilitate better spatial reasoning capabilities, we create instruction-following tasks to identify the locations of specific changes which are referred to by natural language expressions. Specifically, we create tasks from xBD where the model is instructed to identify the locations of only the buildings which experienced a specific damage type. We also construct tasks from S2Looking where the model is required to detect only the constructed buildings or only the demolished buildings. These temporal tasks are analogous to the referring expressions for single images used in the GeoChat dataset (Kuckreja et al., ).

Change Question Answering (QA)  To develop the model’s ability to recognize change and reply to a diverse array of user questions, we construct multiple visual question answering tasks about changes between temporal images. From xBD, we create tasks such as instructing the model to output the type of disaster from the provided labels in the dataset, the location in the region (a cell of a 3$\times$3 grid specified with top left, top center, etc.) most affected by the disaster, or a binary label indicating if there are any destroyed buildings in the area. Similarly, we add tasks from S2Looking to identify whether any buildings have been constructed or demolished in the region.

Region-based Change Question Answering (RQA)
We also create question answering tasks based on region prompt inputs to improve reasoning about spatial inputs. We generate the same tasks from xBD and S2Looking as used in change question answering, but constrain the question to a bounding box input representing a specific area within the image.

We also create tasks using QFabric (Verma et al., ), a dataset for multi-task urban change detection consisting of sequences of five images, where every sequence is labeled with several polygons representing urban change. Each polygon has a sequence-level label indicating the broad type of change (change type classification) and an image-level label indicating the specific temporal change that occurred between adjacent timesteps (change status classification). We construct instruction-following tasks for these two tasks by mirroring the building damage classification task, instructing the model to classify the change type or change status of an input bounding box into the label categories. Following the other classification tasks, we include the label categories in the prompt.

Temporal Referring Expression (TRE)  To enable the model to temporally localize change, we include multiple tasks instructing the model to identify the specific image(s) in which a given change occurred. We use the change type labels from QFabric which indicate the type of urban development that has occurred in each image in the temporal sequence. Because these labels are tied to specific regions of the images, we include the associated bounding box in the prompt and task the model with identifying the image(s) in the sequence where the change occurred. To perform this task, the model must be able to model spatial inputs and generate temporal outputs.

Region-based Temporal Question Answering (RTQA)
In order to support the development of sophisticated spatial and temporal reasoning capabilities together, the final tasks we curate for TEOChatlas instruct the model to answer questions about a given region, inputting or outputting references to images in the sequence. Specifically, we create questions about the presence of any urban development across two images within a box specified in the prompt and we generate tasks instructing the model to identify all images in the sequence where a given change type occurred.

### 3.2 TEOChatlas: EO Image Sequences

To allow the model to reason about varying input sequence lengths and sensor types, we ensure that TEOChatlas includes a diverse representation of image sequence sizes and image providers. We note that TEOChatlas also has considerable geographic diversity (Figure 5). We provide more detail about how we process the images of each dataset in Appendix A.

For sequence lengths, the temporal EO datasets we include consist of bitemporal (xBD and S2Looking), pentatemporal (QFabric), and multitemporal (variable length) sequences (fMoW). Due to memory constraints, we limit the sequences to a maximum of 8 images, and randomly sample 8 images without replacement from longer sequences. To increase the representation of other sequence lengths in the dataset, we also randomly sample sequences of shorter lengths from QFabric.

For sensor types, the EO datasets in TEOChatlas span 8 sensors (Table 7). To increase the representation of Sentinel-2, a publicly available sensor with global coverage that is commonly used for EO tasks, we include Sentinel-2 image sequences from fMoW-Sentinel (Cong et al., ).

### 3.3 TEOChatlas: Prompt Design

We design the prompts in TEOChatlas to enable the model to reference images and converse in a user-friendly manner. The temporal referring expression and region-based temporal question answering tasks require the model to input and output references to specific images in the sequence. To enable this, we interleave image identifiers among the visual tokens in the sequence input to the LLM (see Figure 3). This differs from prior methods for modeling temporal sequences of natural images, which simply concatenate the visual representations (Maaz et al., ; Lin et al., ; Zhang et al., ; Jin et al., ; Li et al., ; Touvron et al., ).

The original GeoChat dataset depends on task tokens to help the model differentiate between tasks, for example to determine whether to include bounding boxes in its response. We remove this requirement, and replace the tokens with short natural language specifications of the task. Examples of this are provided in Table 8 in the Appendix.

Finally, we modify the system prompt to (1) specify that the input contains a sequence of satellite images and (2) randomly include the image resolution (high or low) and sensor name in the prompt to allow TEOChat to leverage that information when provided (see Appendix).

Figure: Figure 3: Overview of TEOChat. TEOChat inputs a temporal sequence of EO images and a user instruction, and outputs a natural language response. It can input and output regions specified through bounding boxes, and can also input and output references to specific timesteps using image identifiers. We show one example from each task category, including temporal scene classification (peach), change detection (yellow), spatial change referring expression (green), change question answering (cyan), region-based change question answering (blue), temporal referring expression (purple), and region-based temporal question answering (pink).
Refer to caption: extracted/5908013/images/Architecture_Figure.png

## 4 TEOChat

### 4.1 TEOChat Architecture

We adopt a LLaVA-1.5-like architecture (Liu et al., ) consisting of (i) a temporally-shared image encoder (CLIP ViT-L/14 (Radford et al., )) to obtain representations of each image in the sequence, (ii) a 2-layer MLP to project the visual representations to the input of the LLM, and (iii) an LLM decoder (Llama 2 (Touvron et al., )) which inputs the instruction and temporal sequence of projected visual representations to generate the response (Figure 3).
A recent extension of LLaVA-1.5 to handle videos called Video-LLaVA (Lin et al., ) uses joint natural image-video instruction-tuning by using the frozen CLIP ViT image encoder for images and a frozen LanguageBind (CLIP-aligned) video encoder for videos (Zhu et al., ), while keeping the rest of the architecture the same as LLaVA-1.5. We test whether these projector and LLM weights learned on video data are better for temporal EO data by comparing Video-LLaVA initialization to LLaVA initialization, as well as freezing vs. fine-tuning the projector, in our experimental ablations.

Another key architectural decision is whether to use the image encoder or video encoder from Video-LLaVA (Lin et al., ) to generate visual representations for the sequence. We choose to use the image encoder for two reasons: (1) prior change detection approaches have demonstrated that Siamese encoders with weights shared across time (temporal-agnostic) are highly effective for identifying changes in sequences of EO data (Zheng et al., ; ; ), and (2) the video encoder was designed to input fixed sequences of 8 images. While it is possible to remove this second constraint, we observed performance degradation when using shorter sequences in preliminary experiments, likely because the video encoder was trained strictly on 8 image sequences (Zhu et al., ).
Replicating images in the sequence may address this but makes training and inference much slower.
Altogether, our vision encoder design is fast, memory efficient, and achieves strong performance.

### 4.2 TEOChat Training Details

Following prior work (Kuckreja et al., ), to retain the strong capabilities of the pre-trained image encoder and LLM, and to minimize memory usage during training, we freeze the vision encoder and projector weights then fine-tune the LLM using Low-Rank Adaptation (LoRA) (Hu et al., ). To further reduce memory footprint, we use 8-bit quantization of the base LLM weights. These strategies allow us to train the large multimodal model on temporal sequences of up to 8 images using an NVIDIA A4000 GPU (16GB VRAM). We provide additional training details in the Appendix.

## 5 Experimental Results

**Table 1: Temporal task performance of TEOChat compared to other VLMs and specialist approaches. We bold the best generalist result on each task. TEOChat outperforms a VLM instruction-tuned on natural images and videos (Video-LLaVA (Lin et al., )) as well as a VLM instruction-tuned on single EO images (GeoChat (Kuckreja et al., )). TEOChat also performs comparably to or outperforms a specialist model trained to perform specific tasks (fMoW (Cong et al., ), xBD (Gupta et al., ), S2Looking (Shen et al., ), QFabric (Verma et al., )). TEOChat-T was only trained on temporal tasks.**
| Task | Dataset/Subtask | Specialist | Video-LLaVA | GeoChat | TEOChat-T | TEOChat |
| --- | --- | --- | --- | --- | --- | --- |
|  | fMoW RGB (Acc.) | 65.9 | 16.6 | 59.2 | 73.5 | 75.1 |
| TSC | fMoW Sentinel (Acc.) | - | 4.9 | 26.3 | 45.5 | 45.5 |
|  | xBD Loc. (IoU) | 66.0 | 3.6 | 7.4 | 28.8 | 35.5 |
|  | xBD Dmg Cls. (F1) | 26.5 | 8.3 | 11.8 | 49.4 | 50.0 |
| CD | S2Looking Det. (F1) | 26.5 | 6.2 | 7.7 | 29.0 | 33.6 |
|  | xBD (F1) | - | 3.6 | 8.7 | 17.1 | 23.2 |
| SRE | S2Looking (F1) | - | 1.7 | 4.5 | 30.0 | 32.3 |
|  | xBD (Acc.) | - | 30.8 | 34.0 | 89.8 | 89.9 |
| QA | S2Looking (Acc.) | - | 41.5 | 51.5 | 75.4 | 73.4 |
|  | xBD (Acc.) | - | 57.3 | 63.6 | 93.7 | 94.0 |
|  | S2Looking (Acc.) | - | 47.8 | 67.2 | 86.7 | 90.0 |
|  | QFabric [2 images] (F1) | 77.0 | 22.6 | 20.9 | 63.9 | 66.7 |
| RQA | QFabric [5 images] (F1) | 81.6 | 26.5 | 21.9 | 72.8 | 74.3 |
| TRE | QFabric (Acc.) | - | 1.9 | - | 73.1 | 74.9 |
|  | QFabric (Acc.) | - | 26.5 | - | 71.4 | 71.7 |
| RTQA | QFabric [5 images] (F1) | 54.5 | 10.5 | 12.5 | 65.2 | 66.4 |

We evaluate TEOChat in a variety of settings.
First, we find that TEOChat demonstrates impressive temporal reasoning capabilities, outperforming two strong VLMs, one for videos (Video-LLaVA) and one for single EO images (GeoChat), and even rivals or outperforms specialist approaches (Section 5.1).
Second, we test model design decisions in experimental ablations, and find our design of TEOChat is superior to many others (Section 5.2).
Third, we demonstrate that TEOChat achieves impressive zero-shot generalization to new temporal datasets not in TEOChatlas  (Section 5.3). Fourth, we find that TEOChat outperforms two strong proprietary foundation models trained to model sequences of images (Section 5.4). Fifth, we observe that TEOChat possesses strong single image capabilities, outperforming GeoChat on average across zero-shot scene classification and visual question answering (Section 5.5).

### 5.1 Comparison to other VLMs and specialist approaches

Baselines and Evaluation Metrics
We compare TEOChat to Video-LLaVA (Lin et al., ), GeoChat (Kuckreja et al., ), and specialist models (Gupta et al., ; Shen et al., ; Verma et al., ). To obtain GeoChat’s predictions on the temporal tasks, we perform temporal post-processing of its single image predictions. For example, for building change detection, we obtain building location predictions in each image separately, then compute a per-pixel difference to identify the changed buildings over time. We describe all temporal post-processing strategies in the Appendix. For the specialist models, we use a strong self-supervised EO model adapted to fMoW (SatMAE (Cong et al., )) and strong models trained on each individual dataset for the change detection tasks (a modified UNet (Ronneberger et al., ) from Gupta et al. (), FC-Siam-Diff (Daudt et al., ) from Shen et al. (), and a modified UNet from Verma et al. ()).

We use task-specific evaluation metrics. For previously existing tasks, we follow the metrics reported in the original works. For new tasks, we use accuracy. See Appendix for a full description of metrics, and see Figure 4 for qualitative examples of TEOChat responses on the temporal tasks.

Temporal Scene Classification
TEOChat achieves impressive performance on TSC with the fMoW RGB (75.1%) and Sentinel (45.5%) validation sets, outperforming both Video-LLaVA (16.6%, 4.9%) and GeoChat (59.2%, 26.3%) (Table 1). We also compare TEOChat to linear probing a specialist encoder that was pre-trained with self-supervision on fMoW images (SatMAE). TEOChat obtains considerably higher accuracy than SatMAE on fMoW RGB (+9.2%), and emphasize the TEOChat is a generalist which is not pre-trained on EO imagery. We also note that, impressively, TEOChat performs comparably to SatMAE+Stack fully fine-tuned on fMoW RGB (-0.8%).

Change Detection and Spatial Referring Expression  TEOChat outperforms Video-LLaVA and GeoChat by a substantial margin on all CD tasks (Table 1). However, on xBD building localization, the model underperforms the specialist approach by a large margin. Qualitatively, the model does moderately well at identifying the building locations but struggles to precisely localize their boundaries. This may be partially due to representing object locations using natural language tokens, and partially due to using axis-aligned boxes rather than general polygons. TEOChat’s change detection performance on S2Looking is higher than the specialist model’s performance (+7.1 F1). On building damage classification, TEOChat outperforms the specialist substantially (+23.5 F1). On SRE, TEOChat achieves much higher performance than both Video-LLaVA (+19.6 F1 on xBD, +30.6 F1 on S2Looking) and GeoChat (+14.5 F1 on xBD, +27.8 F1 on S2Looking).

**Table 2: Model ablations testing the impact of different design decisions in TEOChat. Our design (bottom row) outperforms all other tested designs on the five canonical temporal tasks.**
| LLM/Projector | Image Encoder | Projector | Image | Fine-tuned on | Train | fMoW | xBD | S2Looking | QFabric | QFabric |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Initialization | Initialization | References | TEOChatlas | Iters | RGB | Dmg Cls. | Det. | RQA-2 | RQA-5 |  |
| Video-LLaVA | CLIP | Frozen | - | - | 7k | 16.6 | 8.3 | 6.2 | 22.6 | 26.5 |
| Video-LLaVA | CLIP | Frozen | - | ✓ | 7k | 63.2 | 22.2 | 9.9 | 59.0 | 65.7 |
| LLaVA | CLIP | Frozen | ✓ | ✓ | 7k | 66.8 | 8.9 | 4.8 | 58.7 | 64.6 |
| Video-LLaVA | SkyScript | Frozen | ✓ | ✓ | 7k | 55.6 | 3.7 | 4.0 | 29.9 | 50.0 |
| Video-LLaVA | SkyScript | Fine-tuned | ✓ | ✓ | 7k | 68.3 | 40.8 | 12.5 | 59.0 | 66.6 |
| Video-LLaVA | CLIP | Frozen | ✓ | ✓ | 7k | 69.2 | 40.6 | 18.1 | 61.4 | 67.9 |
| Video-LLaVA | CLIP | Fine-tuned | ✓ | ✓ | 7k | 71.7 | 47.2 | 22.7 | 61.7 | 69.2 |
| Video-LLaVA | CLIP | Frozen | ✓ | ✓ | 14k | 71.5 | 45.8 | 30.4 | 66.6 | 74.3 |
| Video-LLaVA | CLIP | Fine-tuned | ✓ | ✓ | 14k | 75.1 | 50.0 | 33.6 | 66.7 | 74.3 |

Change Question Answering and Region-Based Change Question Answering
On QA and RQA, TEOChat achieves much higher performance than the baseline VLMs on all tasks (Table 1). On QA, TEOChat achieves 89.9% and 73.4% accuracy on xBD and S2Looking respectively, compared to GeoChat which attains 34.0% and 51.5% on the respective datasets. Similarly, on RQA, TEOChat outperforms GeoChat by 30.4% accuracy on xBD and 22.8% on S2Looking, and drastically outperforms on both QFabric tasks. TEOChat underperforms both specialist models on QFabric but achieves a much more similar F1 to the specialists compared to both GeoChat and Video-LLaVA.

Temporal Referring Expression and Region-based Temporal Question Answering  We find TEOChat obtains much higher accuracy than Video-LLaVA on TRE (74.9% vs. 1.9%) and RTQA (71.7% vs. 26.5%). On 5 image RTQA in QFabric, TEOChat outperforms the specialist by 11.9 F1.

### 5.2 Model ablations

Temporal task vs. joint single-temporal training.
We investigate the effect of joint single image and temporal task training by training a model on only the temporal tasks which we call TEOChat-T. TEOChat-T underperforms TEOChat on all but two tasks, matching TEOChat on fMoW Sentinel and slightly outperforming it on S2Looking QA (Table 1). This suggests that joint training on single image tasks also largely benefits temporal reasoning capabilities.

Model design
We test various design decisions of TEOChat by comparing their performance on the canonical temporal tasks in TEOChatlas, namely fMoW RGB, xBD damage classification, S2Looking change detection, and QFabric RQA with 2 and 5 images. Specifically, we test the impact of using a LLaVA initialization instead of Video-LLaVA, using an image encoder initialization from strong remote sensing pre-trained weights (SkyScript (Wang et al., )) instead of CLIP, fine-tuning versus freezing the projector layer, and including image references in the prompt. We note that we couple the LLM and projector initialization as they were pretrained together. Furthermore, to fairly compare to a new image encoder initialization, we also test the impact of using a fine-tuned projector with the alternate initialization to allow it to adapt to the new image encoder outputs. Finally, we measure the performance improvement from training on TEOChatlas (without using image references) and the effect of training for longer (7k iters [1 epoch] vs 14k [2 epochs]).

Our design of TEOChat outperforms all tested designs on the five canonical temporal tasks (Table 2). Fine-tuning on TEOChatlas leads to large performance improvements across the tasks and including image references leads to further improvements. We further measure the impact of including image identifiers in the test-time prompt input to TEOChat for temporal referring tasks (Table 9), and find that the identifiers lead to substantial performance improvements on both tasks. Using LLaVA weights for the LLM and projector initialization underperforms VideoLLaVA across all tasks which we attribute to the fact that Video-LLaVA is a stronger model and was trained with natural video data. The use of remote sensing pre-trained image encoder also underperforms a CLIP-pretrained image encoder even when allowing projector fine-tuning, which we believe could be partially because the LLM weights were pre-trained with a frozen CLIP-initialized image encoder during multi-modal instruction-tuning on natural video data. We also find that fine-tuning outperforms freezing the projector, and training the model for longer leads to further performance improvements on all tasks, with especially large improvements on fMoW RGB and xBD damage classification.

### 5.3 Zero-shot performance on change detection tasks

We assess TEOChat’s performance on two new temporal remote sensing datasets, namely change detection using ABCD (Fujita et al., ) and change visual question answering using CDVQA (Yuan et al., ). TEOChat demonstrates impressive generalization to thse new datasets, outperforming Video-LLaVA (+35.6 on ABCD, +17.4 on CDVQA) and approaching specialist performance.

### 5.4 Comparison to proprietary foundation models

We compare TEOChat to GPT-4o and Gemini 1.5 Pro with three demonstrating examples on xBD damage classification and S2Looking change detection (Table 4). TEOChat performs better than both proprietary foundation models with in-context learning on xBD damage classification (+11.7 F1 compared to GPT-4o, +12.8 F1 compared to Gemini 1.5 Pro) and S2Looking change detection (+14.2 F1 compared to GPT-4o, +17.1 F1 compared to Gemini 1.5 Pro). We could not evaluate the models on fMoW or QFabric (which are larger both in size and sequence length) due to cost.

**Table 3: Zero-shot results on ABCD (building CD) and CDVQA (change QA).**
| Model | ABCD | CDVQA |
| --- | --- | --- |
| Specialist | 95.2 | 63.4 |
| Video-LLaVA | 50.0 | 29.8 |
| GeoChat | - | - |
| TEOChat | 85.6 | 47.2 |

**Table 5: Single image accuracies for scene classification (AID, UCMerced) and visual question answering (LRBEN, HRBEN). TEOChat has strong single image capabilities, outperforming both VLMs on average.**
|  |  |  | LRBEN | HRBEN |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Model | AID | UCMerced | Pres. | Comp. | Rural/Urban | Pres. | Comp. | Average |
| Video-LLaVA (Lin et al., ) | 52.4 | 46.5 | 55.8 | 65.1 | 61.0 | 64.5 | 66.8 | 58.5 |
| GeoChat (Kuckreja et al., ) | 72.0 | 84.4 | 91.1 | 90.3 | 94.0 | 58.5 | 83.2 | 79.9 |
| TEOChat (ours) | 80.9 | 86.3 | 91.7 | 92.7 | 94.0 | 67.5 | 81.1 | 83.4 |

### 5.5 Single image performance

We measure TEOChat’s single EO image capabilities by evaluating it on (1) scene classification tasks (AID (Xia et al., ) and UCMerced (Yang & Newsam, )) and (2) visual question answering tasks (LRBEN and HRBEN (Lobry et al., )). We note that evaluation on AID, UCMerced, and HRBEN is zero-shot, as they are not in TEOChat’s training set.

TEOChat achieves better performance than both models on all but one single image task (Table 5). It outperforms GeoChat on both LULC classification tasks, notably by a large margin on AID (+8.9), a much larger-scale dataset than UCMerced. TEOChat also outperforms GeoChat on the low resolution VQA tasks and the high resolution presence task, while slightly underperforming on the high resolution comparison task. Notably, GeoChat uses higher resolution images than TEOChat (504 vs. 224). TEOChat attains the highest average performance by a considerable margin (+3.5), suggesting it has stronger single image capabilities on top of the new temporal reasoning capabilities. We provide examples of TEOChat single image responses in Figure 10.

Figure: Figure 4: Qualitative examples of responses produced by TEOChat on temporal tasks.
Refer to caption: extracted/5908013/images/Qualitative_Temporal_Figure.png

## 6 Conclusion

In this work, we propose TEOChat, the first vision and language assistant that can engage in conversation about temporal EO imagery. We introduce a novel dataset called TEOChatlas to train TEOChat on a diverse set of spatial and temporal instruction-following tasks. Our experiments suggest that TEOChat exhibits strong performance, outperforming previous VLMs substantially, and even competing with or outperforming specialist models. TEOChat also demonstrates impressive zero-shot generalization to two change detection datasets and outperforms strong proprietary models which can handle sequences of images.

There are several interesting directions for future work. First, it would be interesting to explore other architectures to process the visual data across time. The LLM is responsible for performing the temporal information aggregation in TEOChat, but it may be helpful to use a dedicated part of the architecture for this. Second, it would be useful to enable the model to process multi-spectral bands which is common in EO data. Third, exploring methods to improve object localization is worthwhile, including regressing the coordinates instead of representing them with natural language tokens (Zhang et al., ), using oriented bounding boxes to allow for tighter localization, or increasing memory efficiency of the architecture to enable inputting higher resolution images.

## Appendix

**Table 6: Comparison of TEOChatlas to other instruction-following EO datasets. TEOChatlas has a large number of instruction-following examples, and is the first EO dataset to have temporal tasks.**
| Dataset | Size | Temporal | Detection |
| --- | --- | --- | --- |
| GeoChat_Instruct (Kuckreja et al., ) | 309k | - | ✓ |
| SkyEyeGPT (Zhan et al., ) | 968k | - | ✓ |
| MMRS1M (Zhang et al., ) | 1M | - | ✓ |
| LHRS-Instruct (Muhtar et al., ) | 42k | - | ✓ |
| ChatEarthNet (Yuan et al., ) | 173k | - | - |
| H2RSVLM-Instruct (Pang et al., ) | 180k | - | ✓ |
| SkySenseGPT (Luo et al., ) | 1.4M | - | ✓ |
| RS-GPT4V (Xu et al., ) | 991k | - | ✓ |
| TEOChatlas (Ours) | 554k | ✓ | ✓ |

**Table 7: Datasets included in the TEOChatlas training set. TEOChatlas is composed of single image and temporal image datasets from a variety of sensors and includes multiple instruction-following tasks for both spatial and temporal reasoning.**
| Dataset | Temporal | Task | Sensor | # Examples |
| --- | --- | --- | --- | --- |
| GeoChat | Single | Object Detection, VQA | GaoFen, Jilin-1, Sentinel-2, | 308,861 |
| Scene Classification | DJI Mavic Pro |  |  |  |
| fMoW | Multitemporal | Sequence Classification | WorldView-2/3, Sentinel-2 | 83,412 |
| xBD | Bitemporal | Building Damage Detection | WorldView-2 | 19,749 |
| S2Looking | Bitemporal | Building Change Detection | GaoFen, SuperView, BeiJing-2 | 17,090 |
| QFabric | Multitemporal | Multi-Task Urban | WorldView-2 | 124,959 |
| Change Detection |  |  |  |  |

### A Training Datasets

TEOChatlas consists of GeoChat (Kuckreja et al., ), an instruction-following dataset with single earth observation images, as well as four temporal earth observation benchmarks (fMoW (Christie et al., ), xBD (Gupta et al., ), S2Looking (Shen et al., ), and QFabric (Verma et al., )). We describe how we process each dataset in more detail below and a summary of the dataset statistics is provided in Table 7. We note that for all datasets, we (1) resize the shorter side to 224 then crop to 224x224 before inputting the images into the network and (2) randomly sample between all instruction-following tasks when creating examples, sometimes randomly sampling between different phrasings of the same task to introduce diversity in the wording. A spatial map of the locations of the examples in TEOChatlas is shown in Figure 5 and a comparison of TEOChatlas to other instruction-following EO datasets is shown in Table 6.

GeoChat  The GeoChat dataset (Kuckreja et al., ) is an instruction-following dataset composed of multiple single image EO datasets spanning tasks including object detection derived from SAMRS (which is itself a composition of DOTA (Xia et al., ), DIOR (Li et al., ), and FAIR1M (Sun et al., )), visual question answering from LRBEN (Lobry et al., ) and Floodnet (Rahnemoonfar et al., ), and scene classification from NWPU-RESISC-45 (Cheng et al., ). The instruction-following tasks in the GeoChat dataset include scene classification (classify the image into one of several classes provided in the prompt), visual question answering (answer a question about the image), referring expression (identify the location of an object given an expression describing its characteristics), region captioning (describe a location represented as a bounding box in the prompt), detailed description (describing the whole image in detail), grounded description (describing the whole image in detail with bounding boxes included in the description), and multi-round conversation (multiple turns of conversation). In total, the GeoChat training dataset consists of 308,861 single instruction-following examples on 106,747 images.

fMoW  We use both the fMoW RGB dataset (Christie et al., ) and the fMoW Sentinel dataset (Cong et al., ), sampling randomly between the two for each example in the provided training set split. We follow standard practice for converting fMoW to a classification dataset and crop each image to the provided bounding box. We create one instruction-following example for every image in the training set, leading to 83,412 examples on 83,412 images. We report model performance on both the fMoW RGB and fMoW Sentinel validation sets.

xBD  We include the xBD (Gupta et al., ) training dataset in TEOChatlas. To both increase the representation of building change detection in TEOChatlas and maintain image resolution, we crop the 1024x1024 images into 16 256x256 images. When creating instruction-following tasks, we convert all building polygons to their minimal axis-aligned bounding boxes. We create one instruction-following example for every image in the training set, leading to 19,749 examples on 19,749 images. We use the original polygons when creating the segmentation masks for evaluation (see below), and report model performance on the test set.

S2Looking  We include the S2Looking (Shen et al., ) training dataset in TEOChatlas. Similarly to xBD, we crop the 1024x1024 images into 16 256x256 images and convert all building polygons to their minimal axis-aligned bounding boxes. We create one instruction-following example for every image in the training set, leading to 17,090 examples on 17,090 images. We use the original polygons when creating the segmentation masks for evaluation (see below), and report model performance on the test set.

QFabric  We include the QFabric (Verma et al., ) training dataset in TEOChatlas. The images in QFabric cover a large area with  10,000 x 10,000 pixels, which we crop into around 1600 256x256 images. We drop any cropped images that have no polygons and convert all polygons to minimal axis-aligned bounding boxes. We create one instruction-following example for every image in the training set, leading to 124,959 examples on 124,959 images. We use the original polygons when creating the segmentation masks for evaluation (see below), and report performance on the test set.

Figure: Figure 5: Locations of the examples in the xBD and fMoW RGB, and fMoW Sentinel subsets of the TEOChatlas dataset. GeoChat, S2Looking, and QFabric do not have geographic locations in the released data. However, the datasets in GeoChat span many locations around the world, and both S2Looking and QFabric contain globally distributed locations as well.
Refer to caption: extracted/5908013/images/Spatial_Map.png

### B Temporal Instruction-Following Tasks

We provide a summary of all temporal instruction-following tasks included in the TEOChatlas training set in Table 8. For description and captioning temporal tasks, we use GPT-4o to increase the diversity of the responses to avoid template-fitting which we found plagued the model in early experiments.

**Table 8: Examples of the many temporal instruction-following tasks in TEOChatlas.**
| Dataset | Task Category | Example |
| --- | --- | --- |
| fMoW | Sequence Classification | User: What class does this sequence of images belong to? |
| AI: Lake or pond. |  |  |
| xBD | Localization | User: Identify all the buildings in the image. |
| AI: <bbox>, <bbox>, <bbox>, <bbox>, <bbox>. |  |  |
| Classification | User: Classify the level of damage experienced by the building at location <bbox>in the second image. Choose from: No damage, Minor Damage, Major Damage, Destroyed. |  |
| AI: Minor Damage. |  |  |
| Change Question Answering | User: What disaster has occurred here? |  |
| AI: An earthquake. |  |  |
| User: Which part of the image was most affected by the disaster? |  |  |
| AI: The center left of the image was most affected by the disaster. |  |  |
| User: How many buildings in the image have been destroyed? |  |  |
| AI: 3. |  |  |
| User: Are there any damaged buildings in the image? |  |  |
| AI: Yes. |  |  |
| Region-based Question Answering | User: Are there any damaged buildings in this region <bbox>? |  |
| AI: Yes. |  |  |
| User: How severe is the damage to this building <bbox>? |  |  |
| AI: The given building has been destroyed. |  |  |
| Spatial Change Referring Expression | User: Identify the severely damaged or destroyed buildings in the image. |  |
| AI: <bbox>, <bbox>. |  |  |
| User: Identify the damaged buildings in the ¡section¿ of the image. |  |  |
| AI: There are no damaged buildings in the bottom right of the image. |  |  |
| User: Identify the destroyed buildings in this area <bbox>. |  |  |
| AI: There are no destroyed buildings in the given area. |  |  |
| Region Captioning | User: How has this building $<$bbox$>$ changed? |  |
| AI: The given building has been destroyed. |  |  |
| Detailed Description | User: Describe how the buildings have changed. |  |
| AI: There has been a flood that has damaged multiple buildings in the area. |  |  |
| Grounded Description | User: Describe how the buildings have changed. Include bounding boxes. |  |
| AI: There has been a flood that has damaged a building at the location <bbox>. |  |  |
| S2Looking | Localization | User: Identify all changed buildings. |
| AI: <bbox>, <bbox>, <bbox> |  |  |
| Change Question Answering | User: Have any buildings been constructed in the area? Please answer with Yes or No. |  |
| AI: Yes. |  |  |
| User: How many buildings in the image have been built or destroyed? |  |  |
| AI: 3. |  |  |
| Region-based Temporal Question Answering | User: Has the area <bbox>changed? Please answer with Yes or No. |  |
| AI: Yes. |  |  |
| Spatial Change Referring Expression | User: Identify the {constructed, destructed} buildings in the image. |  |
| AI: <bbox>, <bbox>, <bbox> |  |  |
| User: What is the largest building that experienced a change? |  |  |
| AI: <bbox>. |  |  |
| Detailed Description | User: Provide a detailed description of the buildings that have changed. |  |
| AI: Five buildings have been constructed. |  |  |
| Grounded Description | User: Provide a detailed description of the buildings that have changed. Include bounding boxes in your output. |  |
| AI: Three buildings have been destroyed <bbox>, <bbox>, <bbox>. |  |  |
| Region Captioning | User: Describe how the buildings have changed in this area: <bbox>. |  |
| AI: Five buildings have been constructed. |  |  |
| QFabric | Temporal Referring Expression | User: Identify all images in which land was cleared in this region <bbox> from the previous image. |
| AI: Image 1, Image 2 |  |  |
| User: In which images is <class> visible in this region <bbox>? |  |  |
| AI: Image 1, Image 2, Image 3 |  |  |
| User: Identify all images in which construction occurred in this region <bbox> from the previous image. |  |  |
| AI: Image 4 |  |  |
| User: Identify all images in which a construction project was begun in this region, area <bbox> from the previous image. |  |  |
| AI: Image 3 |  |  |
| User: Identify the image when <change phrase> in this region <bbox>. |  |  |
| AI: Image 2 |  |  |
| Region-Based Question Answering | User: Has there been urban development in this area <bbox>? |  |
| AI: Yes |  |  |
| User: Identify the type of urban development that has occurred in this area <bbox>. |  |  |
| AI: Residential |  |  |
| Temporal Region-Based Question Answering | User: Has there been urban development in this area <bbox> between image N and image M? |  |
| AI: No |  |  |
| User: What is the development status in this region <bbox> in image N? |  |  |
| AI: Finished Construction |  |  |
| Region Captioning | User: How has the area <bbox> changed as a result of urban development? |  |
| AI: This region was greenland at first, and then became finished construction. |  |  |
| Temporal Region Captioning | User: How has the area <bbox> changed due to urban development from image N to image M? |  |
| AI: Between the first and fourth images, this area was cleared land at first, and then became midway-finished construction and then became finished construction. |  |  |

### C Single Image Datasets

We evaluate TEOChat on the following datasets:

- 1.
AID (Xia et al., ), a scene classification dataset composed of 10,000 images each classified into one of 30 scene classes.
- 2.
UCMerced (Yang & Newsam, ), a land use classification dataset consisting of 2100 images each classified into one of 21 land use classes.
- 3.
LRBEN and HRBEN (Lobry et al., ), low resolution and high resolution datasets for visual question answering. We consider the presence task (answering yes or no for whether an object is in the image) and comparison tasks (answering yes or no to a comparison question, for example are there more buildings than roads in the image).

### D System Prompt

We use the following prompt for the tasks in TEOChatlas.

[
backgroundcolor=gray!10, linecolor=white, skipabove=10pt,
skipbelow=10pt,
innerleftmargin=10pt,
innerrightmargin=10pt,
innertopmargin=10pt,
innerbottommargin=10pt
]lightgreybox

A chat between a curious user and an artificial intelligence assistant.
The assistant gives helpful, detailed, and polite answers to the user’s
questions. USER: This is a sequence of {high resolution, optical}
satellite images {from ¡sensor¿}: ¡video¿.

where we randomly inject the information in curly brackets when metadata (resolution or sensor) is available and replace the “¡video¿” token with the “¡image¿” token $N$ times, with $N$ the number of images in the sequence. After this prompt, we add the task instruction then “ASSISTANT:”.

### E Additional Training Details

We fine-tune the LLM in TEOChat using LoRA rank 128. Before inputting the images into the image encoder, we resize the shorter dimension of each image to 224 pixels, and then apply a center crop to obtain a 224x224 image. These design decisions allow us to train the model with sequences of up to 8 images on a single NVIDIA A4000 GPU (16B of VRAM) with a batch size of 1. To increase the effective batch size, we use 8 steps of gradient accumulation and train the network on 10 GPUs using data parallelism together with optimizer state partitioning, gradient partitioning, parameter partitioning, and CPU offloading with DeepSpeed Zero-3 Offload (Rasley et al., ).
We optimize the network using AdamW (Loshchilov & Hutter, ) with a cosine learning rate scheduler, a peak learning rate of 2e-5, and a warmup of 3% of the epoch. The model takes 125 hours to train per epoch and we train for 2 epochs.

### F GeoChat Temporal Post-processing

We describe how we obtain GeoChat predictions on the temporal prediction tasks below.

#### F.1 Temporal Scene Classification

We run GeoChat on each image in the sequence to produce a per-image classification, then take the majority vote across the sequence as GeoChat’s prediction over the whole sequence.

#### F.2 Change Detection

xBD Building Localization  We prompt GeoChat to identify all buildings in the first image then use the output bounding boxes to obtain the predicted segmentation mask.

xBD Damage Classification  We input each ground truth box and the second image into GeoChat as separate examples, instructing it to classify the given box into the four damage categories, and use the classification labels output by GeoChat with the boxes to construct the segmentation mask.

S2Looking Building Change Detection  We prompt GeoChat to identify all buildings in first and second images individually. From the bounding box predictions on each image, we create binary segmentation masks then take the difference to obtain GeoChat’s predicted change mask, masking out pixels from overlapping boxes across the two images to avoid small deviations in box predictions between the two images from harming performance.

#### F.3 Spatial Referring Expression

xBD  We use the same procedure as for building localization, but prompt GeoChat to identify only the ‘destroyed‘ buildings, for example.

S2Looking  The spatial referring expression tasks for S2Looking are to identify the constructed or destructed buildings. For both, we prompt GeoChat to identify all buildings in the first image and in the second image, construct a segmentation mask for each, then take a difference. We use the pixels in the first image but not the second as destructed and the pixels in the second but not in the first as constructed.

#### F.4 Change Question Answering

xBD
As the xBD questions pertain to building damage levels and there is no damage to buildings in the pre-disaster images, we input the second image and the question (e.g. are there any severely damaged or destroyed buildings in this area?) to GeoChat to obtain the answer.

S2Looking  The change questions for S2Looking ask the model to identify whether or not buildings changed between the images. To obtain this binary prediction from GeoChat, we follow a similar approach and instruct it to identify the buildings in the first image and the buildings in the second, then consider the predicted building from each image as changed if it does not overlap a box in the other image. If any buildings changed, the binary prediction is yes, otherwise it is no.

#### F.5 Region-based Change Question Answering

xBD  As the input bounding box is the only difference between this task and change question answering, we use similar postprocessing as that task, inputting the second image and a question as well as a bounding box to GeoChat to obtain the answer.

S2Looking  Similarly, this task only differs from change question answering due to the bounding box input, so we perform the same postprocessing as for that task, but only taking into account the changes inside the input bounding box.

QFabric  These tasks are standard classification tasks, so we follow a similar procedure as for temporal scene classification and feed each image in the sequence (either 2 or 5 images) along with the bounding box in the prompt into GeoChat, then take a majority vote across the images to get the final classification.

### G Evaluation Setup

Change Detection and Spatial Referring Expression  To evaluate the performance of TEOChat on building change detection (CD) and spatial referring expression (SRE), we follow the standard performance metrics of the building localization (Loc.) and damage classification (Dmg Cls.) subtasks on xBD and building change detection (Det.) subtask on S2Looking. Specifically, for building localization, we convert the bounding box predictions and ground truth polygons to segmentation masks and compute IoU between the predicted and ground truth segmentation masks. For building change detection, we do the same thing but compute per-pixel F1 between the masks. For building damage classification, we input the ground truth polygon to the model to obtain a classification label, then create a segmentation mask where the pixels within the ground truth polygon are assigned the predicted label, and compute class-weighted F1 between this mask and the ground truth mask. Finally, for SRE, we use the same evaluation scheme as building change detection.

Change Question Answering and Region-Based Change Question Answering
We measure accuracy on change question answering (QA) and region-based change question answering (RQA) for xBD and S2Looking. For QFabric, we use the proposed evaluation metrics from Verma et al. () for comparability to the specialists, namely per-pixel F1 after converting the classified polygons to segmentation masks as also done in xBD damage classification. We consider both a 2 image task and 5 image task as done in Verma et al. ().

Temporal Referring Expression and Region-based Temporal Question Answering
For temporal referring expression (TRE), we measure performance of the models to identify the correct image(s) using accuracy. For region-based temporal question answering (RTQA), we use F1 for the change status task (classifying the development status of each image in a sequence of 5 images) following Verma et al. (), and use accuracy for the other tasks.

### H Image Identifier Ablation

We run an ablation to measure the importance of including the image identifiers in the prompt on the temporal referring tasks (Table 9).

**Table 9: Impact of including image identifiers like ‘Image 1: ¡image¿ Image 2: ¡image¿‘ in the test-time prompt on TEOChat performance on the region-based temporal question answering and temporal referring expression tasks. Including image references leads to substantial benefits on both tasks.**
| Image Identifiers | RTQA QFabric (Acc.) | TRE QFabric (Acc.) |
| --- | --- | --- |
| - | 59.2 | 58.6 |
| ✓ | 71.7 | 74.9 |

### I Single Image Qualitative Results

We provide examples of responses output by TEOChat on the single image tasks (Figure 10).

Table: Table 10: Examples of responses produced by TEOChat on single image tasks.

### J Broader Impacts

Automated methods for processing temporal EO data have the potential to lead to significant, positive societal impacts, from supporting disaster relief to monitoring deforestation. We show that TEOChat is capable of temporal reasoning, achieving strong performance on several real world tasks like building damage detection. We hope our work can inspire more work on developing vision and language assistants for important temporal EO tasks and lead to helpful new technologies that can benefit society.

We highlight a few potential negative societal impacts of TEOChat. First, TEOChat can automatically extract information from Earth observation imagery, which may lead to privacy concerns, especially as the imagery achieves higher and higher spatial resolution. Second, TEOChat, like other large multimodal models, does not always produce accurate outputs, sometimes hallucinating information or objects in the images, which has the potential to mislead users if misused. It is also possible but unlikely that fine-tuning on TEOChatlas may affect safeguards employed in Llama 2. Ongoing work on mitigating hallucinations in LLMs (Tonmoy et al., ) and LMMs (Zhong et al., ), as well as work addressing safety of LLMs (Huang et al., ), both have the potential to translate to LMMs for Earth observation, but while these approaches continue to mature, users should check outputs for important applications.

#### Author Contributions

- 1.
Jeremy Andrew Irvin: Developed the initial research idea, formulated instruction-following tasks, prepared datasets, implemented training experiments and evaluation code, designed and conducted training experiments, drafted the paper.
- 2.
Emily Ruoyu Liu, Joyce Chuyi Chen, Ines Dormoy: Prepared datasets, helped design architecture and formulate instruction-following tasks, implemented evaluation code, drafted and edited the paper.
- 3.
Jinyoung Kim: Prepared data, edited the paper.
- 4.
Samar Khanna: Helped design architecture, designed and conducted training experiments, edited the paper.
- 5.
Zhuo Zheng: Supervised the project, helped design architecture and formulate instruction-following tasks, edited the paper.
- 6.
Stefano Ermon: Co-developed the research idea, supervised the project, edited the paper.

## Peer Reviews

confidence: 5
contribution: 1
initial_rating: 1
presentation: 3
questions: As mentioned in the weakness
soundness: 2
strengths: This paper introduces TEOChat, which is the first unified model for temporal earth observation (EO) data
The authors create a novel EO dataset
summary: The work attempts to develop the first vision-language model for temporal earth observation data (EOD). When an EOD is prompted with temporal images and NLP, the proposed model can perform tasks that need temporal reasoning,
weaknesses: It would be more convincing if the authors could provide more evidence on the diversity and representativeness of the dataset, as well as fine-tuning results on supervised datasets.

Why only a comparison with GeoChat is made, whereas many other similar works are available?

What is the performance for other RS captioning datasets such as UCM-CAPTIONS and SYDNEY-CAPTIONS datasets which are mentioned in RSGPT[1].

It will also be important to showcase the performance w.r.t other performance measures such as recall etc

[1] RSGPT: A Remote Sensing Vision Language Model and Benchmark

In addition, the architecture and concept are very similar to Video-ChatGPT which has never been discussed or compared. Also, there are RS-based VLM which target change detection (https://github.com/lzw-lzw/awesome-remote-sensing-vision-language-models) which will be important to compare.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 2
questions: Please address the weaknesses above.
soundness: 3
strengths: S1. Applications around geo-spatial data are important and have potential for great societal impact.

S2. Applying VLMs to video data is a timely direction.

S3. Additional evidence that state-of-the-art fine-tuning methods are broadly applicable to specific target domains.
summary: The paper introduces a instruction tuning dataset for temporal earth observation and a corresponding fine-tuned model trained on this data.
weaknesses: W1. The introduction often reads as a related work section. Consider moving some of the contextualization of the field in the related work section and focus a bit more on the core story and contributions in the intro.

W2. While there is a long history of geo-spatial data in computer vision, the paper contribution does strike me a little narrow in scope. 

W3. Methodological novelty. The paper applies a fairly straightforward fine-tuning recipe. While the paper does not oversell the method, this combined with the narrow scope (W2) do make me a bit concerned.

W4. What are some broader applications of these agents? Is it possible to deploy these agents at scale to understand patterns in data that would be intractable for humans to otherwise deduce themselves?

W5. The task presented seem like important capabilities but not particularly difficult for specialized models to solve. How do baseline classifiers, referring expression, change detection, etc. algorithms perform on the tasks. What generalization capabilities can be expected in the proposed model that are not present in these baselines?

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: My concerns are listed in the ''Weaknesses".
soundness: 2
strengths: + The paper develops the first vision-language model (VLM) for temporal EO data and introduces the first temporal EO dataset for multimodal instruction-tuning, expanding the application scope for VLMs.

+ The proposed method shows impressive zero-shot performance on several datasets, and outperforms foundation models (i.e., GPT-4o and Gemini-1.5 Pro) for modeling sequences of images.
summary: In this paper, the authors develop a novel vision and language assistant (TEOChat) to engage in conversations about temporal sequences of earth observation data. Their extensive experiments show that TEOChat can perform a variety of spatial and temporal reasoning tasks, achieving comparable or better performance than specialist models.
weaknesses: - While contributing a novel application and a new dataset is encouraging, the technical contribution of this paper is insufficient.  This  paper primarily focuses on employing Low-Rank Adaptation (LoRA) to fine-tune a LLaVA-1.5-like architecture, which allows the model to handle both temporal and Earth Observation (EO) data. This paper could benefit from a more in-depth analysis of why such a design for the framework was chosen.

-  Although the authors mention that their approach achieves impressive performance on various datasets (refer to Table 1), the analysis of these results lacks specificity. It is recommended to present relevant visualizations to demonstrate the hypothesis.

- While the study provides a valuable analysis of the proposed approach against existing benchmarks, it would be enhanced by incorporating more recent specialist methods into the comparative analysis.

- There are some typos in the paper. For example, Line 106: “To align the the EO imagery”, double “the”.

---

confidence: 5
contribution: 3
initial_rating: 8
presentation: 3
questions: * A number of open VLMs now allow the processing of multiple images. Can the authors evaluate such options e.g., Qwen2VL, InternVL2 on temporal EO tasks explored in the paper?
* Some recent parallel works (e.g., CDchat https://arxiv.org/html/2409.16261v1) have also explored extending EO VLMs for change detection. Authors are encouraged to explain differences with them (although this is an Arxiv work, and comparing them is not mandatory).
* The VLM responses are textual, the metrics used for eval are accuracy, IoU and F1 measures. Its not clear how the textual outputs are parsed to obtain object coordinates, outputs classes etc. How robust is this process to avoid any errors due to incorrect parsing?
* How much the choice of low-image resolution affect the performance on localization tasks like detection when the objects are quite small. Authors use all the GeoChat data for training, including referring and grounding instructions, but do not compare single-image grounding tasks. It will be good to understand how the choice of vision encoder affects the performance on these tasks. 
* Why GPT4 and Gemini models were evaluated/compared on only two tasks? Analysis on the kind of cases where these models fail will be important.
soundness: 3
strengths: * A VLM tailored for temporal EO data addresses an important gap in current literature. Prior VLMs could only work with either natural video/multi-images or single EO images.
* The dataset extends GeoChat datasets to the temporal domain. The temporal component of the TEOchatlas dataset is comprised of 4 datasets (fMoW, xBD, S2Looking, QFabric) that are converted to the following temporal tasks: Temporal Scene Classification, Change Detection, Spatial Change Referring Expression, Change QA, Region-based Change QA, Temporal Referring Expression and Region-based Temporal QA.
* The model is comprehensively evaluated across multiple datasets and tasks. TEOChat consistently performs well on both generalist and specialist models on tasks such as change detection, temporal scene classification, and change question answering.  Its performance on zero-shot tasks is also significant. 
* Ablations done on the type of projector initialization, its finetuning startegy, and choice of image/video encoder are comprehensive and justify the design choices.
summary: The paper presents TEOchat, a vision-language model (VLM) explicitly designed for the temporal sequences of EO data. While existing temporal sequences models focus on natural images/videos and the EO VLMs focus mainly on single-images, TEOchat fills a gap by being the first to manage temporal EO data. Authors propose a new dataset called TEOchatlas with 554k examples of various single-image and temporal tasks instructions. TEOChat performs well in tasks such as change detection and temporal reasoning, outperforming both generalist and specialist VLMs designed for EO tasks.
weaknesses: * Similar to video models like Video-LLaVa, the authors limit the sequences to a maximum of 8 images. For longer sequences, how this restriction can affect the model results?
* TEOchat randomly sample 8 images without replacement from longer sequences, the choice of samples selected can significantly change the results. Can the authors report variance in results due to different random samplings?
* The statement "TEOChat also performs comparably to or outperforms a specialist model trained to perform specific tasks" seems quite a strong claim given the proposed model performs quite low in certain cases: e.g., xBD Loc. (IoU) and QFabric [2 and 5 images].
* Table 1, I believe the GeoChat model is evaluated zero-shot given it cannot be trained with multiple images? This should be clarified clearly in Table 1 that the comparison is between a zero-shot model evaluation (not built for multiple images) and TEOchat that is specifically trained with multiple images. Similarly, as Table 2 shows, the VideoLLava is also evaluated zero-shot in Table 1, and compared with the version that is finetuned on the TEOchatlas dataset, I do not think its a fair comparison and authors should clearly mention that they are comparing zero-shot models with a supervised model. 
* Contribution 2 gives the impression that the 554k instruction set is this paper's contribution while over 55% of it is included from previous work GeoChat. It would be reasonable to clarify this and change the claim accordingly. 
* The model uses interleaved image identifiers among the visual tokens in the sequence input to the LLM (Figure 3). This differs from prior methods for modeling temporal sequences of natural images which commonly concatenate the tokens and leads to significant gains as per ablation study. However, it is not clear why the image identifier would help in bi-temporal datasets like xBD and S2Looking, since tagging two images should not make much of a difference. Can the authors explain this?

Meta-review: The manuscript received ratings of 5, 8, 6, and 1. Reviewers appreciated that the manuscript introduces a model tailored for temporal Earth observation (EO) data along with tailored instruction set for tempora domain in remote sensing, thereby addressesing an important gap in existing literature. Reviewers also rasied several issues in the manuscript including, processing longer sequences (beyond  8 images), certain claims regarding the performance of TeoChat (e.g., xBD Loc. (IoU) and QFabric [2 and 5 images]), evaluating existing models (e.g., Qwen2VL, InternVL2) on temporal EO tasks, and comparison with other remote sensing captioning datasetss such as UCM-Captions and Sydney-Captions. Authors submitted a rebuttal to address the concerns of reviewers. One reviewer remained concerns on single image performance of proposed TeoChat on some datasets. Authors have detailed discussions with the reviewer explaining the results further (e.g., lack of open-source code availability of RSGPT) while also highlighting efficacy of the proposed approach in terms of its temporal capabilities. Two reviewers mentioned that their respective concerns are addressed during the rebuttal and remained positive, while the fourth reviewer remained concer about the narrow scope of the work.  The meta-reviewer believes that the proposed work has merits overall as it goes beyond single image remote sensing understanding to handling temporal Earth observations. Results on temporal Earth observation tasks are promising, while the authors have also provided additional results and comparisons with single EO image instruction-following models in the rebuttal. Given the reviewers comments, rebuttal and the detailed discussions, the recommendation is accept. Authors are strongly encouraged to take into consideration reviewers feedback when preparing the revised manuscript.

Decision: Accept (Poster)