TEOChat: A Large Vision-Language Assistant for Temporal Earth Observation Data

Authors: Jeremy Andrew Irvin, Emily Ruoyu Liu, Joyce C. Chen, Ines Dormoy, Jinyoung Kim, Samar Khanna, Zhuo Zheng, Stefano Ermon
Primary area: foundation or frontier models, including LLMs
Keywords: vision-language model, large multimodal model, satellite imagery, earth observation, change detection

Abstract:
Large vision and language assistants have enabled new capabilities for interpreting natural images. These approaches have recently been adapted to earth observation data, but they are only able to handle single image inputs, limiting their use for many real-world tasks. In this work, we develop a new vision and language assistant called TEOChat that can engage in conversations about temporal sequences of earth observation data. To train TEOChat, we curate an instruction-following dataset composed of many single image and temporal tasks including building change and damage assessment, semantic change detection, and temporal scene classification. We show that TEOChat can perform a wide variety of spatial and temporal reasoning tasks, substantially outperforming previous vision and language assistants, and even achieving comparable or better performance than several specialist models trained to perform specific tasks. Furthermore, TEOChat achieves impressive zero-shot performance on a change detection and change question answering dataset, outperforms GPT-4o and Gemini 1.5 Pro on multiple temporal tasks, and exhibits stronger single image capabilities than a comparable single image instruction-following model on scene classification, visual question answering, and captioning. We publicly release our data, models, and code at https://github.com/ermongroup/TEOChat .

Decision: accept (poster)
Reviewer scores: 1, 5, 6, 8

Reviews and discussion:
<thread>
creation date:
1729445133940

thread_type:
official_review

<thread_object>
creator: reviewer_3yvb

<content>
summary:
The paper presents TEOchat, a vision-language model (VLM) explicitly designed for the temporal sequences of EO data. While existing temporal sequences models focus on natural images/videos and the EO VLMs focus mainly on single-images, TEOchat fills a gap by being the first to manage temporal EO data. Authors propose a new dataset called TEOchatlas with 554k examples of various single-image and temporal tasks instructions. TEOChat performs well in tasks such as change detection and temporal reasoning, outperforming both generalist and specialist VLMs designed for EO tasks.

strengths:
* A VLM tailored for temporal EO data addresses an important gap in current literature. Prior VLMs could only work with either natural video/multi-images or single EO images.
* The dataset extends GeoChat datasets to the temporal domain. The temporal component of the TEOchatlas dataset is comprised of 4 datasets (fMoW, xBD, S2Looking, QFabric) that are converted to the following temporal tasks: Temporal Scene Classification, Change Detection, Spatial Change Referring Expression, Change QA, Region-based Change QA, Temporal Referring Expression and Region-based Temporal QA.
* The model is comprehensively evaluated across multiple datasets and tasks. TEOChat consistently performs well on both generalist and specialist models on tasks such as change detection, temporal scene classification, and change question answering.  Its performance on zero-shot tasks is also significant. 
* Ablations done on the type of projector initialization, its finetuning startegy, and choice of image/video encoder are comprehensive and justify the design choices.

weaknesses:
* Similar to video models like Video-LLaVa, the authors limit the sequences to a maximum of 8 images. For longer sequences, how this restriction can affect the model results?
* TEOchat randomly sample 8 images without replacement from longer sequences, the choice of samples selected can significantly change the results. Can the authors report variance in results due to different random samplings?
* The statement "TEOChat also performs comparably to or outperforms a specialist model trained to perform specific tasks" seems quite a strong claim given the proposed model performs quite low in certain cases: e.g., xBD Loc. (IoU) and QFabric [2 and 5 images].
* Table 1, I believe the GeoChat model is evaluated zero-shot given it cannot be trained with multiple images? This should be clarified clearly in Table 1 that the comparison is between a zero-shot model evaluation (not built for multiple images) and TEOchat that is specifically trained with multiple images. Similarly, as Table 2 shows, the VideoLLava is also evaluated zero-shot in Table 1, and compared with the version that is finetuned on the TEOchatlas dataset, I do not think its a fair comparison and authors should clearly mention that they are comparing zero-shot models with a supervised model. 
* Contribution 2 gives the impression that the 554k instruction set is this paper's contribution while over 55% of it is included from previous work GeoChat. It would be reasonable to clarify this and change the claim accordingly. 
* The model uses interleaved image identifiers among the visual tokens in the sequence input to the LLM (Figure 3). This differs from prior methods for modeling temporal sequences of natural images which commonly concatenate the tokens and leads to significant gains as per ablation study. However, it is not clear why the image identifier would help in bi-temporal datasets like xBD and S2Looking, since tagging two images should not make much of a difference. Can the authors explain this?

questions:
* A number of open VLMs now allow the processing of multiple images. Can the authors evaluate such options e.g., Qwen2VL, InternVL2 on temporal EO tasks explored in the paper?
* Some recent parallel works (e.g., CDchat https://arxiv.org/html/2409.16261v1) have also explored extending EO VLMs for change detection. Authors are encouraged to explain differences with them (although this is an Arxiv work, and comparing them is not mandatory).
* The VLM responses are textual, the metrics used for eval are accuracy, IoU and F1 measures. Its not clear how the textual outputs are parsed to obtain object coordinates, outputs classes etc. How robust is this process to avoid any errors due to incorrect parsing?
* How much the choice of low-image resolution affect the performance on localization tasks like detection when the objects are quite small. Authors use all the GeoChat data for training, including referring and grounding instructions, but do not compare single-image grounding tasks. It will be good to understand how the choice of vision encoder affects the performance on these tasks. 
* Why GPT4 and Gemini models were evaluated/compared on only two tasks? Analysis on the kind of cases where these models fail will be important.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730627210217

thread_type:
official_review

<thread_object>
creator: reviewer_qa8u

<content>
summary:
In this paper, the authors develop a novel vision and language assistant (TEOChat) to engage in conversations about temporal sequences of earth observation data. Their extensive experiments show that TEOChat can perform a variety of spatial and temporal reasoning tasks, achieving comparable or better performance than specialist models.

strengths:
+ The paper develops the first vision-language model (VLM) for temporal EO data and introduces the first temporal EO dataset for multimodal instruction-tuning, expanding the application scope for VLMs.

+ The proposed method shows impressive zero-shot performance on several datasets, and outperforms foundation models (i.e., GPT-4o and Gemini-1.5 Pro) for modeling sequences of images.

weaknesses:
- While contributing a novel application and a new dataset is encouraging, the technical contribution of this paper is insufficient.  This  paper primarily focuses on employing Low-Rank Adaptation (LoRA) to fine-tune a LLaVA-1.5-like architecture, which allows the model to handle both temporal and Earth Observation (EO) data. This paper could benefit from a more in-depth analysis of why such a design for the framework was chosen.

-  Although the authors mention that their approach achieves impressive performance on various datasets (refer to Table 1), the analysis of these results lacks specificity. It is recommended to present relevant visualizations to demonstrate the hypothesis.

- While the study provides a valuable analysis of the proposed approach against existing benchmarks, it would be enhanced by incorporating more recent specialist methods into the comparative analysis.

- There are some typos in the paper. For example, Line 106: “To align the the EO imagery”, double “the”.

questions:
My concerns are listed in the ''Weaknesses".
</content>
</thread_object>

</thread>
<thread>
creation date:
1730701053528

thread_type:
official_review

<thread_object>
creator: reviewer_e7d5

<content>
summary:
The paper introduces a instruction tuning dataset for temporal earth observation and a corresponding fine-tuned model trained on this data.

strengths:
S1. Applications around geo-spatial data are important and have potential for great societal impact.

S2. Applying VLMs to video data is a timely direction.

S3. Additional evidence that state-of-the-art fine-tuning methods are broadly applicable to specific target domains.

weaknesses:
W1. The introduction often reads as a related work section. Consider moving some of the contextualization of the field in the related work section and focus a bit more on the core story and contributions in the intro.

W2. While there is a long history of geo-spatial data in computer vision, the paper contribution does strike me a little narrow in scope. 

W3. Methodological novelty. The paper applies a fairly straightforward fine-tuning recipe. While the paper does not oversell the method, this combined with the narrow scope (W2) do make me a bit concerned.

W4. What are some broader applications of these agents? Is it possible to deploy these agents at scale to understand patterns in data that would be intractable for humans to otherwise deduce themselves?

W5. The task presented seem like important capabilities but not particularly difficult for specialized models to solve. How do baseline classifiers, referring expression, change detection, etc. algorithms perform on the tasks. What generalization capabilities can be expected in the proposed model that are not present in these baselines?

questions:
Please address the weaknesses above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730821883621

thread_type:
official_review

<thread_object>
creator: reviewer_7dge

<content>
summary:
The work attempts to develop the first vision-language model for temporal earth observation data (EOD). When an EOD is prompted with temporal images and NLP, the proposed model can perform tasks that need temporal reasoning,

strengths:
This paper introduces TEOChat, which is the first unified model for temporal earth observation (EO) data
The authors create a novel EO dataset

weaknesses:
It would be more convincing if the authors could provide more evidence on the diversity and representativeness of the dataset, as well as fine-tuning results on supervised datasets.

Why only a comparison with GeoChat is made, whereas many other similar works are available?

What is the performance for other RS captioning datasets such as UCM-CAPTIONS and SYDNEY-CAPTIONS datasets which are mentioned in RSGPT[1].

It will also be important to showcase the performance w.r.t other performance measures such as recall etc

[1] RSGPT: A Remote Sensing Vision Language Model and Benchmark

In addition, the architecture and concept are very similar to Video-ChatGPT which has never been discussed or compared. Also, there are RS-based VLM which target change detection (https://github.com/lzw-lzw/awesome-remote-sensing-vision-language-models) which will be important to compare.

questions:
As mentioned in the weakness
</content>
</thread_object>

</thread>
<thread>
creation date:
1732006176504

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Initial Response to Reviewers

comment:
We kindly thank the reviewers for their helpful feedback on our work. We appreciate that they find our approach effective (`7dgE`, `e7d5`, `qA8u`, `3yvB`), our experiments extensive (`qA8u`, `3yvB`), and our dataset novel, filling an important gap in the literature (`7dgE`, `qA8u`, `3yvB`).

The major concerns raised by the reviewers include:
1. Comparing to other single image and multi-image instruction-following approaches. 
2. Comparing to state-of-the-art (SOTA) specialist approaches.
3. Adding EO captioning and single image grounding results.
4. Measuring the impact of long sequence subsampling in fMoW and benefit from image references on the bitemporal datasets.
5. Adding more qualitative examples demonstrating TEOChat’s capabilities.

To address reviewer feedback, we have included or plan to include the following in our revision:
1. **A comparison to several single EO image instruction-following models and open-source multi-image instruction-following models.** We find TEOChat also achieves better performance than all single EO image instruction-following models on four of seven tasks, similar performance to the best performing models on two, and underperforms two models on one task, which we attribute to dataset composition differences. We are currently evaluating two open-source multi-image instruction-following models for comparison as well.
2. **A comparison to SOTA specialist approaches.** TEOChat’s performance is impressively close to specialist SOTA on fMoW RGB (75.1% vs. 79.2%), and lower than SOTA on xBD localization and S2Looking detection, which we mostly attribute to TEOChat’s constrained output format of boxes versus segmentation masks.
3. **Additional evaluations on three EO captioning datasets and on single image grounding.** We find TEOChat exceeds the performance of specialists and matches generalist single image EO models on the larger two of the three captioning datasets, and shows strong captioning abilities qualitatively on all three datasets. We are working on getting single image grounding results as well.
4. **New ablations testing the effect of long sequence subsampling in fMoW and investigating the reason why image references benefit bitemporal performance.** We find minimal variance due to the sampling (0.1-0.2%) and that the `Image 2` reference explains most of the performance gains in the bitemporal case (85% on xBD damage classification, 100% on S2Looking detection), suggesting it is primarily the delineation of visual tokens from the images that is beneficial in the bitemporal setting.
5. **Additional qualitative examples of TEOChat’s predictions on various EO tasks.** We will be adding these in the next revision.

As mentioned, we are running a few additional experiments and will follow up with results as we obtain them. In the meantime, we would be very grateful if reviewers could discuss our initial responses to their comments and [the first version of our revised paper](https://openreview.net/pdf?id=pZz0nOroGv), which has changes marked in `red`. Except for a minor error we corrected in our per-pixel F1 implementation that affected a few results, all changes to the paper are described in our reviewer-specific responses below.

Thank you all again for your review of our work.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733276272682

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of Paper Changes during Discussion Period

comment:
As the discussion period comes to a close, we want to sincerely thank the reviewers again for their valuable feedback on our work. 

Below, we summarize the major changes made to our paper based on the reviewer feedback. All of the changes to the paper are marked in `red` in [our revised paper](https://openreview.net/pdf?id=pZz0nOroGv).

1. **A comparison to additional single EO image instruction-following models (Table 1), SOTA specialists (Section 5.1), and open-source multi-image instruction-following models (Table 4).** TEOChat achieves similar or higher single image performance than all models on all but one task (Table 1), achieves impressively close to SOTA results on fMoW but lower on xBD localization and S2Looking detection (Section 5.1), and considerably higher performance than two open-source multi-image instruction-following models (Table 4). 

2. **Additional evaluations on three single EO image captioning datasets (fine-tuning in Tables 11-14, zero-shot in Tables 15-18) and on single image grounding (Table 19).**  TEOChat achieves captioning performance competitive with other single image methods on average quantitatively in both the fine-tuning and zero-shot settings (Table 14, Table 18) and demonstrates strong performance qualitatively (Figure 8). On single image grounding, TEOChat outperforms MiniGPTv2 across all grounding tasks and outperforms GeoChat in detecting large objects but underperforms GeoChat across the other grounding tasks (Table 19), which we largely attribute to the 2.25x lower spatial resolution required to handle the extra temporal inputs. We emphasize that TEOChat has additional temporal capabilities that GeoChat does not.

3. **Additional qualitative examples of TEOChat’s predictions on various EO tasks (Figures 4, 6-8).** Interestingly, the examples demonstrate TEOChat’s ability to compose its spatial and temporal reasoning capabilities in novel ways not explicitly represented in the training set (Figure 4).

4. **New ablations testing the effect of long sequence subsampling in fMoW (Table 20) and investigating the reason why image references benefit bitemporal performance (Table 9).** We find minimal variance due to the sampling (Table 20) and that it is primarily the delineation of visual tokens between the images that makes the image references beneficial in the bitemporal setting (Table 9).

5. **More discussion of general motivation for the work and a separate related works section (Section 1, Appendix Section A).** We add further motivation and context to the Introduction (Section 1), and include a separate related works section (Appendix Section A) to better contextualize our contributions.

We believe these changes have substantially improved our paper, which we are very grateful for. Thank you all again for your thoughtful review of our work.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The manuscript received ratings of 5, 8, 6, and 1. Reviewers appreciated that the manuscript introduces a model tailored for temporal Earth observation (EO) data along with tailored instruction set for tempora domain in remote sensing, thereby addressesing an important gap in existing literature. Reviewers also rasied several issues in the manuscript including, processing longer sequences (beyond  8 images), certain claims regarding the performance of TeoChat (e.g., xBD Loc. (IoU) and QFabric [2 and 5 images]), evaluating existing models (e.g., Qwen2VL, InternVL2) on temporal EO tasks, and comparison with other remote sensing captioning datasetss such as UCM-Captions and Sydney-Captions. Authors submitted a rebuttal to address the concerns of reviewers. One reviewer remained concerns on single image performance of proposed TeoChat on some datasets. Authors have detailed discussions with the reviewer explaining the results further (e.g., lack of open-source code availability of RSGPT) while also highlighting efficacy of the proposed approach in terms of its temporal capabilities. Two reviewers mentioned that their respective concerns are addressed during the rebuttal and remained positive, while the fourth reviewer remained concer about the narrow scope of the work.  The meta-reviewer believes that the proposed work has merits overall as it goes beyond single image remote sensing understanding to handling temporal Earth observations. Results on temporal Earth observation tasks are promising, while the authors have also provided additional results and comparisons with single EO image instruction-following models in the rebuttal. Given the reviewers comments, rebuttal and the detailed discussions, the recommendation is accept. Authors are strongly encouraged to take into consideration reviewers feedback when preparing the revised manuscript.