Temporal Environment-Aware Image Generation via Latent Diffusion

Authors: Nasrin Kalanat, Yiqun Xie, Yanhua Li, Xiaowei Jia
Primary area: generative models
Keywords: Generative models, Latent Diffusion, Time series data, Multi-modal learning

Abstract:
Low-cost cameras have recently become widely used to monitor environmental ecosystems. This paper focuses on scene prediction for monitoring small streams, which is critical for ensuring water supply and informing early actions for floods and droughts. In contrast to traditional stream models that typically rely on coarse-resolution weather data, stream images provide detailed information about water properties and local environment at a higher temporal frequency. This paper presents a multi-modal generative framework designed for frequent temporal stream imagery datasets, aimed at generating the subsequent stream images. This task is challenging due to the variability of stream images caused by changes in time and local environmental conditions. Our method captures scene changes in both stream and surrounding environment by incorporating temporal context of weather, water flow, and time information. We also introduce a domain-discriminative learning approach to enforce the learning of domain-specific information in generating images. Our experiments demonstrate the superior performance of the proposed method in preserving semantics of water and environmental properties, using real data from the West Brook area in western Massachusetts, USA.

Decision: desk reject

Reviews and discussion:

Metareview: