LITE: Modeling Environmental Ecosystems with Multimodal Large Language Models
Authors: Haoran Li
Junqi Liu
Zexian Wang
Shiyuan Luo
Xiaowei Jia
Huaxiu Yao
Venue: COLM (2024)
Keywords: Environmental Ecosystems, Foundation Models, Spatio-temporal Data, Multimodal Learning
TL;DR: A novel multimodal large language model for cross-domain environmental ecosystem modeling, demonstrating SOTA performance across all domains and maintaining robustness to incomplete observations and distribution shifts in environmental data.

Abstract: The modeling of environmental ecosystems plays a pivotal role in the sustainable management of our planet. Accurate prediction of key environmental variables over space and time can aid in informed policy and decision-making, thus improving people's livelihood. Recently, deep learning-based methods have shown promise in modeling the spatial-temporal relationships for predicting environmental variables. However, these approaches often fall short in handling incomplete features and distribution shifts, which are commonly observed in environmental data due to the substantial cost of data collection and malfunctions in measuring instruments. To address these issues, we propose LITE -- a multimodal large language model for environmental ecosystems modeling. Specifically, LITE unifies different environmental variables by transforming them into natural language descriptions and line graph images. Then, LITE utilizes unified encoders to capture spatial-temporal dynamics and correlations in different modalities. During this step, the incomplete features are imputed by a sparse Mixture-of-Experts framework, and the distribution shift is handled by incorporating multi-granularity information from past observations. Finally, guided by domain instructions, a language model is employed to fuse the multimodal representations for the prediction. Our experiments demonstrate that LITE significantly enhances performance in environmental spatial-temporal prediction across different domains compared to the best baseline, with a 41.25\% reduction in prediction error. This justifies its effectiveness.

Published as a conference paper at COLM 2024
LITE: Modeling Environmental Ecosystems with Multimodal Large Language Models
Haoran L Junqi Liu', Zexian WangS, Shiyuan Luo", Xiaowei Jia" Huaxiu Yao* "UNC-Chapel Hill, +PolyU, SUMich, Independent Researcher, "University of Pittsburgh haoran. 1i, cisgmaill com, huaxiuêcs, enc.edu
Abstract
The modeling of environmental ecosystems plays a pivotal role in the sus- tainable management of our planet. Accurate prediction of key environmen- tal variables over space and time can aid in informed policy and decision- making, thus improving people's livelihood. Recently, deep learning- bassed methods have shown promise in modeling the spatial-trmporal relation- ships for predicting environmental variables. However, these approaches often fall short in handling incomplete features and distribution shifts, which are commonly observed in environmental data due to the substantial cost of data collection and malfunctions in measuring instruments. To ad- dress these issues, we propose LITE - a multimodal large language model for environmental ecosystems modeling. Specifically, LITE unifies different environmental variables by transforming them into natural language de- scriptions and line graph images. Then, LITE utilizes unified encoders to capture spatial-temporal dynamics and correlations in different modalities. During this step, the incomplete features are imputed by sparse Mixture- of-Experts framework, and the distribution shift is handled by incorporat- ing multi-granularity information from past observations. Finally, guided by domain instructions, a language model is employed to fuse the multi- modal representations for the prediction. Our experiments demonstrate that LITE significantly enhances performance in environmental spatial- temporal prediction across different domains compared to the best baseline, with 41.25% reduction in prediction error. This justifies its effectiveness. /ttp//gttthhhcc cactuuu.com///e caaalbliccc/chc/////o/// hrrics/LITE LITE
Our data and code are available at /ttps://///t
1 Introduction
Sustainable management of environmental ecosystems has been gaining massive attention due to its impact on global climate, and food and water security (Tamurini et al. 2020 O'Donnell et al., 2023 Li et al. 2023). However, this task has become more challenging due to the rising demand for environmental services from the growing population, more frequent extreme weather events, and the climate change (Zurek et al., 2022 Jasechko et al. 2024). The modeling of environmental ecosystems is critical for the sustainable management, as it provides important information about the spatial-tempore variables. For example, accurate prediction of streamflow and water temperature for large river basins can aid in the decision-making for water resource allocation and water quality control. An agricultural monitoring system can help develop policies that maintain farming efficiency and sustainability, and stabilize economies in pnrricltture-ntensiin areas.
dynamics of key physical
Physics-based models Markstrom et al. 2015 Theurer et al. 1984 Zhou et al. 2021) and data- driven models (Zaremba et 2014 Hochreiter & Schmidhuber [997] Moshe et al. 2020 Jia et al. 2021 have been developed for spatial-temporal prediction in environmental appli- cations. In particular, machine learning models (e.g., recurrent neural networks (Zarembal
et al. 2014 Hochreiter & Schmidhuber 1997) and graph neural network based models (Chen al. 2021 Sun et 2021)) significantly improve the performance of environmental spatial- temporal prediction by modeling the complex spatial-tene -ettial-emmpoal correlations automatically from observed data. Despite the improved predictive accuracy, two key challenges remain
1

---

Published as a conference paper at COLM 2024
unsolved. Firstly, existing methods exhibit vulnerability to incomplete features, which can be a common issue in environmental applications due to sensor failures and the high cost of field surveys needed to measure certain variables across different regions and time. Second, these prior approaches can be severely affected by data distribution shifts, which are commonly observed in environmental applications due to variations in human management and changes in weather conditions.
To address these challenges, we propose general large language model (LLM)-based framework, termed LITE, to handle incomplete features and distribution shifts behind the environmental data. Specifically, we first transform the spatial-temporal (ST) data into semantic time-series and temporal trend images. The semantic time-series reflects inter-variable correlations while the temporal trend image depicts temporal dynamics of all variables in the form of line graph images. In this step, we tackle the incomplete features by replacing the absent variables with special token (e.g,, [MASK]) semantic time-series and linearly interpolating the temporal trend image. Then, LITE leverages semantic time-series encoder and a vision encoder to jointly capture spatial-temporal dynamics and correlations. During this process, incorporate multi-granullrrit information, i.e, merging previous observational data from different granularity such as week, month, and year to facilitate the handling of distribution shifts. Finally, the information, representing the spatial-tem poral dynamics and dependencies, will be processed by frozen LLM under the guidance of domain instructions, including dataset description, task description, and target statistics.
The primary contribution of this paper is the development of the new framework LITE, which introduces innovative multimodal representation learning framework for model- ing environmental ecosystems. This approach not only exploits learned knowledge within foundation models to handle the heterogeneity of different environmental domains, but also presents consistent robustness to different levels of incomplete observations and distribu- tion shifts. We evaluate LITE on three environmental spatial-temporal prediction scenarios: streamflow, water temperature, and agricultural nitrous oxide (N₂0) emissions. For all scenarios, our empirical results demonstrate an improvement of 12.2%, 56.0%, and 55.6% in terms of RMSE, verifying the the effectiveness of LITE. Furthermore, we conduct experi- ments to demonstrate that the superiority of LITE remains or is even more significant under (1) data distribution shift; (2) extreme incomplete features, offering further justifications the robustness and effectiveness of LITE.
2 Preliminaries
In this section, we introduce the notations used in this paper and formally define the problem setup.
Definition 1 (Spatial-Temporal Series): For region R in an environmental ecosystem, we denote the current timestamp as t, and the time range as a set T {i 1,,,,, t} consisting of |T| evenly split non-overlapping time intervals. Then, the series of target physical variables Yr, (e.g., streamflow, water temperature, agricultural nitrous oxide (N2O))
over multiple spatial locations is represented
VA = (vrrle ET,ER).
(1)
Problem Definition. Given the input physical drivers X' = ( TTr R) for multiple regions until the current time f, our goal is to predict the spatial-temporal series of the target variable Vr, until the current time, as
) = fo(G))
(2)
where fe represents the ST model parameterized by 8, which predicts the target spatial- temporal series using input physical drivers. The input drivers are physical variables that can affect the state of target environmental ecosystems, e.g,, meteorological and soil-related variables.
2

---

Published as a conference paper at COLM 2024
input Feotures
Pescription
Semantic Time-series Encoder
Decoder
Corment
Enport
band to dalerre Sesiaiai
domain instruction
wwwil:
waek
TransFormer Encoder
year
Expert
Current
SWoE Imputation
LLM
your
Vision Encoder
FFN
Image
Date image
mordh
month
Figure 1: Overview of our proposed LITE model for environmental ecosystem modeling, which consists of (1) Transforming environmental data into natural language descriptions and line graph images; (2) multimodal representation learning; (3) multimodal fusion by LLM decoder.
Multimodal Large Language Model for Environmental Ecosystem Modeling
This section presents our proposed Multimodal Large Language Model for EnvIronmen Ecosystem Modeling (LITE). The goal of LITE to capture ST dynamics and correlations from a multimodal perspective, and process this information within an LLM-based frame- work, as depicted Figure To elaborate, we first transform the ST data into semantic time-series data and temporal trend images. Incomplete features are addressed by replacing missing variables with a special token (e.g,, [MASK]) linearly interpolating in the temporal trend images. Subsequently, we utilize seman- tic time-series encoder and a vision encoder to learn multimodal representations for the environmental ST data. To enhance the robustness to distribution shifts, incorporate multi-granularrty information from previous observations in this step. Finally, the multi- modal representations are fused by a frozen LLM decoder to perform ST prediction under the guidance of domain instruction, i.e., dataset description, task description, and target statistics. Below, we detail each component of LITE.
the semantic time-series, and by
3.1 Representation Learning from Semantic Time-Series
In this section, we first discuss how to learn representation from environmental data via semantic time-series, which includes three steps: (1) transforming environmental data into semantic time-series, (2) imputation of incomplete observations, and (3) multi-granularity information integration. We detail these steps below.
Transforming Environmental Data into Semantic Time-Series. Given the varying types and quantities of features across different environmental datasets, we propose to first unify these features and transform them into corresponding semantic descriptions. Specifically, we first format each data point X,, of K dimensions into a sequence of key-value pairs. For each pair, the key represents the feature description & while the value is the corresponding feature value x This can be formulated as follows:
Linearize(x,,)) {(tt 1
(3)
Then, we utilize a prompt to instruct an LLM to transform the linearized data point to natural language, where the prompt consists of prefix to describe the schema of the input features, the linearization, and suffix (see the detailed prompts in Appendix A.1.I. Given the prompt, the LLM will generate readable and concise natural description Zr which can be formulated as:
Zrt LLM(p, (x,:))
(4)
3

---

Published as a conference paper at COLM 2024
is worth noting that the missing variables will be replaced by special token (e.g., [MASK]) for further imputation.
Imputation of Incomplete Observations. To handle the incomplete observation issue, utilize a Sparse
(SMoE) layer for imputing the incomplete observations. After transforming each data point into natural language description Zr,1, feed it into semantic time-series encoder Fi to get the encoded representations {mr Ji for each special token (i.e., incomplete feature), with H denotes the number of missing variables. The encoding process can be formulated as follows:
(5) Subsequently, to accommodate different characteristics of diverse physical variables, we apply Sparse coaturee-ofExperrr (SMoE) layer to impute the missing variables. Specifi- cally, the encoded representation , will be dynamically routed by an observation- independent noisy top-k gating network & to subset of shared expert models (E),,, to facilitate the proper imputation of different observations. Following the original design of SMoE (Shazeer et al. (017), the gating process can be formulated as:
| ) Fi(,)))
(mm,,) Softmax(TopK(((mnn
P(m)))) m* Wg + ySoftplus(m"); Wnoise)
(6)
D; is in the top elements of -00, otherwise
ojr
TopK(v,k);
where the j is random noise sampled from standard normal distribution, W3 and Wnoise are learnable parameters. Then, the encoded representations n/, will be routed only to the shared expert models (E with top-k gating scores generated by the gating network G. The predicted value mř, can then be calculated by combining the encoding results from the top-k expert models with their corresponding gating scores as:
, = (mm))) {mm,))
Σ ge (mm,, Em))))
(7)
Then, we can replace the incomplete observations in Zr with the corresponding predicted values {,7 h== obtaining the complete data ε,
H
Multi-granularity information integration. To address the distribution shift issue, we pro- pose to incorporate multi-gran information from previous observations. Specifically, for imputed time-series ž, in region at timestamp f, the incorporated multi-granularityy information includes: (1) information from the past week in the same region, denoted as by (ze f C (t 1, I,...,t 7}); (2) information from the same day of each month of the year in the same region, denoted as = (zr, f {t -t-300
30,..., (0.30,.... 360) }. Subsequently, we feed the current observation ž,, into the semantic time-series encoder Fi to get the current embedding ,r,,- Likewise, weekly information 2μ, and yearly information E are encoded into weekly embedding U.Y, and yearly embedding u respectively. Finally, we concatenate all the granuligra information as the representation of semantic time-series the LLM decoder, which can be formulated as:
U, U + Uy + UN,
(8)
where the is the concatenation operation along the representation dimension.
3.2 Representation Learning from Temporal Trend Image
Aside from learning representations from semantic time-series, we also propose to capture the dynamics physical variables in the temporal trend image. This image depicts the recent temporal dynamics of all variables simultaneously, making the capture of inter- variable relations more natural and easier. Specifically, we first convert the previous feature values into temporal trend images, which depict the temporal dynamics of all variables in the form of line graph images. Then, we utilize a vision encoder to encode the ST dynamics and correlations in the temporal trend image. These two steps are detailed below:
4

---

Published as a conference paper at COLM 2024
Transforming environmental data into temporal trend im- age. Given the current observation Xy,, we utilize the line graph image to present each variables' (including the target variable's) observations in past & days in the same region, i.e, xy {xye f' 1f.., ()}- For each variable in x/, each point represents observation identified by its time and value. As illustrated in Figure for each variable (one subimage), the horizontal axis represents timestamps, and the vertical axis specifies values. Here, incomplete features are lin- early interpolated, and the line graph images of all variables will be finally aggregated into one temporal trend image i,, Following Li al. (2024), we apply Z-normalization on all variables due to the different scales of environmental variables.
MA
MMM
Figure 2: Illustration of tem- poral trend image.
Image Recognition for temporal trend image. After transforming the observations into the temporal trend image ir,,, the temporal trend image i,, is then embedded into a visual representation Or, by the vision encoder Fú, i.e., pre-trained vision transformer (Liu al. 2021b):
O₁, (i,,)
(9)
is worth noting that the temporal trend image presents two types of information for vision transformer: (1) the temporal dynamics of single variable in line graph, and (2) the spatial relationships amongst the variables across different line graphs. The vision encoder can explicitly capture the inter-variable and intra-variable dependencies.
3.3 Inference with Large Language Models
After obtaining representations from semantic time-series and temporal trend image, we employ an LLM--ased framework to fuse multimodal representations for prediction. In order to adapt the LLM to different applications, we propose to guide the frozen LLM with domain instructions. Concretely, the domain instruction consists of three parts: (1) dataset description, which provides essential background information of specific environmental application; (2) task description, which guides the L.LM to perform multimodal fusion for the given task; (3) target statistics, which serves crucial information for LLM to capture the recent dynamics of the target variable (see Appendix A.1.2 for detailed descriptions of domain instruction). For each data point Xy,1, concatenate the learned representations from different modalities with domain instructions, and then forward them through a frozen LLM to obtain the output representation
Q. = d,,
(10)
where de represents the domain instructions for a environment a. Subsequently, we employ linear layer on the top of output representation Qr to derive the final prediction Vr,-- During training, we employ the prediction loss L, and the imputation loss C; of incomplete observations, which are defined as follows:
C, -- /
Iy., E
1 HE
11 4
I TEEE M FERIET HEH.
moth
(11)
FERIET
where N denotes the total number of observed target variables, Hy, denotes the set of incomplete features in region at time f, and M denotes the total number of incomplete features. The overall loss Lill is defined as follows:
Cal = C, + /22,
(12)
where 1/1 and !/2 are coefficients to balance these loss terms. The whole algorithm for training is illustrated in Alg. in Appendix A.2
Related work
Modeling of Environmental Ecosystems. To address the environmental challenges, physics- based models have been commonly adopted in many areas such as hydrology and agricul- ture (Regan et al.) 2018 Grant et al. 2010 Zhoo et al. 2021), specifically like multi multi-purpose
5

---

Published as a conference paper at COLM 2024
terrestrial simulator (Coon et al. 2019) and hydrological budget and crop yield predic- tor (Srinivasan et al. 2010). These models often rely on parameterizations and approxima- excessive complexity in modeling certain physical
tions due to incomplete knowledge processes. As a result, they are often biased even after being calibrated using sufficient training data. As an alternative, advanced machine learning (ML) has been increasingly adopted to help understand the underlying complex physical processes and deal with large amounts of data. In particular, prior research has shown the promise of ML-based approaches in modeling agroecosystems (Jia et al. 2019 Licheng et al. 2021 Liu et al. 2022) and freshwater ecosystems (Read et al. 2019] et a and more recently, encouraging
results by large language models in modeling environmental ecosystems (Luo et al. 2023).
Both physics-based and ML-based models would benefit mecision-making activities relevant
to society, highlighting the intersection of environmental science, technology, and policy.
Spatial-Temporal Prediction. The ST prediction serves as crucial topic in spatial-temporal data mining. Earliest models, e.g., ARIMA (Faruk 2010 only focus on temporal dynamics. Later, multiple CNN-based and RNN-based models are proposed to simultaneously capture spatial-temporal correlations (Su et al. 2020 Sun et al. 2021 Yao et al. 2018 2019b a) Recently, with advanced deep learning methods such as graph neural networks and trans- formers, the performance
ST prediction has been significantly improved (Cini et al. 2023
Tang et al. 2023 Liu et al. 2023). ) However, ST prediction for environmental ecosystems still
suffers from incomplete observations and distribution shifts.
Imputation of Incomplete Observations. One common issue in environmental data is the incomplete data, which is mainly caused by sensor failures and the high cost of measuring certain variables. To build general framework for environmental ecosystems modeling, the first step is to properly handle the incomplete features. Previous research has proposed to address the incomplete observations by various data imputation methods (Berg et al.
2017: Gondara & Wang 2018 Yoon et al. 2018 Spinelli et al. 2020 or by directly im imputing
the incomplete features when making decisions (Goodfellow et al., 2013 Smieja et al. 2018
Ma et al. 2021 2022: Tang et al. 2020). Nevertheless, these methods do not accommodate
the heterogeneity amongst different variables in the incomplete observations, potentially leading to uninformed guessing when there are various missing values in one feature.
5 Experiments
In this section, we evaluate the performance of LITE, aiming to answer the following questions: Q1: Compared to previous methods, can LITE consistently perform well across different environmental science applications? Q2: Can the proposed modules (e-g., semantic time-series encoder) effectively improve performance? Q3: How does LITE perform when confronted with the challenge of incomplete features? Q4: Can LITE be robust to distribution shifts in environmental data?
5.1 Experimental Setup
Datasets Descriptions. In this subsection, we will briefly introduce three datasets from environmental ecosystems and Appendix A.3 provides further details.
CRW-Temp. CRW-Temp is stream water temperature prediction dataset, aiming to predict the daily average water temperature on a given day based on the observed physical variables on the same day. It is collected from the Christina River Watershed (CRW), which consists of 42 river segments. Water temperature observations are available for 22 segments on certain dates. The number of observations for each segment ranges from 1 to 3, 572 with total of 506 observations across all dates and segments. The temporal span of the dataset extends from October 31, 2006, to March 30, 2020, covering 4, 900 days. For the experiments, the first 2, 450 days are selected as the training split, while the subsequent 2, 450 days serve as the test split.
CRW-Flow. The target of CRW-Flow is to predict the streamflow of river segments based on the observed physical variables. This dataset is also obtained from the Christina River

---

Published as a conference paper at COLM 2024
Table 1: Prediction RMSE and MAE for stream water temperature, stream flow, and agricul- tural nitrous oxide (N₂O) emission prediction. The best results are bold, while the second best results are underlined.
TempTemp
CRW-Flow
AGR
Class
Method
RMSE
MAE
RMSE
MAE
RMSE
MAE
Traditional
LSIM
2.28 + 0.26 1.82 + 0.20
4.30
0.78 2.67 + 0.23 0.51 0.12
0.41 + 0.09
RGRN
1.81 + 0.05
1.30 + 0.03
7.88 + 0.86 2.80 + 0.10
0.95 + 0.01
0.76 + 0.03
Gr-CNN
1.86 + 0.14
1.43 0.12
8.77 0.10 3.11 + 0.06
0.34 + 0.03
0.18 + 0.05
Graph-based
HydroNets
1,87 = 0.31
1.41 = 0.20
7.21
+ 0.35 3,02 3.02:0001
0.18 + 0.01
0.21 = 0.01
HRGN-DA
1.87 0.01 1.40 0.20
6.49 0.56 2.19
0.19
0.03
0.22 + 0.01
Multimodal
LITE
1.59 = 0.10
1.26 = 0.08
1.89 0.11 0.84 + 0.04
0.08 0.01
0.06 + 0.01
Watershed (CRW), while only 16 river segments in which have streamflow observations. The number of streamflow observations available for each segment ranges from 16 to 4, 900 with total of 63, 501 observations across all dates and segments. The temporal span and dataset partition strategy of CRW-Flow are the same as CRW-Temp.
AGR. AGR is an agricultural nitrous oxide (N₂O) emission prediction dataset acquired from controleedenvviromee used to plant continuous corn during 2015-2018 and monitor the N₂O emission hourly. The number of N2O emission observations av available for each chamber ranges from 5,780 to 6, 168 with total of 35, 711 observations.
mesocosm facility in Minnesota, US. Six chambers were
Baselines. For all datasets, include five state-of-the-art baselines for environmental ecosystems modeling, including LSTM (Hochreiter & Schmidhuber 1997), Graph-temporal convolutional network (Gr-CNN) (Sun et al. 2021), PPhysics-guided Recurrent Graph model (RGRN) (Jia et al.) 2021), Hydronets (Moshe et al. 2020), and Heterogeneous Recurrent Graph Networks with data assimilation (HIRGN-DA) (Chen al. 2021).
Implementation Details. Following (Chen et al. 2021 jia et al. 2021) we use RMSE and MAE evaluate the performance of all methods. The detault loo--back window size in temporal trend image is set to 30 for all three datasets. Moreover, we utilize the pre-trained distilberr-bas--uccseed model (Sanh al. (2019) as the semantic time-series encoder and the tiny Swin Transformer model (Liu et al.) 2021a) as our vision encoder. The hidden dimension of the two pre-trained models 768. Additionally, Llama-2-7b-hf (Touvron) et al. 2023) is selected as the default backbone of our LLM decoder. For the sparsely -gatd mixture-of-experts (SMoE) layer, each expert is a feed-forward network with hidden units of 1768,3072, 768), the number of experts is and top experts will be selected by the router. We train our model via the AdamW optimizer with an initial learning rate of 0.00001. All the experiments are implemented in PyTorch with one NVIDIA RTX A6000 GPU.
5.2 Main Results
Table ssoows the performance of different approaches for predicting stream water tempera- ture, streamflow, and agricultural nitrous oxide (N2O) emission. Firstly, it can be observed that the graph neural networks-based based methods, e.g., HRGN-DA(Chen sistently outperforms the traditional approaches by a significant margin, demonstrating the importance of accurate modeling of non-stationary ST correlations. This observation is not surprising, since the distribution shifts in correlations of physical variables happen frequently, and will make learned models irrelevant. Second, LITE consistently outperforms the previous SOTA methods across all environmental domains, confirming its effectiveness in unifying environmental spatial-temporal prediction by representing data from multimodal perspective.
al. 2021), con-
7

---

Published as a conference paper at COLM 2024
Table 2: Results of ablation study, which demonstrates the impact of different components on the overall performance of our model.
CRW-lemp
CRW-Flow
AGR
RMSE
MAE
RMSE
MAE
RMSE
MAE
LITE-text
2.97 = 0.09
2.53 + 0.09
2.35 = 0.12
1.20 + 0.04
0.12 0.02
0.10 = 0.01
LITE-image
1.99 + 0.04
1.66 1 0.04
2.29 1 0.08
0.96 1 0.03
0.15 - 0.01
0.13 = 0.01
LITE-LLM
1.69 0.04
1.44 0.04
2.21 +: 0.09
0.97 1 0.03
0.15 - 0.00
0.13 = 0.00
LITE-imp
1.65 0.09
1.32 0.07
1.14
0.91 1 0.07
0.07
1.02
0.12 = 0.02
LITE-SMoE
1.84 0.07
1.98 + 0.06
0.88 + 0.04
0.11 0.00
0.14 0.00
LITE-mtg
2.09 + 0.08
1.76 0.07
2.31 + 0.08
0.93 + 0.06
0.10 = 0.01
0.09 = 0.00
LITE
1.59 + 0.10
1.26 + 0.08
1.89 + 0.11
0.86 + 0.04
0.08 + 0.01
0.06 0.01
5.3
Ablation Studies
In this subsection, we conduct comprehensive ablation studies to demonstrate the effective- ness of our key modules, including the semantic time-series encoder, the temporal trend image, the SMoE for incomplete feature imputation, and the LLM decoder to perceive multimodal information. The ablation models include:
LITE-text: In LITE-text, the semantic time-series encoder is removed and the representa- tion from the vision encoder is directly fed into the LLM decoder.
LITE-image, the vision encoder removed and the representation from the semantic time-series encoder is directly fed into the LLM decoder.
LITE-image:
LITE-LLM: In LITE-LLM, the LLM decoder is replaced with a linear layer.
LITE-imp: In LITE-imp, the imputation module is removed. The special tokens (e.g., [MASK]) will not be imputed, leaving the input physical variables incomplete.
LITE-SMoE: In LITE-SMoE, we replace the SMoE layer with linear layer to impute incomplete features in semantic time-series encoder. Different characteristics physical variable are not taken into account in the imputation step.
LITE-mtg: In LITE-mtg, we do not incorporate multi-granular features from previous time in the language modality. Thus, data patterns of different granularities are not assimilated, which could hinder the ability to handle the common distribution shifts in environmental data.
The results are shown in Table 2 and the results of LITE are also reported for comparison. From this table, we observe that (1) LITE outperforms all the variants without certain com- ponents, e.g., LITE-text, LITE-image, and LITÉ-LLM, which demonstrates the effectiveness and the complementary nature of time-series and temporal trend modalities. This also confirms the effectiveness of multiple components in LITE, including the imputation for incomplete features, the incorporation of multi-grame nualtigranullrrtty information, and the domain- aware multimodal ST prediction with LLM. (2) LITE-SMoE performs worse than LITE, which implies the significance of using different experts for imputing different missing phys- ical variables. Neglecting the heterogeneity among diverse variables will cast negative impact on performance. LITE slightly outperforms LITE-imp, which shows the importance of imputing missing features. (3) LITE significantly outperforms LITE-mtg. This result is not surprising, as the incorporation of multi- mullt-granularity information indeed enhanced the model's 's ability to handle the temporal distribution shift.
5.4 Performance Under Leaving-Sensors-Out Settings
To evaluate our model in more challenging conditions, we conduct further experiments where a subset of features are missing during testing. This could commonly occur in real scenarios due to the difficulty in measuring certain physical variables or sensor malfunctions. Following the previous approach Zhang et al. (2022), we carried out assessments under two settings with the CRW-Temp dataset: (1) Iileev--iieedds-essssooool which selects fixed
8

---

Published as a conference paper at COLM 2024
HRGN-DA
80656
MNE
RMSSE
MAE
HydroN ets
Gr-CNN
RGRN
8.0
25.0
6.0
25.0
0.0
12.5 Ratio
25.0
25,0 Masking Rabio (8)
Masking Ratio
Masking Ratio
Figure 3: Experimental results under the leave-fixed-sennsossoutt (left two) and leave- sendooosensors-out (right two) settings on CRW-Temp dataset.
set of sensors (i.e., features) and hides all their measured values in the test set, and (2) sevee-radoo--ssoos----
which randomly selects sensors and hide all their measured values in the test set. See more detailed descriptions of both settings in Appendix A.4 In both settings, we set the missing ratio from 12.5% to 50.0%.
We present our results in Figure which demonstrates our LITE model consistently out- performs all the baselines by significant margin, especially under extreme conditions, e.g., when half of the physical variables are missing. We can observe that with the missing ratio increasing from 0% to 25% most models' performance exhibits stability. However, under extreme conditions, i.e., 25%-50% of the features are missing, only our LITE model maintains strong performance, with the other methods showing significant performance drop (higher RMSE and MAE). Specifically, when increasing the missing ratio from 12.5% to 50% the prediction RMSE of LITE only increases by 25%, while the average RMSE increase ratio of the other baselines 185%. These results justifies the robustness of LITE to varying degrees of incomplete features, which are common in environmental ecosystem data.
5.5 Analysis on Domain Generalization Ability
In this subsection, we further test LITE jus- tify its robustness to distribution shifts in envi- ronmental data. Specifically, we often need to simulate environmental ecosystems across dif- ferent regions, and these regions can exhibit di- verse patterns due to their distinct system char- acteristics, e.g., soil and groundwater properties, weather conditions, etc. In this test, aim to generalize the model to test domains (regions) Dlest that are disjoint from the training domains (regions) train i.e., Divain n Diest = @. For ex- ample, on the CRwp dataset, we train each model on three river segments that have the most observations and test on the remain- ing river segments. We present the experimental results in Table 3 can seen that our LITE shows superiority under out-of- out-of-distributioo (OOD) testing. This result verifies LITE's robustness to distribution shift that frequently happens in environmental ecosystems, demonstrating its potential general framework for environmental ST prediction.
Method
RMSE
MAE
LSTM
7.10 0.06
5.90 + 0.06
RGRN
2.71 0.03
2.08 0.04
Gr-CNN
2.41 0.20
2.30
0.16
HydroNets
1.85 0.12
1.52 1: 0.03
HRGN-DA
1.73 0.16
1.33 0.10
LITE
1.41 0.23
1.31 1: 0.18
Table Prediction RMSE and MAE un- der out-of-distribution regions.
Conclusion
This paper presents LITE, multimodal Large Language Model for environmental Ecosys- tem Modeling. By transforming the spatial-temporal data into semantic time-series and temporal trend images, we could effectively learn multimodal representations of environ- mental data from pre-trained foundation models. We employ a Sparse Mixture-of- Experts (SMoE) layer to impute the incomplete observations and incorporate multi-granularity information to handle the distribution shifts in environmental data. Additionally, we uti- lize domain instructions to guide frozen LLM to fuse multimodal representation. Our extensive experiments demonstrate the effectiveness and robustness of our LITE method, particularly with varying degrees of missing features and distribution shifts.

---

Published as a conference paper at COLM 2024
Acknowledgement
We thank Google Cloud Research Credits program for supporting our computing needs.
References
Rianne van den Berg, Thomas N Kipf, and Max Welling. Graph convolutional matrix completion. Xiv preprint ar arXiv:1706.02263 2017.
Shengyu Chen, Alison Appling, Samantha Oliver, Hayley Corson-Dosch, Jordan Read, Jeffrey Sadler, Jacob Zwart, and Xiaowei Jia. Heterogeneous stream-reservoir graph networks with data assimilation. In 2021 IEEE International Conference on Data Mining (ICDM), Pp. 1024-1029, IEEE, 2021.
Andrea Cini, Ivan Marisca, Filippo Maria Bianchi, and Cesare Alippi. Scalable spatiotem- poral graph neural networks. In Proceedings of the AAAI conference on artificial intelligence, volume 37, PP. 7218-7226, 2023.
Ethan Coon, Daniil Svyatsky, Ahmad Jan, Eugene Kikinzon, Markus Berndt, Adam Atch- ley, Dylan Harp, Gianmarco Manzini, Eitan Shelef, Konstantin Lipnikov, Rao Garimella, Chonggang Xu, David Moulton, Satish Karra, Scott Painter, Elchin Jafarov, and Sergi Molins. Advanced terrestrial simulator. [Computer Software] https://doi.on 11578/dc.2011 20190911.
sep 2019. URL httpss://di..rrg/// 11578/ dc. 20190911.1
Durdu Ömer Faruk. hybrid neural network and arima model for water quality time series prediction. Engineering applications of artificial intelligence, 23(4):586-594, 2010.
Lovedeep Gondara and Ke Wang. Mida: Multiple imputation using denoising autoencoders. In Advances in Knowledge Discovery and Data Mining: 22nd Pacific-Asia Conference, PAKDD 260.260-22...
2018, Melbourne, VIC, Australia, June 3-6, 2018, Proceedings, Part III 22, pp 260 272. Springer, 2018.
lan Goodfellow, Mehdi Mirza, Aaron Courville, and Yoshua Bengio. Multi-prediction deep boltzmann machines. Advances in Neural Information Processing Systems, 26, 2013.
RF Grant, Andrew Black, Rachhpal Jassal, and Christian Bruemmer. Changes in net ecosystem productivity and greenhouse gas exchange with fertilization of douglas fir: Mathematical modeling in ecosys. Journal of Geophysical Research: Biogeosciences, 115(G4), 2010.
Paul C Hanson, Aviah B Stillman, Xiaowei Jia, Anuj Karpatne, Hilary A Dugan, Cayelan C Carey, Joseph Stachelek, Nicole K Ward, Yu Zhang, and Jordan S Read. Predicting lake surface water phosphorus dynamics using process-guided machine learning. 430:109136.
ISSN 0304-3800.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, (8):1735-1780, 1997.
Scott Jasechko, Hansjörg Seybold, Debra Perrone, Ying Fan, Mohammad Shamsudduha, Richard G Taylor, Othman Fallatah, and James W Kirchner. Rapid groundwater decline and some cases of recovery in aquifers globally. Nature, 625(7996):715-71,, 2024.
Xiaowei Jia, Ankush Khandelwal, David J Mulla, Philip Pardey, and Vipin Kumar. Bring- ing automated, remote-sensed, machine learning methods to monitoring crop landscapes at scale. Agricultural Economics, 50:41-50, 2019.
Xiaowei Jia, Jacob Zwart, Jeffrey Sadler, Alison Appling, Samantha Oliver, Steven Mark- strom, Jared Willard, Shaoming Xu, Michael Steinbach, Jordan Read, et al. Physics-guided recurrent graph model for predicting flow and temperature in river networks. In Proceed- ings of the 2021 SIAM International Conference on Data Mining (SDM), pp. 612-620. SIAM, 2021.
10

---

Published as a conference paper at COLM 2024
Fei Li, Jingfeng Xiao, Jiquan Chen, Ashley Ballantyne, Ke Jin, Bing Li, Michael Abraha, and Ranjeet John. Global water use efficiency saturation due to increased vapor pressure deficit. Science, 381(6658) 2023.
Zekun Li, Shiyang Li, and Xifeng Yan. Time series as images: Vision transformer for irregularly sampled time series. Advances in Neural Information Processing Systems, 36, 2024.
LIU Licheng, Wang Zhou, Zhenong Jin, Jinyun Tang, Xiaowei Jia, Chongya Jiang, Kaiyu Guan, Bin Peng, Shaoming Xu, Yufeng Yang, et al. Estimating the autotrophic and heterotrophic respiration in the us crop fields using knowledge guided machine learning. In AGU Fall Meeting 2021. AGU, 2021.
Hangchen Liu, Zheng Dong, Renhe Jiang, Jiewen Deng, Jinliang Deng, Quanjun Chen, and Xuan Song. Spatio-temporal adaptive embedding makes vanilla transformer sota for traffic forecasting. In Proceedings of the 32nd ACM international conference on information 2023.
and knowledge management, pp.
Licheng Liu, Shaoming Xu, Jinyun Tang, Kaiyu Guan, Timothy J Griffis, Matthew D Erickson, Alexander I. Frie, Xiaowei Jia, Taegon Kim, Lee Miller, et al. Kgml-ag: modeling machine learning to simulate agroecosystems: case
framework of knowledge-guide study of estimating n o emission using data from mesocosm experiments. Geoscientific Model Development, 15(7):2839-288 2858, 2022.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. CoRR, 2021a.
14030
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 10012-10022, 2021b.
Shiyuan Luo, Juntong Ni, Shengyu Chen, Runlong Yu, Yiqun Xie, Licheng Liu, Zhenong Jin, Huaxiu Yao, and Xiaowei Jia. Free: The foundational semantic recognition for modeling environmental ecosystems. Xiv preprint ar Xix:2331. 2023.
Mengmeng Ma, Jian Ren, Long Zhao, Sergey Tulyakov, Cathy Wu, and Xi Peng. Smil: Multimodal learning with severely missing modality. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 2302-2310, 2021.
Mengmeng Ma, Jian Ren, Long Zhao, Davide Testuggine, and Xi Peng. Are multimodal transformers robust to missing modality? In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 18177-18186, 2022.
Steven Markstrom, R Steve Regan, Lauren E Hay, Roland Viger, Richard M Webb, Robert A Payn, and Jacob H LaFontaine. Prms-iv, the precipitation-runoff modeling system, version 4. Technical report, US Geological Survey, 2015.
Zach Moshe, Asher Metzger, Gal Elidan, Frederik Kratzert, Sella Nevo, and Ran El- Yaniv. Hydronets: Leveraging river structure for hydrologic modeling. arXiv preprint arXiv:2007.00595 2020.
Erin O'Donnell, Melissa Kennedy, Dustin Garrick, Avril Horne, and Rene Woods. Cultural water and indigenous water science. Science, 38166653):119-61,, 2023.
Emily K Read, Lindsay Carr, Laura De Cicco, Hilary A Dugan, Paul C Hanson, Julia A Hart, James Kreft, Jordan S Read, and Luke A Winslow. Water quality data for national-scale aquatic research: The water quality portal. Water Resources Research, 53(2):1735-1745 2017.
Jordan S Read, Xiaowei Jia, Jared Willard, Alison P Appling, Jacob A Zwart, Samantha K Oliver, Anuj Karpatne, Gretchen JA Hansen, Paul C Hanson, William Watkins, et al. Process-guided deep learning predictions of lake water temperature. Water Resources Research, 55(11):9173-9999 2019.
11

---

Published as a conference paper at COLM 2024
R Steven Regan, Steven L Markstrom, Lauren E Hay, Roland J Viger, Parker A Norton, Jessica M Driscoll, and Jacob H LaFontaine. Description of the national hydrologic model for use with the precipitation-runnff modeling system (prms). Technical report, US Geological Survey, 2018.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, distilled version of bert: smaller, faster, cheaper and lighter. ArXiv, abs/ 0bss/190.00108, 2019.
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture- of-experts layer. arXio preprint ar Xfo::7701.06638, 2017.
Marek Smieja, Lukasz Struski, Jacek Tabor, Bartosz Zieliñski, and Przemyslaw Spurek. Processing of missing data by neural networks. Advances in neural information processing systemss, 31, 2018.
Indro Spinelli, Simone Scardapane, and Aurelio Uncini. Missing data imputation with adversarially-traine graph convolutional networks. Neural Networks, 129:249-225 260, 2020.
Raghavan Srinivasan, Xuesong Zhang, and Jeffrey Arnold. Swat ungauged: hydrological budget and crop yield predictions in the upper mississippi river basin. Thansactions of the ASABE, 53111:333165 2010.
Jiahao Su, Wonmin Byeon, Jean Kossaifi, Furong Huang, Jan Kautz, and Anima Anandku- mar. Convolutional tensor-train Istm for spatio-temporal learning. Advances in Neural Information Processing Systems, 33:13714-13726 2020.
Alexander Y Sun, Peishi Jiang, Maruti K Mudunuru, and Xingyuan Chen. Explore spatio- temporal learning of large sample hydrology using graph neural networks. Water Re- sources Research, 57(2))e2021 2021.
US Geological Survey. National water information system data available on the world wide web (usgs water data for the nation), 2016.
Giovanni Tamburini, Riccardo Bommarco, Thomas Cherico Wanger, Claire Kremen, Mar- cel GA Van Der Heijden, Matt Liebman, and Sara Hallin. Agricultural diversification promotes multiple ecosystem services without compromising yield. Science advances, 6 (45)eabb1771,, 2020.
Song Tang, Chuang Li, Pu Zhang, and RongNian Tang. Swinlstm: Improving spatiotemporal prediction accuracy using swin transformer and lstm. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 13470-13479, 2023.
Xianfeng Tang, Huaxiu Yao, Yiwei Sun, Charu Aggarwal, Prasenjit Mitra, and Suhang Wang. Joint modeling of local and global temporal dynamics for multivariate time series forecasting with missing values. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 5956-5963, 2020.
Fred D Theurer, Kenneth A Voos, and William J Miller. Instream water temperature model. instream flow information paper 16. Technical report, repoor, US Fish and Wildlife Service, 1984.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. Xrv preprint arXin:2307.09928 2023.
Huaxiu Yao, Fei Wu, Jintao Ke, Xianfeng Tang, Yitian Jia, Siyu Lu, Pinghua Gong, Jieping Ye, and Zhenhui Li. Deep multi-view spatial-temporal network for taxi demand prediction. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
Huaxiu Yao, Yiding Liu, Ying Wei, Xianfeng Tang, and Zhenhui Li. Learning from multiple cities: A meta-learning approach for spatial-temporal prediction. In The world wide web conference, pp. 2181-2191, 2019a.
12

---

Published as a conference paper at COLM 2024
Huaxiu Yao, Xianfeng Tang, Hua Wei, Guanjie Zheng, and Zhenhui Li. Revisiting spatial- temporal similarity: deep learning framework for traffic prediction. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pp. 5668-5675, 2019b.
Jinsung Yoon, James Jordon, and Mihaela Schaar. Gain: Missing data imputation using generative adversarial nets. In International conference machine learning, pp. 5689-5698. PMLR, 2018.
Wojciech Zaremba, Ilya Sutskever, and Oriol Vinyals. Recurrent neural network regulariza- tion. arviv preprint erXiv:1409.2333 2014.
Xiang Zhang, Marko Zeman, Theodoros Tsiligkaridis, and Marinka Zitnik. Graph-guided network for irregularly sampled multivariate time series. In International Conference Learning Represenlations, 2022.
Wang Zhou, Kaiyu Guan, Bin Peng, Jinyun Tang, Zhenong Jin, Chongya Jiang, Robert Grant, and Symon Mezbahuddin. Quantifying carbon budget, crop yields and their responses to environmental variability using the ecosys model for us midwestern agroecosystems. Agricultural and Forest Meteorology, 307:108521, 2021.
Monika Zurek, Aniek Hebinck, and Odirilwe Selomane. Climate change and the urgency to transform food systems. Science, 37666600):111--1422,, 2022.
A Appendix
A.1 Prompt
We provide detailed descriptions and examples of prompts used in this paper.
A.1.1 Prompt to Guide Semantic Time-series Generation
In Sec 3.1. we utilize a prompt to instruct an LLM to transform the linearized data point to natural language, where the prompt consists of: prefix to describe the schema of the input features, the linearization and a suffix We detail the prompt below.
Table 4: Prompt for generating semantic time-series.
System Message
You are helpful assistant.
Prompting
Here is the schema definition of the table: $schema, _dffiittiiiiiiitionnn
This a sample from the table: Slinearization
Please briefly summarize the sample with its value in one sentence.
You should describe the important values, like rainfall and average cloud cover fraction, instead of just the names of columns in the table.
A brief summarization of another sample may look like: "On November 14, 2009, the rainfall in the Delaware River Basin was 0.00152 inches. The average air temperature was [MASK] degrees Celsius. The solar radiation recorded was 125.31325 units. The average cloud cover fraction was 0.4352."
Note that the example is not the summarization of the sample you have to summarize.
Response
$ samaaati.iieeserre.......eeeeeee
A.1.2 Domain Instruction
Here, we provide two examples for the domain instruction we used to guide the frozen LLM, which consists of three parts: dataset description, task description, and target statistics.
13

---

Published as a conference paper at COLM 2024
|start-prompt] Dataset description: The Christina River Watershed Flow (CRW- Flow) is a dataset containing streamflow observations from 16 river segments. It is worth noting that the streamflow becomes hundreds of times higher than usual when it rains. Task description: predict the stremflow given the observed meteorological features represented in the image and text spaces; Target statistics: min value (min_values sstr),,, max value amax_valuee... sstr), median value (median_ siliiianvaluesstryy the trend of input is [trend] < end prompt
The Agriculture nitrous oxide (AGR) is a dataset containing agricultural nitrous oxide emission observations from 6 chambers. Task de- scription: predict the nitrous oxide emission given the observed meteorological features represented in the image and text spaces; Target statistics: min value (min.values values str), max value (max values str), median value (median values str}, the trend of input is (trend) < < end_prompt
|start-prompii Dataset description:
A.2 Pseduocodes of training algorithm
The pseuduocodes of training algorithm is shown in Alg.
Algorithm 1 Training algorithm of LITE
Input: Training dataset D {x yra)
2: step 1. Transforming the data into semantic time series and temporal trend image;
3: for +- to do
4: Zr, < LLM Lineariza(xx;))..
5: ir.1 6 x
6: end for
7: step 2. Train LITE;
8: for each minibatch B in dataset D do
9: for t +- to batch size b do
10:
impute the missing variables with SMoE and obtain a complete semantic time-series 2r, as Equation 4
U₁ < (z,,,,) Fi (1,,
12:
O₂ F
13: Fuse multimodal representations Equation (10)
14: end for
15:
end for
16: Compute the losses L, Ci and Call following Equation
11
12
Optimize the parameters 0 of LITE by minimizing Cal!
A.3 Dataset Description
In this subsection, we provide detailed descriptions for the three datasets we used in experiments, i.e., CRW-Temp, CRW-Flow, and AGR.
CRW-Temp. This is a stream water temperature dataset originating from the Delaware River Basin, area recognized for its ecological diversity and significant watershed on the United States' east coast. It originates from the U.S. Geological Survey's National Water Information System (Survey (2016) and the Water Quality Portal (Read al. 2017) which is the most extensive standardized dataset for water quality in both inland and coastal waters. The data focuses on specific geographic coordinates, aligning observations with river segments that range in length from 48 to 23, 120 meters. These segments are delineated according to the national geospatial framework utilized by the National Hydrologic Model (Regan et al. 2018). To ensure observations are accurately associated with the corresponding river segments, we align observations to the nearest stream segment within 250-meter tolerance. It is worth noting that: 1) For segments with more than one observation site, data was consolidated into a singular daily average water temperature; 2) Observations located more than 5, 000 meters along the river channel
14

---

Published as a conference paper at COLM 2024
from the end of a segment were excluded from the dataset. Our study focuses specific portions of the Delaware River Basin, particularly those that merge into the main stem of the Delaware River in Wilmington, DE. We refer to this large dataset as the Christina River Watershed (CRW), which includes 42 river segments. The temporal span of the dataset extends from October 31, 2006, to March 30, 2020, covering 4, 900 days. For given region and date f, the daily meteorological features are: the day of the year, rainfall, daily average air temperature, solar radiation, average cloud cover fraction, groundwantr temperature, subsurface temperature, and potential eoapotranspiration Considering that all physical variables (both features and targets) are sourced from sensors, they display various levels of sparsity. Overall, the dataset contains 12, 506 valid temperature observations.
CRW-Flow. CRW-Flow is a streamflow prediction dataset that is also collected on the Christina River Watershed (CRW). The temporal span, meteorological features, and all other default settings of CRW-Flow are the same with CRW-Temp. The only difference is that the target of CRW-Temp to predict streamflow instead of stream water temperature. There are in total 63, 501 valid observations of streamflow in CRW-Flow.
AGR. AGR is an agricultural nitrous oxide (N₂O) emission prediction dataset acquired from controlled-environmeen mesocosm facility in Minnesota, US. Six chambers with a soil surface area of 2 m² and column depth of 1.1 m were used to plant continuous corn during 2015-2018 and monitor the N₂O emission hourly (mgNm 1/2 1) with a N₂O analyzer (Teledyne M320EU, Teledyne Technologies International Corp, Thousand Oaks, CA). The number of N₂O emission observations av available for each chamber ranges from 5,780 to 6, 168 with a total of 35, 711 observations. We apply Z-normalization to the data from AGR before feeding it into our model.
A.4 Leave-Sensors-Out Settings
Here, we provide detailed description of two leave-sensors-out settings we adopt on CRW- Temp dataset, i.e, leave-fixed-sensorss
and leaverandoom-sensos-oo
ot For a fair comparison, we hided the same set of sensors across all methods. Specifically, selected features from CRW-Temp, which are rainfall, average cloud cover fraction, groundwater temperature, and subsurface temperature. The selection process was incremental, starting with the exclusion of one feature and gradually expanding to encompass all four.
Leave-random-sensoos-oo Under this setting, features in the test set are randomly dropped to introduce variability. Same with the wwwre--xeddseesors--eee dropping one random feature, and finally drop four random features. This stochastic feature omission simulates real-world scenarios where data may be missing unpredictably, thus providing further insights into our model's adaptability under less controlled conditions.
setting, we will start with
15


Reviews:
[{"review_id": "e4jUAWw7la", "reviewer": "Reviewer_hAX4", "summary": "The authors present LITE, a multimodal Large Language Model for environmental Ecosystem\nModeling. LITE uses spatial-temporal data and transforms these into two data modalities: semantic time-series and\ntemporal trend images. These transformed data are then encoded to learn multimodal representations of environmental data from pre-trained foundation models. The work addresses gaps frequently arising in real-world data: by imputing the incomplete observations; and incorporating multi-granularity\ninformation to handle data distribution shifts in environmental data. \nA decoder, in the form of a frozen LLM, is used to fuse these multimodal representations. The authors demonstrate the feasibility and generalizability of LITE on real-world environmental data.\n\nThe motivation, ideas and previous literature are well discussed, but there are points that need further clarification.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable and detailed reviews.\n\n**Variables** in the CRW-Temp dataset includes rainfall, air temperature, etc. For instance, the rainfall on a given day can influence the air temperature.\n\n**Eqn 3, 5, 9.** Example of Eqn 3: time: 2017-01-07, radiation: 0.5, air temperature: 25, precipitation: 1.2, spRH: 0. For Eqn 5, please refer to the fourth to second lines before it. In Eqn 9, we encode the temporal trend image with a swin transformer.\n\n$\\mathbf{x_{r,t}}$ is a data point, which has K features.\n\n**Notations about ‘missing variables’**. Please refer to the first sentence after Eqn 7, and lines 4 and 10 of Algorithm 1.\n\n**V in Eqn 6** is the input to the TopK function, i.e., the representation of each missing variable.\n\n**E in Eqn 7** is the shared expert models which consists of E expert models.\n\n**Figure 2** is an example of a single $i_{r,t}$ in the CRW-Temp dataset, which contains the observations of 9 variables over the past 30 days ($\\beta=30$). From left to right and top to bottom, the variables follow the same order as listed in lines 14 to 16 on page 14. The x-axis represents timestamps for all subplots, while the y axis specifies the values of each variable in a subplot. The units of the y-axis in different subplots are different, so we apply z-normalizaition on all variables before plotting the temporal trend image.\n\n**LLMs in Eqn 4 and 10** are GPT-3.5-turbo and LLaMA2-7b, respectively. We will modify the notation of LLM in Eqn 4 in the future version.\n\n**Eqn 10 in the algorithm** is in line 13.\n\n**Linear layer after Eqn 10** is: $(Dropout(Q_{r, t}W_{1}+b_{1}))W_{2}+b_{2}$. It is used to transform the output representation from the LLM ($d=d_{LLM}$) to the final prediction ($d=1$).\n\n$\\eta_{1}$  **and** $\\eta_{2}$ **in the loss function** are used to balance the regression loss and the imputation loss, which is much higher than the former one. The hyperparameters of $\\eta_{1}$ and $\\eta_{2}$ are decided through cross validation. This is a very common practice in literature (e.g., [1, 2]).\n\n[1] Sener et al. “Multi-task learning as multi-objective optimization.” NeurIPS’18.\n\n[2] Bhattacharjee et al. “Mult: An end-to-end multitask learning transformer.” CVPR’22.\n\n**Figure 3.** From left to right, the first two are fixed settings while the second two are random settings.\n\n**Average RMSE increase ratio** of the baselines is: $[(5.92-2.35)/2.35+(6.35-1.87)/1.87+(7.01-2.58)/2.58+(6.94-2.49)/2.49]/4=185$%. We will fix this typo.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is well written, but does need tidying up. Please see comments.", "reasons_to_reject": "None", "questions_to_authors": "The motivation, ideas and previous literature are well discussed, but there are points that need further clarification. In general; a) the functional forms of Equations are missing; and b) a table with variables and their usage/function in the Appendix will be a good reference since there are a handful of variables that are not defined. \n\nPlease note these points below.\n\n1.\tPage 2: 2nd paragraph. Please provide examples of ‘variables’, and therefore ‘inter-variable correlations’. One could use the same environmental application to provide this.\n2.\tEqns 3, 5, 9; what are these functions? Provide a functional form or a reference.\n3.\tComparing Eqns 3 and 2: Is \\x_{r,t} the same as a data point or feature value or physical driver or an observation? Can these terms be used interchangeably? Provide additional clarification on these definitions.\n4.\tAnother term that caused confusion is ‘missing variables’. On page 4, the first line mentions that missing variables are replaced. After Eqn 5, there is mention that missing variables are imputed using SMoE, as well as in the Algorithm 1. If both procedures are done, then update the text and Algorithm accordingly. Please clarify. \n5.\tEqn 6. What is v (parameter to TopK)?\n6.\tEqn 7. What is E?\n7.\tFigure 2: needs additional details. Is this a single i_{r,t} or a group of them shown?\n* What variable is being plotted\n* Label what each subplot is\n* what the x-axis and y-axis are\n* are the axes the same for each subplot\n* is {\\beta} = 9 which is why there are 9 subplots\n* are these z-normalized values?\n8.\tWhat is the difference between the LLMs in Eqn 4 and 10\n9.\tThe algorithm is still incomplete, for example, it does not mention Eqn 10 or steps needing Eqn 10.\n10.\tLine immediately after Eqn 10: ‘we employ a linear layer on the top of output representation….”. What is the functional form of this layer, and why was it needed? \n11.\tWhat is the role of \\eta_1 and \\eta_2 in the Loss function?\n* What does it mean to ‘balance loss terms’? \n* Do these have constraints?\n* where there any bounds to the loss function?\n12.\tLegend in Fig 3 is not clear. Which curves relate to the fixed versus random sensors? \n13.\tIs the average RMSE increase in ratio ‘223%’?", "extra_scores": {}}, {"review_id": "x3s3XTwiLg", "reviewer": "Reviewer_cTdF", "summary": "This paper proposes a new hybrid arch to model environmental ecosystems, predicting physical variables such as water temperatures in a given region at a specific time step. The authors first prompt LMs to transform input features into natural language descriptions, then employ a semantic time-series encoder to encode these descriptions. These text features are fused with image features as well. The arch allows for imputation of missing features and the authors further integrate multi-granularity information to mitigate distribution shift issues. Finally, all these features are fed into a frozen llama LLM to make the final prediction. Comprehensive experiments and ablation analysis demonstrate the effectiveness of the approach.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable reviews and for acknowledging the novelty and effectiveness of our approach.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Applying LLMs to spatial-time series data prediction in the environmental context is very interesting and potentially impactful for society. The authors fuse both text and image features that are novel.  \n2. The main results are quite good, greatly improving previous baselines that are kind of traditional.  \n3. While many designs zoom into specific details, the authors try to conduct comprehensive ablation analysis to justify the effectiveness of each component. For example, I appreciate the ablation experiments that focus on the settings with missing features and distribution shifts.", "reasons_to_reject": "I don’t have any significant concerns for this paper.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DL0LLTHMUc", "reviewer": "Reviewer_DnLc", "summary": "For handling incomplete features and distribution shifts, this paper proposes Multimodal Large Language Model for EnvIronmenTal\nEcosystem Modeling (LITE) to better capture and process ST dynamics and correlations from a multimodal perspective. The experiments\ndemonstrate that LITE significantly enhances performance in environmental spatial-temporal prediction.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable reviews and we have conducted experiments according to your suggetions.\n\n**Latest baseline.** While the modeling of environmental ecosystems is crucial to people’s livelihood, this area is under-explored. We summarize the past efforts at the accurate modeling of environmental ST data and propose a novel multimodal LLM-based method that can serve as a new strong baseline and provide useful insight for future research.\n\n**Replace LLM decoder with LM.** We have conducted experiments using a smaller LM - distilbert, to replace the frozen LLM (LITE-LM).\n\n|  | CRW-Temp |  | CRW-Flow |  | AGR |  |\n| --- | --- | --- | --- | --- | --- | --- |\n|  | RMSE | MAE | RMSE | MAE | RMSE | MAE |\n| LITE-LM | 1.98 | 1.57 | 2.96 | 1.11 | 0.15 | 0.14 |\n| LITE | 1.59 | 1.12 | 1.89 | 0.86 | 0.08 | 0.06 |\n\nAlthough we tune the LM, it performs worse than the LLM version. This is due to its limited ability to fuse the multimodal representations under domain instructions. In contrast, the LLM in LITE is frozen and performs much better.\n\n**Week and year data.** In LITE-year, we keep only the week data. In LITE-week, we keep only the year data. We also include results of LITE-mtg (remove week and year data) and LITE for comparison.\n\n|  | CRW-Temp |  | CRW-Flow |  | AGR |  |\n| --- | --- | --- | --- | --- | --- | --- |\n|  | RMSE | MAE | RMSE | MAE | RMSE | MAE |\n| LITE-year | 1.65 | 1.33 | 2.02 | 0.88 | 0.09 | 0.08 |\n| LITE-week | 1.87 | 1.53 | 2.29 | 0.92 | 0.13 | 0.11 |\n| LITE-mtg | 2.09 | 1.76 | 2.31 | 0.93 | 0.10 | 0.09 |\n| LITE | 1.59 | 1.12 | 1.89 | 0.86 | 0.08 | 0.06 |\n\nOur observations are: (1) both week and year data contribute to the superior performance of LITE; (2) week data is of greater significance than year data in terms of the above benchmarks. (3) combing week and year data, which is our multi-granularity information, yields best performance.\n\n**Linearized data to natural language?** Linearized data lacks the smoothness of natural language, which is important for modeling the inter-variable correlations in the semantic space.\n\n**How to find missing variables?** For any missing variable, like air temperature, we represent it as “air temperature: None” in the linearized data. After transforming it into natural language, we replace “None” with a special token, [MASK], for further imputation.\n\n**The role of imputation loss** is to guide the imputation of missing variables. When we remove it in LITE-imp, this lead to a performance drop.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The proposed method is effective;", "reasons_to_reject": "1. The  latest baselin is in 2021, which is relatively old. Why are there no updated baselines?\n2. In ablation Studies, because the difference between the parameters of a linear layer and Llama-2-7b is too great, the effect of LLM, the role of LLM is not prominent enough. If the LLM decoder is replaced with small LM, it can better reflect the role of LLM.\n3. In Multi-granularity information, The contribution of the two types of data, week data and year data, is unclear.", "questions_to_authors": "1. Why dose transform the linearized data point to natural language?\n2. How to find missing variables after transforming the linearized data point to natural language?\n3. What is the role of imputation loss？What if delete it？", "extra_scores": {}}, {"review_id": "BFoAYIlS1N", "reviewer": "Reviewer_CCyC", "summary": "This work proposed a multi-model LLM framework for modeling time series data.\nTwo different representations 1) text description 2) trend image and a domain description are combined to represent each instance for prediction. In order to deal with various challenges in the environmental ecosystem data 1)  incomplete features 2) distribution shifts, mixture of experts and multi-granularity features are used.\n\nExperiment on three datasets (water temp, stream flow, agricultural N2O) shows significant improvement over various SOTA approaches. Ablation study shows the importance of each component and especially the text description, the trend image and multi-granularity features.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable and insightful reviews.\n\nIn our ablation studies, we have explored some simplified versions of LITE, e.g., replacing the frozen LLM with a linear layer (LITE-LLM), and removing the multi-granularity information (LITE-mtg). However, such changes will compromise some performance degradation. From the experimental results, we observe the most necessary parts aside from language including vision (temporal trend image), multi-granularity information to handle distribution shifts, and imputation to handle incomplete observations. The other designs, e.g., the SMoE layer for imputation, can be removed or replaced under some circumstances. Our results in ablation studies can provide some useful guide for future research or applications.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This work shows the importance of combining image and text representation for modeling time series data. It can seem as an extension to the ViTST approach which only leverage the trend image representation.", "reasons_to_reject": "This is a fairly complex system. Although an ablation study is provided, I still feel that the paper can be stronger if alternative solution for each challenge can be discussed and compared to.", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces a multimodal LLM-based model for modeling environmental ecosystems, characterized by sequential data (e.g. series of observations and predictions over time), missing data, and multiple modalities (here discrete numerical observations that are transformed into images depicting trends and into natural language).  Reviewers generally found this work be well written and experimentally robust, with good ablation studies disentangling the important modeling contributions.  Overall this is solid work that illustrates how numerical data can be transformed into text and images to improve predictive accuracy.
Decision: Accept