LITE: Modeling Environmental Ecosystems with Multimodal Large Language Models

Authors: haoran  li
junqi  liu
zexian  wang
shiyuan  luo
xiaowei  jia
huaxiu  yao
Venue: colmweb_org_COLM_2024_Conference
Published: 2024-03-20 16:37:17

Abstract: The modeling of environmental ecosystems plays a pivotal role in the sustainable management of our planet. Accurate prediction of key environmental variables over space and time can aid in informed policy and decision-making, thus improving people's livelihood. Recently, deep learning-based methods have shown promise in modeling the spatial-temporal relationships for predicting environmental variables. However, these approaches often fall short in handling incomplete features and distribution shifts, which are commonly observed in environmental data due to the substantial cost of data collection and malfunctions in measuring instruments. To address these issues, we propose LITE -- a multimodal large language model for environmental ecosystems modeling. Specifically, LITE unifies different environmental variables by transforming them into natural language descriptions and line graph images. Then, LITE utilizes unified encoders to capture spatial-temporal dynamics and correlations in different modalities. During this step, the incomplete features are imputed by a sparse Mixture-of-Experts framework, and the distribution shift is handled by incorporating multi-granularity information from past observations. Finally, guided by domain instructions, a language model is employed to fuse the multimodal representations for the prediction. Our experiments demonstrate that LITE significantly enhances performance in environmental spatial-temporal prediction across different domains compared to the best baseline, with a 41.25\% reduction in prediction error. This justifies its effectiveness.

Main paper: 
 introduction: 
 Sustainable management of environmental ecosystems has been gaining massive attention due to its impact on global climate, and food and water security (Tamburini et al., 2020; O'Donnell et al., 2023; Li et al., 2023) . However, this task has become more challenging due to the rising demand for environmental services from the growing population, more frequent extreme weather events, and the climate change (Zurek et al., 2022; Jasechko et al., 2024) . The modeling of environmental ecosystems is critical for the sustainable management, as it provides important information about the spatial-temporal dynamics of key physical variables. For example, accurate prediction of streamflow and water temperature for large river basins can aid in the decision-making for water resource allocation and water quality control. An agricultural monitoring system can help develop policies that maintain farming efficiency and sustainability, and stabilize economies in agriculture-intensive areas.Physics-based models (Markstrom et al., 2015; Theurer et al., 1984; Zhou et al., 2021) and datadriven models (Zaremba et al., 2014; Hochreiter & Schmidhuber, 1997; Moshe et al., 2020; Jia et al., 2021) have been developed for spatial-temporal prediction in environmental applications. In particular, machine learning models (e.g., recurrent neural networks (Zaremba et al., 2014; Hochreiter & Schmidhuber, 1997) and graph neural network based models (Chen et al., 2021; Sun et al., 2021) ) significantly improve the performance of environmental spatialtemporal prediction by modeling the complex spatial-temporal correlations automatically from observed data. Despite the improved predictive accuracy, two key challenges remain unsolved. Firstly, existing methods exhibit vulnerability to incomplete features, which can be a common issue in environmental applications due to sensor failures and the high cost of field surveys needed to measure certain variables across different regions and time. Second, these prior approaches can be severely affected by data distribution shifts, which are commonly observed in environmental applications due to variations in human management and changes in weather conditions.To address these challenges, we propose a general large language model (LLM)-based framework, termed LITE, to handle incomplete features and distribution shifts behind the environmental data. Specifically, we first transform the spatial-temporal (ST) data into semantic time-series and temporal trend images. The semantic time-series reflects inter-variable correlations while the temporal trend image depicts temporal dynamics of all variables in the form of line graph images. In this step, we tackle the incomplete features by replacing the absent variables with special token (e.g., [MASK] ) in semantic time-series and linearly interpolating the temporal trend image. Then, LITE leverages a semantic time-series encoder and a vision encoder to jointly capture spatial-temporal dynamics and correlations. During this process, we incorporate multi-granularity information, i.e., merging previous observational data from different granularity such as week, month, and year to facilitate the handling of distribution shifts. Finally, the information, representing the spatial-temporal dynamics and dependencies, will be processed by a frozen LLM under the guidance of domain instructions, including dataset description, task description, and target statistics.The primary contribution of this paper is the development of the new framework LITE, which introduces an innovative multimodal representation learning framework for modeling environmental ecosystems. This approach not only exploits learned knowledge within foundation models to handle the heterogeneity of different environmental domains, but also presents consistent robustness to different levels of incomplete observations and distribution shifts. We evaluate LITE on three environmental spatial-temporal prediction scenarios: streamflow, water temperature, and agricultural nitrous oxide (N 2 O) emissions. For all scenarios, our empirical results demonstrate an improvement of 12.2%, 56.0%, and 55.6% in terms of RMSE, verifying the the effectiveness of LITE. Furthermore, we conduct experiments to demonstrate that the superiority of LITE remains or is even more significant under (1) data distribution shift; (2) extreme incomplete features, offering further justifications on the robustness and effectiveness of LITE. 
preliminaries: 
 In this section, we introduce the notations used in this paper and formally define the problem setup.Definition 1 (Spatial-Temporal Series): For a region r ∈ R in an environmental ecosystem, we denote the current timestamp as t, and the time range as a set T = {t -|T | + 1, ..., t} consisting of |T | evenly split non-overlapping time intervals. Then, the series of target physical variables y r,t (e.g., streamflow, water temperature, agricultural nitrous oxide (N 2 O)) over multiple spatial locations is represented asY t = {y r,t ′ |t ′ ∈ T , r ∈ R}.(1)Problem Definition. Given the input physical drivers X = {x r,t ′ |t ′ ∈ T , r ∈ R} for multiple regions until the current time t, our goal is to predict the spatial-temporal series of the target variable y r,t until the current time, asY t = f θ (X t ) (2)where f θ represents the ST model parameterized by θ, which predicts the target spatialtemporal series using input physical drivers. The input drivers are physical variables that can affect the state of target environmental ecosystems, e.g., meteorological and soil-related variables.  
multimodal large language model for environmental ecosystem modeling: 
 This section presents our proposed Multimodal Large Language Model for EnvIronmenTal Ecosystem Modeling (LITE). The goal of LITE is to capture ST dynamics and correlations from a multimodal perspective, and process this information within an LLM-based framework, as depicted in Figure 1 . To elaborate, we first transform the ST data into semantic time-series data and temporal trend images. Incomplete features are addressed by replacing missing variables with a special token (e.g., [MASK] ) in the semantic time-series, and by linearly interpolating in the temporal trend images. Subsequently, we utilize a semantic time-series encoder and a vision encoder to learn multimodal representations for the environmental ST data. To enhance the robustness to distribution shifts, we incorporate multi-granularity information from previous observations in this step. Finally, the multimodal representations are fused by a frozen LLM decoder to perform ST prediction under the guidance of domain instruction, i.e., dataset description, task description, and target statistics. Below, we detail each component of LITE.In this section, we first discuss how to learn representation from environmental data via semantic time-series, which includes three steps: (1) transforming environmental data into semantic time-series, (2) imputation of incomplete observations, and (3) multi-granularity information integration. We detail these steps below.Transforming Environmental Data into Semantic Time-Series. Given the varying types and quantities of features across different environmental datasets, we propose to first unify these features and transform them into corresponding semantic descriptions. Specifically, we first format each data point x r,t of K dimensions into a sequence of key-value pairs. For each pair, the key represents the feature description c k r,t , while the value is the corresponding feature value x k r,t . This can be formulated as follows:Linearize(x r,t ) = {[c k r,t : x k r,t ]} K k=1 (3)Then, we utilize a prompt to instruct an LLM to transform the linearized data point to natural language, where the prompt consists of a prefix p to describe the schema of the input features, the linearization, and a suffix s (see the detailed prompts in Appendix A.1.1). Given the prompt, the LLM will generate a readable and concise natural description z r,t , which can be formulated as:EQUATIONIt is worth noting that the missing variables will be replaced by a special token (e.g., [MASK] ) for further imputation.Imputation of Incomplete Observations. To handle the incomplete observation issue, we utilize a Sparse Mixture-of-Experts (SMoE) layer for imputing the incomplete observations. After transforming each data point into natural language description z r,t , we feed it into a semantic time-series encoder F l to get the encoded representations {m h r,t } H h=1 for each special token (i.e., incomplete feature), with H denotes the number of missing variables. The encoding process can be formulated as follows:EQUATIONSubsequently, to accommodate different characteristics of diverse physical variables, we apply a Sparse Mixture-of-Experts (SMoE) layer to impute the missing variables. Specifically, the encoded representation {m h r,t } H h=1 will be dynamically routed by an observationindependent noisy top-k gating network G to a subset of shared expert models {E e } E e=1 to facilitate the proper imputation of different observations. Following the original design of SMoE (Shazeer et al., 2017) , the gating process can be formulated as:G(m h r,t ) = Softmax(TopK(P (m h r,t ), k)) P (m h r,t ) = m h r,t • W g + µSoftplus(m h r,t • W noise ) TopK(v, k) j = v j , if v j is in the top k elements of v -∞, otherwise (6)where the µ is random noise sampled from a standard normal distribution, W g and W noise are learnable parameters. Then, the encoded representations m h r,t will be routed only to the shared expert models {E e } E e=1 with top-k gating scores generated by the gating network G. The predicted value m h r,t can then be calculated by combining the encoding results from the top-k expert models with their corresponding gating scores as:EQUATIONThen, we can replace the incomplete observations in z r,t with the corresponding predicted values { m h r,t } H h=1 , obtaining the complete data z r,t . Multi-granularity information integration. To address the distribution shift issue, we propose to incorporate multi-granularity information from previous observations. Specifically, for imputed time-series z r,t in region r at timestamp t, the incorporated multi-granularity information includes: (1) information from the past week in the same region, denoted asz w r,t = {z r,t | t ∈ {t -1, . . . , t -7}};(2) information from the same day of each month of the year in the same region, denoted as z y r,t = {z r,t | t ∈ {t -30, . . . , t -360}}. Subsequently, we feed the current observation z r,t into the semantic time-series encoder F l to get the current embedding U c r,t . Likewise, weekly information z w r,t and yearly information z y r,t are encoded into weekly embedding U w r,t and yearly embedding U y r,t , respectively. Finally, we concatenate all the multi-granularity information as the representation of semantic time-series to the LLM decoder, which can be formulated as:EQUATIONwhere the ⊕ is the concatenation operation along the representation dimension.Aside from learning representations from semantic time-series, we also propose to capture the dynamics of physical variables in the temporal trend image. This image depicts the recent temporal dynamics of all variables simultaneously, making the capture of intervariable relations more natural and easier. Specifically, we first convert the previous feature values into temporal trend images, which depict the temporal dynamics of all variables in the form of line graph images. Then, we utilize a vision encoder to encode the ST dynamics and correlations in the temporal trend image. These two steps are detailed below: Transforming environmental data into temporal trend image. Given the current observation x r,t , we utilize the line graph image to present each variables' (including the target variable's) observations in past β days in the same region, i.e.,x m r,t = {x r,t ′ | t ′ ∈ {t -1, . . . , t -β}}. For each variable in x m r,t , each point represents an observation identified by its time and value. As illustrated in Figure 2 , for each variable (one subimage), the horizontal axis represents timestamps, and the vertical axis specifies values. Here, incomplete features are linearly interpolated, and the line graph images of all variables will be finally aggregated into one temporal trend image i r,t . Following Li et al. (2024) , we apply Z-normalization on all variables due to the different scales of environmental variables.Image Recognition for temporal trend image. After transforming the observations into the temporal trend image i r,t , the temporal trend image i r,t is then embedded into a visual representation O r,t by the vision encoder F i , i.e., a pre-trained vision transformer (Liu et al., 2021b) :EQUATIONIt is worth noting that the temporal trend image presents two types of information for vision transformer: (1) the temporal dynamics of a single variable in a line graph, and (2) the spatial relationships amongst the variables across different line graphs. The vision encoder can explicitly capture the inter-variable and intra-variable dependencies.After obtaining representations from semantic time-series and temporal trend image, we employ an LLM-based framework to fuse multimodal representations for prediction. In order to adapt the LLM to different applications, we propose to guide the frozen LLM with domain instructions. Concretely, the domain instruction consists of three parts: (1) dataset description, which provides essential background information of specific environmental application;(2) task description, which guides the LLM to perform multimodal fusion for the given task;(3) target statistics, which serves as crucial information for LLM to capture the recent dynamics of the target variable (see Appendix A.1.2 for detailed descriptions of domain instruction). For each data point x r,t , we concatenate the learned representations from different modalities with domain instructions, and then forward them through a frozen LLM to obtain the output representation as:EQUATIONwhere d α represents the domain instructions for a environment α. Subsequently, we employ a linear layer on the top of output representation Q r,t to derive the final prediction y r,t .During training, we employ the prediction loss L r and the imputation loss L i of incomplete observations, which are defined as follows:EQUATION)where N denotes the total number of observed target variables, H r,t denotes the set of incomplete features in region r at time t, and M denotes the total number of incomplete features. The overall loss L all is defined as follows:L all = η 1 L r + η 2 L i (12)where η 1 and η 2 are coefficients to balance these loss terms. The whole algorithm for training is illustrated in Alg. 1 in Appendix A.2. 
related work: 
 Modeling of Environmental Ecosystems. To address the environmental challenges, physicsbased models have been commonly adopted in many areas such as hydrology and agriculture (Regan et al., 2018; Grant et al., 2010; Zhou et al., 2021) , specifically like multi-purpose terrestrial simulator (Coon et al., 2019) and hydrological budget and crop yield predictor (Srinivasan et al., 2010) . These models often rely on parameterizations and approximations due to incomplete knowledge or excessive complexity in modeling certain physical processes. As a result, they are often biased even after being calibrated using sufficient training data. As an alternative, advanced machine learning (ML) has been increasingly adopted to help understand the underlying complex physical processes and deal with large amounts of data. In particular, prior research has shown the promise of ML-based approaches in modeling agroecosystems (Jia et al., 2019; Licheng et al., 2021; Liu et al., 2022) and freshwater ecosystems (Read et al., 2019; Hanson et al.) , and more recently, encouraging results by large language models in modeling environmental ecosystems (Luo et al., 2023) .Both physics-based and ML-based models would benefit decision-making activities relevant to society, highlighting the intersection of environmental science, technology, and policy.Spatial-Temporal Prediction. The ST prediction serves as a crucial topic in spatial-temporal data mining. Earliest models, e.g., ARIMA (Faruk, 2010), only focus on temporal dynamics. Later, multiple CNN-based and RNN-based models are proposed to simultaneously capture spatial-temporal correlations (Su et al., 2020; Sun et al., 2021; Yao et al., 2018; 2019b; a) . Recently, with advanced deep learning methods such as graph neural networks and transformers, the performance on ST prediction has been significantly improved (Cini et al., 2023; Tang et al., 2023; Liu et al., 2023) . However, ST prediction for environmental ecosystems still suffers from incomplete observations and distribution shifts.Imputation of Incomplete Observations. One common issue in environmental data is the incomplete data, which is mainly caused by sensor failures and the high cost of measuring certain variables. To build a general framework for environmental ecosystems modeling, the first step is to properly handle the incomplete features. Previous research has proposed to address the incomplete observations by various data imputation methods (Berg et al., 2017; Gondara & Wang, 2018; Yoon et al., 2018; Spinelli et al., 2020) or by directly imputing the incomplete features when making decisions (Goodfellow et al., 2013; Śmieja et al., 2018; Ma et al., 2021; 2022; Tang et al., 2020) . Nevertheless, these methods do not accommodate the heterogeneity amongst different variables in the incomplete observations, potentially leading to uninformed guessing when there are various missing values in one feature. 
experiments: 
 In this section, we evaluate the performance of LITE, aiming to answer the following questions: Q1: Compared to previous methods, can LITE consistently perform well across different environmental science applications? Q2: Can the proposed modules (e.g., semantic time-series encoder) effectively improve performance? Q3: How does LITE perform when confronted with the challenge of incomplete features? Q4: Can LITE be robust to distribution shifts in environmental data?Datasets Descriptions. In this subsection, we will briefly introduce three datasets from environmental ecosystems and Appendix A.3 provides further details.• CRW-Temp. CRW-Temp is a stream water temperature prediction dataset, aiming to predict the daily average water temperature on a given day based on the observed physical variables on the same day. It is collected from the Christina River Watershed (CRW), which consists of 42 river segments. Water temperature observations are available for 22 segments on certain dates. The number of observations for each segment ranges from 1 to 3, 572 with a total of 12, 506 observations across all dates and segments. The temporal span of the dataset extends from October 31, 2006, to March 30, 2020, covering 4, 900 days. For the experiments, the first 2, 450 days are selected as the training split, while the subsequent 2, 450 days serve as the test split.• CRW-Flow. The target of CRW-Flow is to predict the streamflow of river segments based on the observed physical variables. This dataset is also obtained from the Christina River Baselines. For all datasets, we include five state-of-the-art baselines for environmental ecosystems modeling, including LSTM (Hochreiter & Schmidhuber, 1997) , Graph-temporal convolutional network (Gr-CNN) (Sun et al., 2021) , Physics-guided Recurrent Graph model (RGRN) (Jia et al., 2021) , Hydronets (Moshe et al., 2020) , and Heterogeneous Recurrent Graph Networks with data assimilation (HRGN-DA) (Chen et al., 2021) .Implementation Details. Following (Chen et al., 2021; Jia et al., 2021) , we use RMSE and MAE to evaluate the performance of all methods. The default look-back window size in temporal trend image is set to 30 for all three datasets. Moreover, we utilize the pre-trained distilbert-base-uncased model (Sanh et al., 2019) as the semantic time-series encoder and the tiny Swin Transformer model (Liu et al., 2021a) as our vision encoder. The hidden dimension of the two pre-trained models is 768. Additionally, Llama-2-7b-hf (Touvron et al., 2023) is selected as the default backbone of our LLM decoder. For the sparsely-gated mixture-of-experts (SMoE) layer, each expert is a feed-forward network with hidden units of {768, 3072, 768}, the number of experts is 8 and top 2 experts will be selected by the router. We train our model via the AdamW optimizer with an initial learning rate of 0.00001. All the experiments are implemented in PyTorch with one NVIDIA RTX A6000 GPU.Table 1 shows the performance of different approaches for predicting stream water temperature, streamflow, and agricultural nitrous oxide (N 2 O) emission. Firstly, it can be observed that the graph neural networks-based methods, e.g., HRGN-DA (Chen et al., 2021) , consistently outperforms the traditional approaches by a significant margin, demonstrating the importance of accurate modeling of non-stationary ST correlations. This observation is not surprising, since the distribution shifts in correlations of physical variables happen frequently, and will make learned models irrelevant. Second, LITE consistently outperforms the previous SOTA methods across all environmental domains, confirming its effectiveness in unifying environmental spatial-temporal prediction by representing spatial-temporal data from a multimodal perspective. In this subsection, we conduct comprehensive ablation studies to demonstrate the effectiveness of our key modules, including the semantic time-series encoder, the temporal trend image, the SMoE for incomplete feature imputation, and the LLM decoder to perceive multimodal information. The ablation models include:• LITE-text: In LITE-text, the semantic time-series encoder is removed and the representation from the vision encoder is directly fed into the LLM decoder. • LITE-image: In LITE-image, the vision encoder is removed and the representation from the semantic time-series encoder is directly fed into the LLM decoder. • LITE-LLM: In LITE-LLM, the LLM decoder is replaced with a linear layer.• LITE-imp: In LITE-imp, the imputation module is removed. The special tokens (e.g.,[MASK]) will not be imputed, leaving the input physical variables incomplete. • LITE-SMoE: In LITE-SMoE, we replace the SMoE layer with a linear layer to impute incomplete features in semantic time-series encoder. Different characteristics of physical variable are not taken into account in the imputation step. • LITE-mtg: In LITE-mtg, we do not incorporate multi-granular features from previous time in the language modality. Thus, data patterns of different granularities are not assimilated, which could hinder the ability to handle the common distribution shifts in environmental data.The results are shown in Table 2 , and the results of LITE are also reported for comparison.From this table, we observe that: (1) LITE outperforms all the variants without certain components, e.g., LITE-text, LITE-image, and LITE-LLM, which demonstrates the effectiveness and the complementary nature of time-series and temporal trend modalities. This also confirms the effectiveness of multiple components in LITE, including the imputation for incomplete features, the incorporation of multi-granularity information, and the domainaware multimodal ST prediction with LLM.(2) LITE-SMoE performs worse than LITE, which implies the significance of using different experts for imputing different missing physical variables. Neglecting the heterogeneity among diverse variables will cast a negative impact on performance. LITE slightly outperforms LITE-imp, which shows the importance of imputing missing features. (3) LITE significantly outperforms LITE-mtg. This result is not surprising, as the incorporation of multi-granularity information indeed enhanced the model's ability to handle the temporal distribution shift.To evaluate our model in more challenging conditions, we conduct further experiments where a subset of features are missing during testing. This could commonly occur in real scenarios due to the difficulty in measuring certain physical variables or sensor malfunctions. Following the previous approach Zhang et al. (2022) , we carried out assessments under two settings with the CRW-Temp dataset: (1) leave-fixed-sensors-out, which selects a fixed set of sensors (i.e., features) and hides all their measured values in the test set, and (2) leave-random-sensors-out, which randomly selects sensors and hide all their measured values in the test set. See more detailed descriptions of both settings in Appendix A.4. In both settings, we set the missing ratio from 12.5% to 50.0%.We present our results in Figure 3 , which demonstrates our LITE model consistently outperforms all the baselines by a significant margin, especially under extreme conditions, e.g., when half of the physical variables are missing. We can observe that with the missing ratio increasing from 0% to 25%, most models' performance exhibits stability. However, under extreme conditions, i.e., 25%-50% of the features are missing, only our LITE model maintains strong performance, with the other methods showing significant performance drop (higher RMSE and MAE). Specifically, when increasing the missing ratio from 12.5% to 50%, the prediction RMSE of LITE only increases by 25%, while the average RMSE increase ratio of the other baselines is 185%. These results justifies the robustness of LITE to varying degrees of incomplete features, which are common in environmental ecosystem data. In this subsection, we further test LITE to justify its robustness to distribution shifts in environmental data. Specifically, we often need to simulate environmental ecosystems across different regions, and these regions can exhibit diverse patterns due to their distinct system characteristics, e.g., soil and groundwater properties, weather conditions, etc. In this test, we aim to generalize the model to test domains (regions) D test that are disjoint from the training domains (regions) D train , i.e., D train ∩ D test = ∅. For example, on the CRW-Temp dataset, we train each model on three river segments that have the most observations and test on the remaining river segments. We present the experimental results in Table 3 . It can be seen that our LITE shows superiority under out-of-distribution (OOD) testing. This result verifies LITE's robustness to distribution shift that frequently happens in environmental ecosystems, demonstrating its potential as a general framework for environmental ST prediction. 
conclusion: 
 This paper presents LITE, a multimodal Large Language Model for environmental Ecosystem Modeling. By transforming the spatial-temporal data into semantic time-series and temporal trend images, we could effectively learn multimodal representations of environmental data from pre-trained foundation models. We employ a Sparse Mixture-of-Experts (SMoE) layer to impute the incomplete observations and incorporate multi-granularity information to handle the distribution shifts in environmental data. Additionally, we utilize domain instructions to guide a frozen LLM to fuse multimodal representation. Our extensive experiments demonstrate the effectiveness and robustness of our LITE method, particularly with varying degrees of missing features and distribution shifts. impute the missing variables with SMoE and obtain a complete semantic time-series z r,t as Equation ( 4), ( 6), ( 7)11: U r,t ← F l ( z r,t ) ⊕ F l ( z w r,t ) ⊕ F l ( z y r,t ) 12:O r,t ← F i (i r,t )13:Fuse multimodal representations as Equation (10) 14:end for 15: end for 16: Compute the losses L r , L i and L all following Equation (11), (12) 17: Optimize the parameters θ of LITE by minimizing L all 
 
 Appendix: In this subsection, we provide detailed descriptions for the three datasets we used in experiments, i.e., CRW-Temp, CRW-Flow, and AGR.• CRW-Temp. This is a stream water temperature dataset originating from the Delaware River Basin, an area recognized for its ecological diversity and as a significant watershed on the United States' east coast. It originates from the U.S. Geological Survey's National Water Information System (Survey, 2016) and the Water Quality Portal (Read et al., 2017) , which is the most extensive standardized dataset for water quality in both inland and coastal waters. The data focuses on specific geographic coordinates, aligning observations with river segments that range in length from 48 to 23, 120 meters. These segments are delineated according to the national geospatial framework utilized by the National Hydrologic Model (Regan et al., 2018) . To ensure observations are accurately associated with the corresponding river segments, we align observations to the nearest stream segment within a 250-meter tolerance. It is worth noting that: 1) For segments with more than one observation site, data was consolidated into a singular daily average water temperature; 2) Observations located more than 5, 000 meters along the river channel from the end of a segment were excluded from the dataset. Our study focuses on specific portions of the Delaware River Basin, particularly those that merge into the main stem of the Delaware River in Wilmington, DE. We refer to this large dataset as the Christina River Watershed (CRW), which includes 42 river segments. The temporal span of the dataset extends from October 31, 2006, to March 30, 2020, covering 4, 900 days. For a given region r and date t, the daily meteorological features are: the day of the year, rainfall, daily average air temperature, solar radiation, average cloud cover fraction, groundwater temperature, subsurface temperature, and potential evapotranspiration. Considering that all physical variables (both features and targets) are sourced from sensors, they display various levels of sparsity.Overall, the dataset contains 12, 506 valid temperature observations.• CRW-Flow. CRW-Flow is a streamflow prediction dataset that is also collected on the Christina River Watershed (CRW). The temporal span, meteorological features, and all other default settings of CRW-Flow are the same with CRW-Temp. The only difference is that the target of CRW-Temp is to predict streamflow instead of stream water temperature.There are in total 63, 501 valid observations of streamflow in CRW-Flow.• AGR. AGR is an agricultural nitrous oxide (N 2 O) emission prediction dataset acquired from a controlled-environment mesocosm facility in Minnesota, US. Six chambers with a soil surface area of 2 m 2 and column depth of 1.1 m were used to plant continuous corn during 2015-2018 and monitor the N 2 O emission hourly (mgNm -1 h -1 ) with a N 2 O analyzer (Teledyne M320EU, Teledyne Technologies International Corp, Thousand Oaks, CA). The number of N 2 O emission observations available for each chamber ranges from 5, 780 to 6, 168 with a total of 35, 711 observations. We apply Z-normalization to the data from AGR before feeding it into our model.Here, we provide detailed description of two leave-sensors-out settings we adopt on CRW-Temp dataset, i.e, leave-fixed-sensors-out and leave-random-sensors-out.Leave-fixed-sensors-out. For a fair comparison, we hided the same set of sensors across all methods. Specifically, we selected 4 features from CRW-Temp, which are rainfall, average cloud cover fraction, groundwater temperature, and subsurface temperature. The selection process was incremental, starting with the exclusion of one feature and gradually expanding to encompass all four.Leave-random-sensors-out. Under this setting, features in the test set are randomly dropped to introduce variability. Same with the leave-fixed-sensors-out setting, we will start with dropping one random feature, and finally drop four random features. This stochastic feature omission simulates real-world scenarios where data may be missing unpredictably, thus providing further insights into our model's adaptability under less controlled conditions. 


Decision: This paper introduces a multimodal LLM-based model for modeling environmental ecosystems, characterized by sequential data (e.g. series of observations and predictions over time), missing data, and multiple modalities (here discrete numerical observations that are transformed into images depicting trends and into natural language).  Reviewers generally found this work be well written and experimentally robust, with good ablation studies disentangling the important modeling contributions.  Overall this is solid work that illustrates how numerical data can be transformed into text and images to improve predictive accuracy.

Reviews: review: paper_summary: paper_summary: The authors present LITE, a multimodal Large Language Model for environmental Ecosystem
Modeling. LITE uses spatial-temporal data and transforms these into two data modalities: semantic time-series and
temporal trend images. These transformed data are then encoded to learn multimodal representations of environmental data from pre-trained foundation models. The work addresses gaps frequently arising in real-world data: by imputing the incomplete observations; and incorporating multi-granularity
information to handle data distribution shifts in environmental data. 
A decoder, in the form of a frozen LLM, is used to fuse these multimodal representations. The authors demonstrate the feasibility and generalizability of LITE on real-world environmental data.

The motivation, ideas and previous literature are well discussed, but there are points that need further clarification.
strength_weakness: strength_weakness: None
questions: questions: The motivation, ideas and previous literature are well discussed, but there are points that need further clarification. In general; a) the functional forms of Equations are missing; and b) a table with variables and their usage/function in the Appendix will be a good reference since there are a handful of variables that are not defined. 

Please note these points below.

1.	Page 2: 2nd paragraph. Please provide examples of ‘variables’, and therefore ‘inter-variable correlations’. One could use the same environmental application to provide this.
2.	Eqns 3, 5, 9; what are these functions? Provide a functional form or a reference.
3.	Comparing Eqns 3 and 2: Is \x_{r,t} the same as a data point or feature value or physical driver or an observation? Can these terms be used interchangeably? Provide additional clarification on these definitions.
4.	Another term that caused confusion is ‘missing variables’. On page 4, the first line mentions that missing variables are replaced. After Eqn 5, there is mention that missing variables are imputed using SMoE, as well as in the Algorithm 1. If both procedures are done, then update the text and Algorithm accordingly. Please clarify. 
5.	Eqn 6. What is v (parameter to TopK)?
6.	Eqn 7. What is E?
7.	Figure 2: needs additional details. Is this a single i_{r,t} or a group of them shown?
* What variable is being plotted
* Label what each subplot is
* what the x-axis and y-axis are
* are the axes the same for each subplot
* is {\beta} = 9 which is why there are 9 subplots
* are these z-normalized values?
8.	What is the difference between the LLMs in Eqn 4 and 10
9.	The algorithm is still incomplete, for example, it does not mention Eqn 10 or steps needing Eqn 10.
10.	Line immediately after Eqn 10: ‘we employ a linear layer on the top of output representation….”. What is the functional form of this layer, and why was it needed? 
11.	What is the role of \eta_1 and \eta_2 in the Loss function?
* What does it mean to ‘balance loss terms’? 
* Do these have constraints?
* where there any bounds to the loss function?
12.	Legend in Fig 3 is not clear. Which curves relate to the fixed versus random sensors? 
13.	Is the average RMSE increase in ratio ‘223%’?
paper_summary: paper_summary: This paper proposes a new hybrid arch to model environmental ecosystems, predicting physical variables such as water temperatures in a given region at a specific time step. The authors first prompt LMs to transform input features into natural language descriptions, then employ a semantic time-series encoder to encode these descriptions. These text features are fused with image features as well. The arch allows for imputation of missing features and the authors further integrate multi-granularity information to mitigate distribution shift issues. Finally, all these features are fed into a frozen llama LLM to make the final prediction. Comprehensive experiments and ablation analysis demonstrate the effectiveness of the approach.
strength_weakness: strength_weakness: I don’t have any significant concerns for this paper.
paper_summary: paper_summary: For handling incomplete features and distribution shifts, this paper proposes Multimodal Large Language Model for EnvIronmenTal
Ecosystem Modeling (LITE) to better capture and process ST dynamics and correlations from a multimodal perspective. The experiments
demonstrate that LITE significantly enhances performance in environmental spatial-temporal prediction.
strength_weakness: strength_weakness: 1. The  latest baselin is in 2021, which is relatively old. Why are there no updated baselines?
2. In ablation Studies, because the difference between the parameters of a linear layer and Llama-2-7b is too great, the effect of LLM, the role of LLM is not prominent enough. If the LLM decoder is replaced with small LM, it can better reflect the role of LLM.
3. In Multi-granularity information, The contribution of the two types of data, week data and year data, is unclear.
questions: questions: 1. Why dose transform the linearized data point to natural language?
2. How to find missing variables after transforming the linearized data point to natural language?
3. What is the role of imputation loss？What if delete it？
paper_summary: paper_summary: This work proposed a multi-model LLM framework for modeling time series data.
Two different representations 1) text description 2) trend image and a domain description are combined to represent each instance for prediction. In order to deal with various challenges in the environmental ecosystem data 1)  incomplete features 2) distribution shifts, mixture of experts and multi-granularity features are used.

Experiment on three datasets (water temp, stream flow, agricultural N2O) shows significant improvement over various SOTA approaches. Ablation study shows the importance of each component and especially the text description, the trend image and multi-granularity features.
strength_weakness: strength_weakness: This is a fairly complex system. Although an ablation study is provided, I still feel that the paper can be stronger if alternative solution for each challenge can be discussed and compared to.

Comments: title: Thanks for adding more details
comment: Thanks for adding more details