SysCaps: Language Interfaces for Simulation Surrogates of Complex Systems
Authors: Patrick Emami
Zhaonan Li
Saumya Sinha
Truc Nguyen
Venue: ICLR 2025 Poster (2025)
Keywords: surrogate models
multimodal text and timeseries models
language-interfaced regression
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)
TL;DR: Augmenting surrogate regression models of complex systems simulations with natural language interfaces.

Abstract: Surrogate models are used to predict the behavior of complex energy systems that are too expensive to simulate with traditional numerical methods. 
Our work introduces the use of language descriptions, which we call "system captions" or SysCaps, to interface with such surrogates. 
We argue that interacting with surrogates through text, particularly natural language, makes these models more accessible for both experts and non-experts.
We introduce a lightweight multimodal text and timeseries regression model and a training pipeline that uses large language models (LLMs) to synthesize high-quality captions from simulation metadata. 
Our experiments on two real-world simulators of buildings and wind farms show that our SysCaps-augmented surrogates have better accuracy on held-out systems than traditional methods while enjoying new generalization abilities, such as handling semantically related descriptions of the same test system.
Additional experiments also highlight the potential of SysCaps to unlock language-driven design space exploration and to regularize training through prompt augmentation.

Published as a conference paper at ICLR 2025
SYSCAPS: LANGUAGE INTERFACES FOR SIMULATION SURROGATES OF COMPLEX SYSTEMS
Patrick Emami, Saumya Sinha," Truc Nguyen' National Renewable Energy Lab [Patrick. Emami, Saumya Saauuya..inbal sinha, Truc. Nguyen) @nrel gov
Zhaonan Li
Arizona State University zhaonan2@asu.c edu
ABSTRACT
Surrogate models are used to predict the behavior of complex energy systems that are too expensive to simulate with traditional numerical methods. Our work introduces the use of language descriptions, which we call "system captions" SysCaps, to interface with such surrogates. We argue that interacting with surrogates through text, particularly natural language, makes these models more accessible for both experts and non-experts. We introduce lightweight multimodal text and timeseries regression model and a training pipeline that uses large language models (LLMs)) to synthesize high-quality captions from simulation metadata. Our experiments on two real-world simulators of buildings and wind farms show that our SysCaps-augmented surrogates have better accuracy on held-out systems than traditional methods while enjoying new generalization abilities, such as handling semantically related descriptions of the same test system. Additional experiments also highlight the potential of SysCaps to unlock language- -riivnn design space exploration and to regularize training through prompt augmentation.
INTRODUCTION
Data-driven surrogates enable computational scientists to efficiently predict the results of expensive numerical simulations that run on supercomputers (Lavin et al., 2021; Carter et al., 2023). Surrogates are particularly valuable for emulating simulations of complex energy systems (CES), which model dynamic interactions between humans, earth systems, and infrastructure. Examples of CES include buildings (Vazquez-Cantell et al., 2019; Dai et al., 2023; Bhavsar et al., 2023), electric vehicle fleets (Vepsäläinen et al., 2019), and microgrids (Du Li, 2019). Advancing the science of CES contributes to efforts aimed at reducing emissions and improving the resiliency of power systems.
These surrogates perform a fairly standard regression task, predicting simulation output quantities of interest from a) input system configuration and b) deployment scenario. For example, might want to predict the amount of energy a building will consume given a) a list of building characteristics and b) weather timeseries spanning entire year. In this case, this involves performing long sequence timeseries regression, which traditional regression techniques such as gradient-boosted decision trees have difficulty with (Bhavsar et al., 2023; Zhang al., 2021).
Surrogate models are not only used by experts. Surrogates are also used to inform highly conse- quential policy and investment decisions about complex systems made by non-experts industry and governments (Rackauckas & Abdelrehim. 2024), such as when planning to build and deploy new renewable energy system (Harrison-Atlas et al., 2024). In this work, we design and analyze language interfaces for such surrogates. Intuitively, language interfaces make surrogate models more accessible, particularly for non-experts, by simplifying how we inspect and alter a complex system's configuration. Language interfaces are powerful-they ground interactions between humans and
"quual contribution.
Work done during internship NREI.

---

Published as a conference paper at ICLR 2025
machines in the human's preferred way (Vaithilingam et al., 2024). The idea of using language to create interfaces for complex data or models is not new (Hendrix et al., 1978; Quamar et al., 2022), but interest has renewed due to the success of large language models (LLMs) and their demonstrated ability to generate high-quality synthetic natural captions (Schick & Schütze, 2021; Doh et al., 2023; Mei et al., 2023a; Hegselmann et al., 2023). Our work defines a "system caption", or SysCap, as text-based descriptions of knowledge about the system being simulated. The only available knowledge our work assumes the system configuration found in simulator metadata files as lists of attributes.
In general, it is unknown whether textual inputs, and particularly natural language inputs, are suitable for real-world tabular regression tasks. Tabular data, such as the system attributes question, are sets of both discrete (categorical, binary, or string) and continuous (numeric) variables. Previous work demonstrated inconclusive evidence when using language models to tabular regression from text-encoded inputs, with and without modifications to the architecture (Dinh et al., 2022; Jablonka al., 2024; Bellamy et al., 2023; Yan et al., 2024), motivating further study. Regression with text- encoded tabular inputs is promising because a) language is a more intuitive and flexible user interface than traditional encoding strategies (e.g., one-hot encodings), and b) using language embeddings to encode system attributes unlocks the ability to exploit the semantic information contained in SysCaps to generalize across related systems.
Our paper introduces framework for training multimodal surrogates for CES with text (for system attributes) and timeseries inputs (for the deployment scenario) and makes contributions towards addressing the following technical challenges:
We introduce a simple and lightweight multimodal surrogate model architecture for time- series regression that a) fuses text embeddings obtained from fine-tuned language models (LMs) with b) timeseries encoded by bidirectional sequence encoder. We expect this to be insightful for future multimodal text and timeseries studies.
To address the lack of labelel natural language descriptions of complex systems, we describe a process that uses LLMs generate high-quality natural language SysCaps from simulation metadata. Although LLMs have previously been used to generate text captions from metadata (Doh al., 2023; Mei et al., 2023b), we believe our application to multi-modal surrogate modeling is novel.
We develop an automatic evaluation strategy to assess caption quality-specificallyy we estimate the rate at which ground truth attributes appear in the synthetic description with a multiclass attribute classifier.
Our experiments are based on two real-world CES simulators of buildings and wind farm wake. We rigorously evaluate accuracy on held-out systems and show that SysCaps-augmented surrogates have better accuracy than one-hot baselines. We also show generalization beyond the capabilities of traditional regression approaches enabled by the use of text embeddings. e.g., robustness to replacing attributes names with synonyms in test captions. We qualitatively show that text interfaces unlock system design space exploration via language. As there are no standard benchmarks for comparing surrogate modeling performance for CES, open-source all code and data at https: //github.
to facilitate future work.
2 RELATED WORK
Language interfaces for scientific machine learning: An increasing amount of work is exploring language interfaces for advanced scientific machine learning (SciML) models, including protein representation learning (Xu et al., 2023), protein design (Liu et al., 2023b), and activity prediction for drug discovery (Seidl al., 2023). I.L.-powered natural language interfaces are also heing designed for complicated scientific workflows including synchrotron management (Potemkin et al., 2023), automated chemistry labs (Bran et al., 2023), and fluid dynamics workflows (Kumar et al., 2023). We add to this body of work by studying language interfaces for lightweight surrogate models.
Large language models for regression: Another line of work asks whether LLMs can perform regression with both text inputs and outputs (numbers encoded tokens), such as for tabular problems (Dinh et al., 2022) or black-box box optimization (Song et al., 2024; Liu et al., 2024). We do not an LLM to do regression directly, but rather train a lightweight multimodal architecture that
2

---

Published as a conference paper at ICLR 2025
predicts continuous outputs instead of tokens. Moreover, study (Dinh et al., 2022) found mixed results when comparing to simple gradient-boosted tree baselines and highlighted difficulty with interpolation, raising questions about the effectiveness of LLMM-aseed regression.
Multimodal text and timeseries forecasting: Timeseries forecasting aims to predict future values of input timeseries given past values. Surrogate modeling can be cast forecasting problem when the goal is to train a model to emulate a dynamical system and predict its future behavior (e.g.. predicting future energy demand from past energy usage). Previous work has explored multimodal timeseries forecasting where auxiliary text data is introduced as covariates to improve the forecasting accuracy (Rodrigues et al., 2019; Emami al., 2023a; Jin al., 2024). Notably, Time-LLM (Jin et al., 2024) "reprograms" an LLM to process both text prompts and timeseries. However, our timeseries regression setting (Section 3) differs in that our surrogates are trained to map simulator inputs (e.g.. building characteristics and a weather timeseries) to simulator outputs (e.g., energy usage). In our problem, models critically depend on the system information encoded as text, whereas in Time-LLM the text only contains auxiliary information that slightly improves forecasting accuracy. This critical dependence partially motivates our development of a lightweight architecture for fusing text and timeseries embeddings. Also, recent evidence (Merrill et al., 2024; Tan et al., 2024) brings into question whether LLMs are useful at all for reasoning about temporal data.
Multimodal text and timeseries contrastive pretraining: Various efforts have explored contrastive pretraining objectives for modeling text and timeseries data (Agostinelli et al., 2023; Huang et al., 2022; Liu al., 2023a; Zhou al., 2023). For example, Agostinelli et al. (2023) aligns embeddings of captions that describe the audio. In our setting, captions describe system attributes, and the timeseries inputs are exogenous simulator inputs. Here, the text and timeseries inputs share no information with which to learn a shared embedding space.
Knowledge-enhaneed PDE surrogates: Numerical simulations of partial differential equations (PDEs) are computationally intensive; thus, much SciML aims to train PDE surrogates which are efficient evaluate on unseen inputs. Recent work tries to encode knowledge about the PDE into the surrogate to facilitate generalization within and across families of PDEs. Specifically, CAPE (Takamoto et al., 2023) and methods explored in Gupta & Brandstetter (2022) embed equation parameters (i.e., the system attributes) within the architecture to generalize unseen parameters. Others embed structural knowledge about the PDE into the surrogate model architecture (Rackauckas et al., 2020; Ye et al., 2024) or the loss (Raissi al., 2019). Concurrent work has explored "PDE captions" " (Lorsung et al., 2024; Yang et al., 2024), which are a type of SysCaps for neural PDE surrogates where the system knowledge is PDE equations encoded as text.
PROBLEM STATEMENT
Our goal is to learn a surrogate f X' x 2 - that regresses the outputs of a simulator F directly from its inputs. We are given a dataset D of pairs of simulator inputs and outputs. The inputs are deployment scenario F1.7 E X (a timeseries) and the tabular system attributes = E Z. The outputs timeseries V1:7 E V. For simplicity, we consider only univariate timeseries outputs in this work (ye E R). However, the number of timesteps T' may be large ( thousands of steps), the timeseries inputs X' are multivariate, and the mapping ff which approximates the simulator is highly nonlinear.
To summarize, we have timeseries regression problem modeled as y1.r = f(x1r; By condition- ing the surrogate on system knowledge 2, can potentially generalize to new system configurations. However, learning transferable representations of variable-length, heterogeneous input features such as z is notoriously difficult for deep neural networks, and is key focus of tabular deep learning (see survey by Badaro et al. (2023)). In our work, we develop and analyze a framework for learning multimodal surrogates where 2 is encoded as text.
Some simulators may have inputs that not clearly distinguishable into what is X and Z. for example, a dynamical system simulation is configured to be in steady-state or assumes fixed exogenous conditions. In these cases, allow X' to be vector of real-valued scalars (a timeseries with T = 1), or, simply an empty set (leaving only 2).
Example: In many CES, the timeseries X are exogenous inputs to the system such weather timeseries consisting of temperature or wind speed. Attributes of a wind farm might include the number of turbines in the wind farm and turbine blade length.
3

---

Published as a conference paper at ICLR 2025
Key-value template SysCaps
"A:1.0 B:Blue C:99"
A=
LLM for scalable natural language SysCaps training data generation
Timeseries simulator inputs
LLM
"This is blue system with type sub-system set to 1.0...
Training: Data generated from simulator metadata
Text Encoder (e.g. BERT)
Multimodal surrogate model
"A:7.0 B:Green C:99"
Trained jointly end-to-end for text- timeserles embedding alignment
"A green system whose type sub-system is assigned a value of 7.0..."
(b) Testing: Prompt the surrogate with descriptions of unseen systems
Our pipeline for augmenting multimodal simulation surrogates with language interfaces using "system captions", or SysCaps. SysCaps are text descriptions of knowledge about the system being simulated. In work, the SysCaps describe the system's characteristics, as found in simulation metadata files. During training (a), we create paired datasets of temporal simulator inputs with key- value template SysCaps LLM-gennrated natural language SysCaps. At test time (b), we prompt the surrogate model with one more key-value template captions or natural language captions. LLMs are only used generate synthetic training data; we use lightweight BERT-style text encoder and an efficient long-sequence encoder to keep the computational cost of our surrogate low.
Figure 1:
SYNTHESIZING SYSTEM CAPTIONS (SYSCAPS) WITH LLMS
Our work is motivated by the idea that language interfaces for surrogates represent path towards improving the accessibility these models for expert and non-expert users, e.g., when using them for downstream system design tasks (Vaithilingam et al., 2024). Our proposed framework for augmenting surrogates with language interfaces is visualized in Figure 1. During training. we create SysCaps out of system attributes specified in simulation metadata. To create large amounts of synthetic natural language SysCaps, we use LLMs. The ultimate goal is to enable scientists to "chat" with the multi-moda surrogate model at test time via prompting. In this section, we describe two approaches for converting system attributes into text: key-value templates and natural language.
For the key-value approach, attributes described key-value pairs key: value and joined by a separator "I" (SysCaps-kv). For example, if a simulation has attributes A=1.0 and B-blue, we create the string A: 1:1. B:blue. Generating these strings is easy to do and incurs negligible amount of extra computational overhead. In the natural language approach (SysCaps-nl, Figure 1), attributes are described in conversational manner, which we believe is more flexible and expressive than key-value captions and thereby more accessible for non-experts. However, we do not have access large quantities of natural language descriptions for each system and simulation. We avoid the Lime-consuming task of enlisting domain experts create this data by instead prompting powerful LLM to generate synthetic natural language descriptions given attributes. In our work, we the open-source LLM 1 11ama-22 -2----chatth (Touvron et al., 2023). The details of the prompt are provided next.
We append carefully written instruction template list of system attributes to help guide the LLM in generating a caption via prompting (see Figure 1). The system prompt is: You <CES> expert who provides <CES> descriptions <STYLES. The user prompt is: Write
Prompt design:
4

---

Published as a conference paper at ICLR 2025
SysCap (key-value template)
y2
Y4
"Plant Layout:
single string
Number of Turbines: 66 Mean
2
ho
Turbine Spacing: 5 Rotor
Diameter: 130.0 meters | Turbine Rated Power: 3.4 MW"
[CLS]
Bidirectional
9
Sequence Encoder
Text Encoder
g
text
SysCap (natural language)
"The wind plant consists 66 turbines arranged in single string configuration, each with rotor diameter of
32 1
2
3
44
power MW. The turbines are spaced at a distance of approximetely times the rotor diameter from each resulting
Mm
effective use the available wind resource. With this layout, the plant capable genereting significant and renewable energy."
Exagenous timeseries inputs
Figure 2: Building blocks of our surrogate model, = hg Iv, that includes multimodal encoder, Ge. and a top model, hg. The multimodal encoder, Рo 9g o text is a composition of a text encoder, gixxt, and bidirectional sequence encoder, S for timeseries inputs. The text embedding vector 2 is broadcasted (dashed lines) to create a sequence that is concatenated with the timeseries input. This multimodal sequence is the input to the sequence encoder.
<CES> description based on the following attributes. Your answer should be <NUM> sentences. Please note that your response should NOT be a list of attributes and should be entirely based on the information provided. The last part is added discourage the LLM from changing omitting attributes. The tags <CES>, <STYLE>, <NUM> are filled in with the CES type (e.g., buildings), the style of the description (e-g, with objective tone), and the number of sentences use in the description (e.g., "4-6"), respectively.
Attribute subset selection: Simulations of real-world systems may have attributes that only weakly correlate with the output quantity of interest, or have a large number of attributes, which can be challenging for deep learning approaches. Since the length of SysCap proportional to the number of attributes, the computational burden incurred by text- eext-based encodings of attributes can grow significantly in these cases. In these cases, reducing the number of attributes can be handled with classic feature selection methods such as recursive feature elimination (RFE) (Guyon et al., 2002) or by recommendations from domain experts, as a pre- pre-processing step.
TEXT AND TIMESERIES SURROGATE MODEL
We now describe lightweight multimodal surrogate model for timeseries regression, The surrogate (Figure 2) is a composition of multimodal encoder function Φy ()) - ( and top model he _ I. where for simplicity, the model parameters are shared timesteps to predict each timeseries output The training objective is to minimize the expected mean square error averaged over simulation timesteps,
1.r
(hol00 yi)²
min E (x,. 0.W
Although more sophisticated loss functions than Eq. 1 could be used to account for predictive uncertainty, we left this extension for future work to simplify our exposition and experiments.
Multimodal encoder gv: A text encoder gett extracts an embedding å from SysCap then broadcasts and concatenates this embedding with the timeseries inputs to create sequence of multimodal feature vectors. These features get processed by bidirectional sequence encoder 9p seq to
produce sequence of time-dependent fused multimodal features CI:T, e1.T 9 which are finally used to regress outputs.
(09 (g (1)xx(2) (2),
5

---

Published as a conference paper at ICLR 2025
Text encoder g To encode textual inputs use pretrained BERT (Devlin et al., 2018) and DistilBERT (Sanh et al., 2019) models that are relatively more efficient than LLMs. We use the model's default pretrained tokenizer. Tokenized sequences are bracketed by [CLS] and [EOS] tokens, and we use the final activation at the [CLS] token position to produce a text embedding 2 E TRo. Following standard fine-tuning practices, all layers for BERT are fine-tuned while only the last layer of DistilBERT is fine-tuned.
Bidirectional sequence encoder g " We broadcast the text embedding create sequence of length T, 2 + {= and concatenate each & with the timeseries input 21:7, { } This simplifies the task of learning timestep-specific correlations between system attributes & and timeseries x I in the multimodal encoder gv. To efficiently embed long timeseries with thousands of timesteps, we explore both bidirectional LSTMs (Hochreiter & Schmidhuber, 1997) and bidirectional SSMs (Goel et al., 2022) for 9g seq Our bidirectional SSM uses stacks of S4 blocks (Gu et al., 2021) without downpooling layers. We use the last layer's ss hidden states as temporal features 61:T for the top model. If = 1 or for non-sequential surrogate models, we instead use an MLP with residual layers (ResNet MLP) to embed each (2r; } per-timestep to get ez.
Top model hy: The multimodal encoder ge produces T feature vectors E1:T. For simplicity, the output dL at each timestep is predicted from by shared MLP with a single hidden layer.
6 EXPERIMENTS
This section presents our experimental results on two real-world CES simulators for buildings (Section 6.146.4) and wind farms (Section 6.5). Our experiments study the quality of LLM-ggnnrated SysCaps (Section 6. accuracy on held-out systems (Section 6.2), generalization under distribution shifts (Section 6.3), show a design space exploration application (Section 6.4), and examine SysCaps prompt augmentation (Section 6.5), All SysCaps are synthetically generated in this work. We provide additional qualitative examples of SysCaps in Appendix B.
For the main experiments in Section 6.1-6.4 we train building stock surrogate models for the building energy simulator EnergyPlus (Crawley et al., 2001). Given an annual hourly weather timeseries (T' 8,760) with 7 variables and list of tabular building attributes, surrogates predict the building's ss energy consumption at each hour of the year. Each building initially has 17 attributes which we reduce to 13 with RFE and LightGBM (Ke et al., 2017). We use commercial buildings from the Buildings-9000 900k dataset (Emami et al., Commmcccl building stock surrogates can provide significant speedups compared to EnergyPlus, e.g., 96% (Zhang et al., 2021). Since this dataset only provides energy timeseries, we manually extracted the building configuration and weather timeseries from the End-Use Load Profiles database (Wilson et al.) for each building. Our training set is comprised of 330K buildings, and use 100 buildings for validation and 6K held-out buildings for testing. We also reserved a held-out set of 10K buildings for RFE. We carefully tune the hyperparameters of all models (details in Appendix A.2).
Building stock simulation data:
We created three SysCaps datasets: a "medium" crattinn length dataset where <NUM> =446"" "short" dataset using 2-3 sentences, and a "long" dataset using 7-9 sentences. The SSMs in experiments are trained with medium captions. Generating these datasets with llama- 2-7b- chat used 1.5K GPU hours on a cluster with 16 NVIDIA A100-40GB GPUs.
6.1 EVALUATING CAPTION QUALITY
The LLM that generates natural language SysCaps may erroncously ignore or hallucinate attributes, which may negatively impact downstream performance. To test this, we propose evaluating generated captions by estimating the fraction of attributes which the LLM successfully includes per caption. To compute this metric, we train a multi-class classifier to predict each categorical attribute in a SysCaps from its text embedding. We used held-out validation set of captions to check that the classifier was not overfitting. The test rate of missing or incorrect attributes is around 9- 1.12% across the "short", "medium", and "long" caption types, with "short" captions having the highest error (Table 2). This increases our confidence that our LLM-based approach for generating natural language SysCaps preserves sufficient information for surrogate modeling.

---

Published as a conference paper at ICLR 2025
Table I: Accuracy. We show the mean NRMSE across random seeds. Lower NRMSE is better. quilding-hourte is the NRMSE normalized per building and per hour. Stock-annual first sums the predictions and targets over all buildings and hours, before computing the NRMSE (equivalent the normalized mean bias error-see Appendix A. When using BERT encoder instead of DistilBERT, SysCaps surrogates achieve better accuracy than one-hot baselines. We trained one SSM without any attribute information (Attribute Encoding "X"") as an ablation study; the poor accuracy shows that our multimodal architecture successfully learns to fuse the text-based attribute and timeseries inputs.
Buildings- (NRMSE)
Stock-Annua (NRMSE)
Model
Text Encoder
Attribute Encoding
SSM
X onchot
1.712 0003
0,658 0.008
SSM
0.450 0.19
0,041 # 0.021
LSTM
onehot
0.449 0.025
0.045 0.024
ResNet
onehot
0.634 0.09
0.072 0.008
LightGBM
onehot
0.679 0014
0.094 0.005
SSM
DistilBERT
SysCaps-nl
0.532 0.010
0.069 0.010
SSM
BERT
SysCaps-nl
0.543 0011
0.035 0.805
SSM
DistilBERT
SysCaps-kv
0,454 0012
0,046 0.003
SSM
BERT
SysCaps-kv
0.450 0007
0.020 0.012
LSTM
DistiBBERT
SysCaps-kv
0.489 0.021
0.063 0,005
LSTM
BERT
SysCaps-kv
0.439 0.037
0.022 0.011
ResNet
DistiBBERT
SysCaps-kv
0.633 0.00
0.08 = 0.011
ResNet
BERT
SysCaps-kv
0.670 0.43
0.049 0.015
the one-story warehouse has area 37, 37,500.0 square feet. It is equipped with PSZ-AC with electric coil system. the building opens at 9:15 AM on weekdaays and 9:00 AM weekends, and closes at 19:15 PM on
the one-story warehouse has total area of 37, 37,500.0 square feet.
a warehouse building.
weekdays 19:15 on weekends.
Warehouse (287)
(287)
(287)
120
120
Ground bouth Fredition
Ground brthh Fredittonn
110
100
100
kWh
80
Ground truth
Fredkt on
90
80
kWh.
100
90
70
50
50
60
20
5
10
15
20
5
10
15
20
Hour day
Hour of
of day
Figure 3: System captions unlock text-prompt-style surrogate modeling for complex systems. We show building stock daily load profiles aggregated for Warehouse building type, created with caption templates. From left to right, we use captions with one, three, and six attributes.
6.2 ACCURACY ON HELD-OUT SYSTEMS
In this section, we compare the accuracy of SysCaps surrogate models that vary by textual attribute encoding (key-value (-kv) and natural language (-nl)), text encoder (BERT, DistilBERT)) and sequence encoder (L.TTM, SSM). We also train SysCaps surrogates with non-sequential encoder (ResNet), without attribute inputs, and without text encoder fine-tuning, as ablations. Baselines are tuned LightGBM Gradient Boosting Decision Tree and a ResNet, LSTM, and SSM surrogate with onchot encoded attributes. Following Emami et al. (2023b), we use the normalized root mean square error (NRMSE) metric averaged across 3 random seeds.
Does the sequential architecture matter? Tes-Table 1 shows that the LSTM and SSM encoders outperform both the ResNet and our carefully tuned LightGBM bascline, and the SSM outperforms the L.T... How do different system attribute encoding approaches compare? Surprisingly,
7

---

Published as a conference paper at ICLR 2025
Table 2: Caption quality. We estimate the presence of each attribute in SysCap, measured by the average test accuracy of a multi-class classifier trained to predict each categorical attribute. Our metric suggests ~ 12% of at- tributes are missing or incorrect per SysCap, due to errors made by 1 1lama - 22-b-chatt
Table SysCaps zero-shot length generalization. NRMSE is per- building-hourly. Results are for the SSM model trained with medium- length SysCaps and evaluated zero-shot on short and long captions.
Caption length (13 attributes)
Accuracy (%)
SysCaps length
NRMSE
Short (2-3 sent.)
88.90
Short
0.57
Medium (4-6 sent.)
90.90
Medium
0.53 0.01
Long (7-9 sent.)
90.38
Long
0.64 0,02
Table 4: Generalization to attribute synonyms. We quantify how text embeddings make our models robust to the use attribute synonyms. The metric is the difference in NRMSE between the original caption and the modified caption. We bold the best method, i.c, closest to O. Column 3 replaces the building type with synonym, column removes the building type and sub-type attributes from the caption, and column 5 randomly swaps the correct building type attributes with incorrect ones.
With Synonym
Without Building Type
Random Building Type
Building Type
Synonym
Fuutterviceessturrat
FineDiningRestaurant
0.52 0.05
0.93 0.1
1.17a 0.7
RetailStripmall
ShoppingCenter
0.01 + 0.00
0.68 0.02
0.28 004
Warehouse
StorageFacility
0.35 0.30
0.55 0.31
4.02+ 0.32
RetailStandalone
SonvenienceStore
0.00 0.01
0.30 0.04
0.40+ oua
SmallOffice
Co- Co-WorkingSpace
0.03 0.01
0.02 0.00
1.95 0.30
PrimarySchool
Elementary-Schoo
0.00 0.01
0.38 0.00
0.52+ 0.17
MediumOffice
Workplace
0.08
0.03 + 0.04
0.91 0.11
Secondary School
HighSchool
-0.01 + 0.04
0.52+ +: 0.06
0.67 + 0.33
Outpatient
MedicalClinic
0.02
0.55 +: 0.09
0.32 0.06
QuickServiceRestaura
FastFoodRestaurant
0.10 0.07
0.83 0.01
0.85 nor
LargeOffice
TfficeTTower
0.12
0. 0.23 + 0.03
0.29 0.11
LorgeHotel
Five-Star Hotel
0.03 0.01
0,46+ 0.06
0,29 0.09
SmallHotel
Motel
0.26 0.07
0.88 0.07
0.70 0.14
Hospital
HealthcareFacility
0.03 0.04
0.62 0.12
0.20 0.07
SysCaps-kv achieves the best accuracy overall at the stock-annual aggregation level (cquivalent to the normalized mean bias error-see Appendix A.I) and comparable building-hourly accuracy one-hot baselines. The SysCaps-nl models have slightly worse accuracy than SysCaps-ky, yet they comfortably outperform the non-sequential models (including LightGBM) and the one-hot baselines at the stock-annual aggregation level. We initially expected to see a non-negligible drop in regression accuracy for SysCaps models, even key-value SysCaps, because the text encoder has to compress the caption into a single embedding vector, which causes information loss. However, the BERT encoder is expressive enough to mitigate this. We observe that about half when switching from DistilBERT to BERT. We believe the gap between key-value and natural language mostly explained by caption quality (Table 2). We ablate the importance of using attributes by training SSM baseline without attributes (Attribute Encoding "X"). This model is unable to learn this task. We trained a SysCaps-nl nl model without fine-tuning BERT, but it does poorly (stock-annual NRMSE of 0.356), showing the importance of fine-tur fine-tuning for multi-modal alignment.
NRMSE reduces by
Figure qualitatively shows how a SysCaps model performs with natural language captions (created with a sentence template) provided to the model. Note that we did not train our models on any captions with missing attributes. Prediction accuracy improves as more information is given; notably, there is large jump in accuracy once the building square footage is known.
8

---

Published as a conference paper at ICLR 2025
00.
0.0
0.4
0.2
0.2
0.0
1.0
1.0
0.8
0.8
5qft
# Of stories
10
20
0.2
of stories
10
20
0.2
25
0.0
25
0.0
(a)
(b)
Figure 4: Design space exploration using language. a) We show that the model has learned physically plausible relationship between building square footage (sqft) and number (#) of stories. b) Failure case: When tested on unseen values of sqft (blue crosses), the model's predictions appear to be physically implausible-the model underestimates the energy consumption at these sqft values.
6.3 CAPTION GENERALIZATION
Length generalization: We assess how accuracy varies when surrogates are provided with natural language SysCaps having different lengths than seen during training. We evaluate zero-shot general- ization to the short and long captions. The results (Table 3) show small increase in error for shorter captions with a larger increase in error for longer captions. However, the error on long captions remains lower than the error achieved by our tuned LightGBM baseline.
Attribute synonyms: To quantitatively evaluate the extent to which natural language SysCaps surrogates gain a level of robustness distribution shifts such as word order changes, synonyms, or writing style (Hendrycks et al., 2020), created captions for the held-out systems where the "building type" attribute is replaced by a synonym. We avoid biasing the choice of building type synonym by 5-shot prompting 11 11ama-2 suggest the synonyms. There two baselines compare the synonym caption accuracy against: 1) accuracy when testing the model on captions with the building type attribute removed, and 2) accuracy when testing the model captions with a random building type. Examples and results are shown in Table 4, where for 1//1 building type synonyms the increase in NRMSE is less than 13%, while the average increase for the baselines is 54% and 90%, respectively.
6.4 DESIGN SPACE EXPLORATION AETIONAAPPPLEEEE APPLICATION USING LANGUAGE
We visualize Figure 4a a use of SysCaps to conduct a sensitivity analysis on two system attributes, might be performed for an carly-stage design space exploration task. We use simple template to create a caption for each test building that enumerates all combinations of the number stories and square footage attributes, totaling 160 configurations; the entire analysis requires simulating 960K buildings, and took hour on a single NVIDIA A100 GPU. We observe that the model has indeed learned physically plausible relationships between these two attributes. However, the model fails to predict the energy usage for buildings over 100K square feet-such buildings are in the "long tail" of the training data distribution. Figure 4b also shows failure case where the model underestimates energy usage at unseen numeric building square footage attribute values.
6.5 PROMPT AUGMENTATION: WIND FARM WAKE
This experiment uses the Wind Farm Wake Modeling Dataset (Ramos et al., 2023), made with the FL. FLORIS simulator, to train surrogate to predict wind farm's 's power generation in steady-state

---

Published as a conference paper at ICLR 2025
atmospheric conditions. The speed-up provided by surrogate models for downstream optimization use-cases is ~~700 (Harrison-Atlas et al., 2024). The difficulty of this task is in modeling losses due to wake effects, given only coarse description of the wind farm layout. There are three numeric simulator inputs x specifying atmospheric conditions, and five system attributes which include categorical variables indicating wind farm shape (four different layout types), number of turbines, and average turbine spacing. We do not use RFE. In this dataset, there are only 500 unique system configurations (split 3:1:1 for train, val, test), although each configuration simulated under 500 distinct atmospheric conditions.
We explore generating multiple captions for each system con- figuration through prompt augmentation to increase diversity. Specifically, we replace the <STYLEF> tag the prompt with phrases encouraging different description styles, e.g., with an objective tone, with an objective tone (creative paraphrasing is acceptable), to colleague, and to classroom. The simulation is run assuming steady-state conditions (i.e., time-independent), so tune hyperparameters for and train the non sequentatt ResNet models. The ResNet baseline with one-hot encoded attributes suffers from severe overfitting (Table 5), likely due to the small number (300) of training systems, whereas the SysCaps models generalize better to unseen systems. This suggests SysCaps can have a regularizing effect in small data settings. Notably, the prompt augmentation helps the natural language SysCaps model to achieve the lowest NRMSE.
Table 5: Wind farm surrogate ac- curacy. The base architecture is ResNet. Average across 3 random seeds. SysCaps-nI# does not use prompt augmentation.
Model
NRMSE
LightGBM
0.19620.000
one-hot
0.212 0.009
SysCaps-ky
0.054= 0.024
SysCaps-n1*
0,038 # 0.001
SysCaps-nl
0.035 e: 0.001
7 CONCLUSION
In this work, we introduced a lightweight, multimodal text and timeseries surrogate models for complex energy systems such buildings and wind farms, and described process for using LLMs to synthesize natural language descriptions of such systems, which we call SysCaps. Our experi- ments showcase SysCaps-augmented surrogates that achieve better accuracy than standard feature engineering (e.g., one-hot encoding) while also enjoying the advantages of using text embeddings such as robustness to caption paraphrasing (e.g.. synonyms of attributes). For a problem with only a small number of training systems available, we showed that SysCaps-ni prompt augmentation has a regularizing effect that helps mitigate overfitting. Overall, these results underscore that language is a viable interface for interacting with real-world surrogate models.
Limitations:
Current BERT-style tokenizers struggle with numerical values (Wallace et al., 2019);
for one example, they interpolate poorly to unseen numbers (Figure 4b). For another, because 11am--2-77 chat tends to add a comma to large numbers (e.g.. 200, 000) when generating SysCaps. found that our surrogates failed to understand large numbers without commas (had high error). Orthogonal research on improving number encodings for language model inputs (Golkar al., 2023; Yan al., 2024) can benefit our framework. Another potential concern is with creating SysCaps for simulators with a large number of attributes (e.g., over 100). A more powerful LLM than 11ama- 2- chat with longer context window may be needed in this case. In general, we expect that more powerful LLMs will further improve the quality of the training captions.
Future work: A future extension of this work might explore how to use language to also interface with the timeseries simulator inputs, possibly through summary statistics. For example, to study how the complex system behaves when the average exogenous temperature is increased by five degrees. is natural to expect that non-experts may benefit more from our approach if the LLM is also instructed to simplify the simulator metadata or to provide explanations of technical concepts. Conducting interactive evaluations with non-experts will be important to obtain feedback for further improving the approach. Likewise, conducting a series of studies with domain scientists to evaluate the quality of SysCaps in the context of, e.g., system design optimization is promising. We did not conduct user studies in this work, as we first aimed to establish technical feasibility of this surrogate modeling important question is how might create surrogate foundation models that
approach. Finally, generalize not only across system configurations for a single simulator, but also generalize across different simulators.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work was authored by the National Renewable Energy Laboratory (NREL), operated by Alliance for Sustainable Energy, LLC, for the U.S. Department of Energy (DOE) under Contract No. DE- AC36-08G028308. This work was supported by the Laboratory Directed Research and Development (LDRD) Program at NREL. The views expressed in the article do not necessarily represent the views of the DOE or the U.S. Government. The U.S. Government retains and the publisher, by accepting the article for publication, acknowledges that the U.S. Government retains a nonexclusive, paid-up, irrevocable, worldwide license to publish or reproduce the published form of this work, or allow others do so, for U.S. Government purposes. The research was performed using computational resources sponsored by the Department of Energy's Office of Energy Efficiency and Renewable Energy and located at the National Renewable Energy Laboratory.
REFERENCES
Andrea Agostinelli, Timo Denk, Zalán Borsos, Jesse Engel, Mauro Verzetti, Antoine Caillon, Qingqing Huang, Aren Jansen, Adam Roberts, Marco Tagliasacchi, et al. Musiclm: Generating music from text. arXiv preprint arXiv:2301. 11000.1132,,,. 2023.
Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. Optuna: A next next-generation hyperparameter optimization framework. In Proceedings of the 25th ACM SIGKDD international conference knowledge discovery & data mining, pp. 2623 2631, 2019.
Gilbert Badaro, Mohammed Saeed, and Paolo Papotti. Transformers for tabular data representation: A survey of models and applications. Transactions of the Association for Computational Linguistics, 11:227-249, 2023.
David R Bellamy, Bhawesh Kumar, Cindy Wang, and Andrew Beam. Labrador: Exploring the limits of masked language modeling for laboratory data. arXiv preprint arXiv:2312.11502, 2023.
S. Bhavsar, R. Pitchumani, M. Reynolds. N. Merket, and J. Reyna. Machine learning surrogate of chysics-basse building-stockk simulator for end-use load forecasting, pp. 113395, 2023. ISSN 03787788. doi: 10. 10110016f.cbbuidd22223.333333331
URL https: htte::///iinnnnnnhiie wwt.:ghhh..eeeeieeeie. elsevier.
Andres M Bran, Sam Cox, Oliver Schilter, Carlo Baldassari, Andrew White, and Philippe Schwaller. Augmenting large language models with chemistry tools. In NeurlPS 2023 for Science Workshop, 2023.
Jonathan Carter, John Feddema, Doug Kothe, Rob Neely, Jason Pruet, Rick Stevens, prasanna Balaprakash, Pete Beckman, Ian Foster, Kamil Iskra, et al. Advanced research directions on ai for science, energy, and security: Report on summer 2022 workshops. 2023.
Drury B Crawley, Linda K Lawrie, Frederick C Winkelmann, Walter F Buhl, Y Joe Huang, Curtis O Pedersen, Richard K Strand, Richard Liesen, Daniel Fisher, Michael Witte, et al. Energyplus: creating new-generation building energy simulation program. Energy and buildings, 33(4): 319-331, 2001.
Ting- Yu Dai, Dev Niyogi, and Zoltan Nagy. Citytft: Temporal fusion transformer for urban building energy modeling. ArXiv preprint, abs/2312.02375,, 2023. URL WRLLttt/////xiv wwwp:iiivorrraaaaa abss/ 2312. 02375.
Jacob Devlin, Ming-We Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04800 2018.
Tuan Dinh, Yuchen Zeng, Ruisu Zhang, Ziqian Lin, Michael Gira, Shashank Rajput, Jy-yong Sohn, Dimitris Papailiopoulos, and Kangwook Lee. Lift: Language-interfaeed fine-tuning for non-language machine learning tasks. Advances in Neural Information Processing Systems, 35: 11763-11784, 2022.
SeungHeon Doh, Keunwoo Choi, Jongpil Lee, and Juhan Nam. LP-MusicCaps: LLMdd pseudo music captioning. ArXiv preprint, abs/2307.1633 16372, 2023. URL htttta://axxiv.ooo hwtp:/iii.rr///...../s.r..s/b 2307. 16372.
11

---

Published as a conference paper at ICLR 2025
Yan Du and Fangxing Li. Intelligent multi muiti-microorid energy management based deep neural network and model-free reinforcement learning. IEEE Transactions on Smart Grid, 11(2):1066 1076, 2019.
Hajar Emami, Xuan-Hong Dang, Yousaf Shah, and Petros Zerfos. Modality-aware transformer for time series forecasting, arXiv preprint arXiv:2310.01232, 2023a.
Patrick Emami, Abhijeet Sahu, and Peter Graf. Buildingshench: large-scale dataset of 900k build- ings and benchmark for short-term load forecasting. Advances in Neural Information Processing Systems, 2023b.
Karan Goel, Albert Gu, Chris Donahue, and Christopher Ré, It's raw! audio generation with state- space models. In Kamalika Chaudhuri, Stefanie Jegelka, Song. Csaba Szepesvári, Gang Niu, and Sivan Sabato (eds.). International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pp. 7616- 7633. PMLR, 2022. URL https: / pwwpeedinggggsoooo/o mlr www.ppessvveee..... /tee//oeeeeeeeee html.
Siavash Golkar, Mariel Pettee, Michael Eickenberg, Alberto Bietti, Miles Cranmer, Geraud Krawezik, Francois Lanusse, Michael McCabe, Ruben Ohana, Liam Parker, et al. xval: A continuous number encoding for large language models. arXiv preprint arXiv:2310.02989 2023.
Albert Gu. Karan Goel, and Christopher Ré. Efficiently modeling long sequences with structured state spaces. arXiv preprint arXiv:2111.000300 2021.
Jayesh K Gupta and Johannes Brandstetter. Towards multi-spatioeemporal-scaee generalized pde modeling. arXiv preprint arXiv:2209.1561/1 2022.
Isabelle Guyon, Jason Weston, Stephen Barnhill, and Vladimir Vapnik. Gene selection for cancer classification using support vector machines. Machine learning, 46:389-422, 2002.
Dylan Harrison-Atlas Atlas, Andrew Glaws, Ryan N King, and Eric Lantz, Artificial intelligence-aidee wind plant optimization for nationwide evaluation of land use and economic benefits of wake steering. Nature Energy, pp. 1-15, 2024.
Stefan Hegselmann, Alejandro Buendia, Hunter Lang, Monica Agrawal, Xiaoyi Jiang, and David Sontag. Tabllm: Few-shot classification tabular data with large language models. In International Conference Artificial Intelligence and Statistics, pp. 5549-5581. PMLR, 2023.
Gary G Hendrix, Earl D Sacerdoti, Daniel Sagalowicz, and Jonathan Slocum. Developing a natural language interface to complex data. ACM Transactions on Database Systems (TODS), 3(2): 105-147, 1978.
Dan Hendrycks, Xiaoyuan Liu, Eric Wallace, Adam Dziedzic, Rishabh Krishnan, and Dawn Song. Pretrained transformers improve out-of-distribution robustness. arXiv preprint vrXiv:2044.06100 2020.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):
1735-1780, 1997.
Qingqing Huang, Aren Jansen, Joonseok Lee, Ravi Ganti, Judith Yue Li, and Daniel P. W. Ellis. MuLan: A joint embedding of music audio and natural language, 2022. URL https: //axxiv.. org/abs/ 2208 12415.
Kevin Maik Jablonka, Philippe Schwaller, Andres Ortega-Guerreeoo and Berend Smit. Leveraging large language models for predictive chemistry. Nature Machine Intelligence, pp. 1-9, 2024.
Ming Jin, Shiyu Wang, Lintao Ma, Zhixuan Chu, James Y Zhang, Xiaoming Shi, Pin-Yu Chen, Yux- uan Liang, Yuan-Fang Li, Shirui Pan, and Qingsong Wen. Time-LLM: Time series forecasting by reprogramming large language models. In International Conference on Learning Representations (ICLR), 2024.
Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, and Tie-Ye Yan Liu. Lightgbm: A highly efficient gradient boosting decision tree. Advances in neural information processing systems, 30, 2017.
12

---

Published as a conference paper at ICLR 2025
Varun Kumar, Leonard Gleyzer, Adar Kahana, Khemraj Shukla, and George Em Karniadakis. Mycrunchept: A Ilm assisted framework for scientific machine learning. Journal of Machine Learning for Modeling and Computing, 4(4), 2023.
Alexander Lavin, David Krakauer, Hector Zenil, Justin Gottschlich, Tim Mattson, Johann Brehmer, Anima Anandkumar, Sanjay Choudry, Kamil Rocki, Atilim Günes Baydin, et al. Simulation intelligence: Towards a new generation of scientific methods. ArXiv preprint, abs/21 12..0333, 2021. URL wwtpt::t///./..... arxiv. www..oorg/bbs.... / 1112 03235.
Haohe Liu, Zehua Chen, Yi Yuan, Xinhao Mei, Xubo Liu, Danilo Mandic, Wenwu Wang, and Mark D Plumbley. Audioldm: Text-to-audio generation with latent diffusion models. ArXiv preprint, abs/2301.12503, 2023a. URL https::t///// /awixiiiiv......///aaa/a/aaaavva org/ abs 2301. 12503.
Shengchao Liu, Yutao Zhu, Jiarui Lu, Zhao Xu, Weili Nie, Anthony Gitter, Chaowei Xiao, Jian Tang. Hongyu Guo, and Anima Anandkumar. A text-guided protein design framework. arXiv preprint arXiv:2302.0461 1, 2023b.
Tennison Liu, Nicolás Astorga, Nabeel Seedat, and Mihaela van der Schaar. Large language models to enhance bayesian optimization. International Conference on Learning Representations, 2024.
Cooper Lorsung, Zijie Li, and Amir Barati Farimani. Physics informed token transformer for solving partial differential equations. Machine Learning: Science and Technology, 2024.
Ilya Loshchilov and Frank Hutter, Decoupled weight decay regularization. International Conference on Learning Representations, 2017.
Xinhao Mei, Chutong Meng, Haohe Liu, Qiuqiang Kong, Tom Ko, Chengqi Zhao, Mark D Plumbley, Yuexian Zou, and Wenwu Wang. Wavcaps: A chatgpt-assisted weakly-labelled audio captioning dataset for audio-language multimodal research. arXiv preprint arXiv:2303. 17395, 2023a.
Xinhao Mei, Chutong Meng, Haohe Liu, Qiuqiang Kong, Tom Ko, Chengqi Zhao, Mark D. Plumbley, Yuexian Zou, and Wenwu Wang. WavCaps: A ChatGPT--ssisted weakly-labelled audio captioning dataset for audio-language multimodal research. ArXiv preprint, abs/2303. 17395, 2023b. URL https: /ttpp://arxiv... httg/bbs//20 2303 17395.
Mike A Merrill, Mingtian Tan, Vinayak Gupta, Tom Hartvigsen, and Tim Althoff. Language models still struggle to zero-shot reason about time series. arXiv preprint arXiv:2404.1155,, 2024.
Daniel Potemkin, Carlos Soto, Ruipeng Li, Kevin Yager, and Esther Tsai. Virtual scientific companion for synchrotron beamlines; A prototype. ArXiv preprint, abs/2312.17180, 2023, URL https: //arxiv. htt///bs/2233 17180.
Abdul Quamar, Vasilis Efthymiou, Chuan Lei, Fatma Özcan, et al. Natural language interfaces to data. 11(4):319-414, 2022.
C. V. Rackauckas and Abdelrehim. Scientific machine learning (seiml) surrogates for industry. part 1: The guiding questions. http://doi.rr/100.111999//
2024. Accessed: 2024-03-22.
Christopher Rackauckas, Yingbo Ma, Julius Martensen, Collin Warner, Kirill Zubov, Rohit Supekar, Dominic Skinner, Ali Ramadhan, and Alan Edelman. Universal differential equations for scientific machine learning. arXiv preprint arXiv:2001.0433 2020.
Maziar Raissi, Paris Perdikaris, and George Karniadakis. Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational physics, 378:686-707, 2019.
Dakota Ramos, Andrew Glaws, Ryan King. and Dylan Harrison-Atlas. Flow redirection and induction in steady state (floris) wind plant power production data sets, 2023. URL https: //data. openei. org/ / submis ssiins/ /884.
Filipe Rodrigues, Ioulia Markou, and Francisco C Pereira. taxi demand prediction event areas: deep learning approach. Information Fusion, 49:120-129, 2019.
Combining series and textual data for
13

---

Published as a conference paper at ICLR 2025
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, distilled version of bert: smaller, faster, cheaper and lighter. ArXiv preprint, abs/1910.01108,, 2019. URL https: / ///aaxixvvv//////////// www.xii..gg/a.oo. abs /1900. 01108.
Timo Schick and Hinrich Schütze. Generating datasets with pretrained language models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 6943-6951, Online and Punta Cana, Dominican Republic, 2021. Association for Computational Linguistics. doi: 10. 18653/ 1e200l..mmlppmmin.. 2021. emnlp-ma i 555.
URL https: htt////aaaantoooooo/oooo org/
Philipp Seidl, Andreu Vall, Sepp Hochreiter, and Günter Klambauer. Enhancing activity prediction models in drug discovery with the ability to understand human language. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 20458-30490 PMLR, 2023. URL https: //proceedings mlr
mwm.m..ppee.. press
Xingyou Song, Oscar Li, Chansoo Lee, Daiyi Peng, Sagi Perel, Yutian Chen, et al. Omnipred: Language models as universal regressors. ArXiv preprint, abs/2402. 14547, 2024. URL https: 14547.
Makoto Takamoto, Francesco Alesiani, and Mathias Niepert. Learning neural pde solvers with parameter-guide guided channel attention. In International Conference on Machine Learning, pp. 33448- 33467. PMLR, 2023.
Mingtian Tan, Mike A Merrill, Vinayak Gupta, Tim Althoff, and Thomas Hartvigsen, Are language models actually useful for time series forecasting? arXiv preprint arXiv:2406., 102006669666 /6964, 2024.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. ArXiv preprint, abs/2307.0928s. 2023. URL pictts://arxlle org/ hbs//bb//a 2307. 09288.
Priyan Vaithilingam, Ian Arawjo, and Elena L Glassman. Imagining future of designing with ai: Dynamic grounding, constructive negotiation, and sustainable motivation. arXiv preprint arXiv:2402.07342,, 2024.
Jose Vazquez-Canteli, Aysegul Dilsiz Demir, Julien Brown, and Zoltan Nagy. Deep neural networks as surrogate models for urban energy simulations. In Journal of Physics: Conference Series, volume 1343, pp. 012002. IOP Publishing, 2019.
Jari Vepsäläinen, Kevin Otto, Antti Lajunen, and Kari Tammi. Computationally efficient model for energy demand prediction of electric city bus in varying operating conditions. Energy, 169: 433 443, 2019.
Eric Wallace, Yizhong Wang. Sujian Li. Sameer Singh. and Matt Gardner. Do NLP models know numbers? probing numeracy in embeddings. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.), Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP- IJCNLP), pp. 5307-5315, Hong Kong, China, November 2019, Association for Computational Linguistics. doi: 10.18653/11/D19-11
URL https: http:///ccaaattoooooo/oo wwg Dtrg/Di--1544
Eric JH Wilson, Andrew Parker, Anthony Fontanini, Elaina Present, Janet L Reyna, Rajendra Adhikari, Carlo Bianchi, Christopher CaraDonna, Matthew Dahlhausen, Janghyun Kim, et al. End-use load profiles for the us building stock: Methodology and results model calibration, validation, and uncertainty quantification. Technical report, National Renewable Energy Lab (NREL). URL https:// www. prw.. gov docs//ccss///ooooo / pdf.
Minghao Xu, Xinyu Yuan, Santiago Miret, and Jian Tang. Protst: multi mudti-modality learning of protein sequences and biomedical texts. In Proceedings of the 40th International Conference on Machine Learning, ICML'23. 23. JMLR.org. 2023.
14

---

Published as a conference paper at ICLR 2025
Jiahuan Yan, Bo Zheng, Hongxia Xu, Yiheng Zhu, Danny Chen, Jimeng Sun, Jian Wu, and Jintai Chen. Making pre-trained language models great on tabular prediction. arXiv preprint arXiv:2403.0188 2024.
Liu Yang, Siting Liu, and Stanley J Osher. Fine-tune language models multi-modal differential equation solvers. arXiv preprini arXiv:2308. 05061 v4, 2024.
Zhanhong Ye, Xiang Huang, Leheng Chen, Hongsheng Liu, Zidong Wang, and Bin Dong. Pdeformer: Towards foundation model for one-dimensional partial differential equations. arXiv preprint arXiv:2402. /2652, 2024.
Liang Zhang, Siby Plathottam, Janet Reyna, Noel Merket, Kevin Sayers, Xinshuo Yang, Matthew Reynolds, Andrew Parker, Eric Wilson, Anthony Fontanini, David Roberts, and Ralph Muehleisen. High-resolution hourly surrogate modeling framework for physics-based large-scale building doi: (0.1016/j.ees22221100033 https: / http///iinnnnn.ub./e///// e lsevier.... com/ http://ecceeeeees/22222000022220
stock modeling. 75:103292, 2021. ISSN 22106707.
URL
Yunjiao Zhou, Jianfei Yang, Han Zou, and Lihua Xie. Tent: Connect language models with iot sensors for zero-shot activity recognition. arXiv preprint 2rXi:2311.008255, 2023.
15

---

Published as a conference paper at ICLR 2025
Hourly electricity demand timeseries outputs
y
y
y
1286-lm embeddiing
Bidirectional
Sequence Encoder
Multi-layer Perceptron (MLP)
3C
T2
One-hot encoded inputs: 13 categorical building attributes
Hourly weather timeseries inputs variables) + per-lime-step calendar features
Figure 5: visualization the sequential surrogate model baseline with one-hot encoded system attributes for the buildings experiment.
A ADDITIONAL EXPERIMENT DETAILS
A.1 METRICS
We the normalized root mean square error (NRMSE) capture the accuracy of the surrogate model. NRMSE is also known as (CV)RMSE.
The building-hour NRMSE where the NRMSE is normalized by building and by hour, where B is the number of buildings in the building stock and T' is the number of hours in year:
B,T Σ
01)
(2)
Σ
BT
BT
6=1,t=1
The stock-annual NRMSE where the NRMSE is normalized the annual stock energy consump- tion:
B.T Σ
B.T
1
3 -
Σ bm 1,tml
(h)
Σ
c
frmm
1.t=1
Notice that the square and square root cancel, making the stock-annual NRMSE equivalent to the normalized mean bias error.
We use the AdamW (Loshchilov & Hutter, 2017) optimizer with 31 = 0.9, B2 = 0.98, = 1c-9, and weight decay of 0.01 for all experiments. The early stopping patience is 50 for all experiments. All models are trained with a single NVIDIA A A100-40GB GPU. The longest training runs take 1-2 days and the shortest 2-3 hours.
A.2 HYPERPARAMETERS
See Table for hyperparameter sweep details for the buildings experiments and Table for hyperpa- rameter sweep details for the wind farm experiments.
A.2.1 BUILDINGS
There are 13 attributes after RFE, which are one-hot encoded into 336-dimensional feature vector that gets embedded into 128 dimensions, whereas the text embeddings 768-dimensional. We
16

---

Published as a conference paper at ICLR 2025
Table 6: Building stock surrogate model hyperparameters.
Model
Hyperparameter
Grid search space
Best values
Learning rate
From 0.01 to 0.1
0,066
Number of leaves
From 40 to 150
149
LightGBM
Subsample
From 0.05 to 1.0
0.178
Feature fraction
From 0.05 to 1.0
0,860
Min number of data in one leaf
From I to 100
12
Hidden layers size
256, 1024, 2048
1024
Number of layers
2
ResNet (one-hot)
Batch
128, 256,
Learning rate
0.0001, 0.0003, 0.001
0.0003
Hidden layers size
256, 1024, 2048
256
Number of layers
2,8
8
ResNet (SysCaps)
Batch size Learning
128, 256, 512
256
0,0001, 0.0003, 0.001
0.0003
[Hidden layer size, Batch size]
[128, 64), [512, 32), [1024, 32]
[128, 64]
Number of layers
1, 4, 6,
4
Bidirectional LSTM (one-hot)
MLP dimension Learning rate
256
256
0,00001, 0.0003, 0.001
0.001
[Hidden layer size, Batch size]
[128, 64), [512, 32], [1024, 32)
[512, 32]
Number of layers
1, 4, 6,
Bidirectional LSTM (SysCaps)
MLP dimension Learning rate
256
256
0.00001, 0.0003, 0.001
0,0003
[Hidden layer size, Num, layers]
[64.8] [128,4]
[128,4]
MI.P dimension
256
256
Ridirectional S4 (one-hot)
Ratch size
64
64
Learning rate
le-5, 3e-4, 1e-3
3e-4
[Hidden layer size, Num. layers]
[64,8] [128.4)
[128, 4]
MLP dimension
256
256
Bidirectional S4 (SysCaps)
Batch size
32, 64
32
Learning rate
le-5, 3e-4, le-3
3e-4
Table Wind farm wake surrogate model hyperparameters.
Model
Hyperparameter
Grid search space
Best values
Learning
From 0.01 to 0.1
0.039
Number leaves
From 120
108
LightGBM
Subsample
From 0.6 to 1.0
0.963
Feature fraction
From 0.6 to 1.0
0.997
Min number of data in one leaf
From 20 to 100
96
Hidden layers size
[256,1024]
256
Number of layers
[2,8]
ResNet (one-hot)
Batch size Learning rate
[128,256]
128
[le-5, 3e-4, 1e-3]
le-5
Hidden layers size
[256,1024]
1024
Number layers
[2,8]
ResNet (SysCaps-kv)
Batch size Learning
[128.256]
256
[le-5.3e-4.1e-3]
3e-4
Hidden layers size
[256,1024]
1024
Number of layers
[2.8]
ResNet (SysCaps-nl)
Batch size Learning rate
[128,256]
256
[le-5,3e-4,1e-3]
le-5
concatenate cyclically encoded calendar features to 7 weather variables, creating a 103-dimensional vector (Emami et al., 2023b). For the one-hot models (Figure 5), this creates 128 + 103 231 dimensional input for the bidirectional sequence encoder, and for the text models, it a 768 + 103 871 dimensional input.
17

---

Published as a conference paper at ICLR 2025
Complex Energy System: Commercial Building
Instruction: <CES>, <STYL <NUM>
[System] You are building expert who provides buildding descriptions with an objective tone, [User] Write building description based on the following attributes. Your answer should
sentences. Please note that your response should NOT be list of attributes should be entirely based on the information provided.
System Knowledge: Tabular attributes
building__yy l_ppe RetailStandalone. number of stories: 1.0, sqft: 7500.0. hvac_system system system_type: PSZ-
AC with gas coll, weekday opening time: 12:15 PM, weekend opening time: 10:30 AM, unoccupied cooling temperature set point difference from occupied state: 0.0, occupied cooling temperature set point: 72.0, unoccupied heating temperature set point difference from occupied state: 0.0, occupied heating temperature set point: 67.0, weekday closingooonmannee time: 21:30 PM. weekend closing time: 16:15
SysCap: Natural language
The retail standalone building a single-story structure with a total area of approximately 7.500 square feet. The building is equipped with PSZ-AC system heating and cooling for the space. The building open from 12:15 PM on weekdays and 10:30 AM weekends, with closing times 21:30 PM weekdays and 16:15 PM weekends. The temperature set points for both heating and cooling are set at 67 and 72"F, respectively, when the space is occupied. However, there is no difference in the temperature set points when the space is unoccupied.
gas coll, which provides
Figure 6: Natural language building SysCap. The instruction and list of key-value attributes (first two paragraphs) are provided to the LLM as the prompt. We observe that the LLM automatically converts numbers to more iemer-interpretable descriptions (e.g., the number of stories changed from 1.0 in the prompt to "single-story"). The I.I. also succinetly (and correctly) states that "there is no difference in the temperature set points when the space is unoccupied" in the output by summarizing the "unoccupied heating/cooling temperature set point difference from occupied state: 0.0" attributes.
LightGBM: As LightGBM does not support batch training out of the box, the entire training data needs to be loaded into the memory to train a LightGBM model. With the train dataset containing 340k buildings, each with 8759 hours and 347 features, we randomly extract 438 hours per building (which about 5% of total hours) limit memory usage. This results in 340, 000 x 438 hours in total for the train dataset, which consumes about 380 GB of memory when being loaded into NumPy object. For the validation and test splits, we retain the full number of hours per building. The LightGBM model is tuned with Optuna (Akiba al., 2019) across 30 trials and achieves the best validation NRMSE of 0.667.
Multi-class attribute classifier: We implement the classifier top of the text encoder by adding linear layer for each attribute type, where this layer predicts logits for each attribute's 's classes. We use AdamW with a learning rate of 3e-4, early stopping with patience 5, batch size 128, and max epochs 100. We do not freeze the text encoder weights.
A.2.2 WIND FARM
LightGBM: The training, validation, and test split for the wind dataset gives us datasets of size 148,650, 49,600, and 49,250 respectively with 190 features after one-hot encoding. The training dataset is loaded into memory to train the LightGBM model. We use Optuna (Akiba et al., 2019) to tune the hyperparameters and the best validation NMRSE achieved is 0.189.
B SYSCAPS PROMPTS
In this section visualize examples of prompts and the corresponding SysCaps LLM outputs for the buildings simulator (Figures 6, 7, and 8) and the wind farm simulator (Figures 9, 10, and 11).
18

---

Published as a conference paper at ICLR 2025
Complex Energy System: Commercial Building
Instruction: <CFS> <STYL Fs <NUM>
[System] You are building expert who provides building descriptions with an objective tone. [User] Write building description based
following attributes. answer should be 4-6 that your response should NOT list of attributes and should
sentences. Please
entirelyy based on the information provided.
System Knowledge: Tabularattriibtte
building_sublype: subtype: strip mall restaurant10, building _eiidnnglypee:: RetailStripmall, number stories: 2.0, sqft: 37500.0, wwac_system_iag type: PSZ-AC with gas coil. weekday _openig time: 8:15 AM, weekend opening time: 9:00 AM, unoccupied cooling temperature set point difference from occupied state: 3.0, occupied cooling temperature set point: 73.0, unoccupied heating temperature set point difference from occupied state: 10.0, occupied heating temperature set point: 67.0, weekday weekday_closing time: 23:30 PM, weekend_ closing time: 20:45 PM
SysCap: Natural language
This strip mail restaurant is two-story building with total floor area of 37 .500 square feet. The building is equipped with a PSZ-AC system with gas coil, which provides heating and cooling to the space. The restaurant is open Monday through Friday from 8:15 AM to 23:30 PM, and Saturday and Sunday from 9:00 AM to 20:45 PM. During unoccupied perlods, the cooling temperature set point set degrees higher than the occupied state, while the heating temperature point is 10 degrees higher than the occupied state. occupied cooling temperature set point 73 degrees Fahrenheit, while the occupled heating temperature set point is 67 degrees Fahrenheit.
Figure 7: Natural language building SysCap. The LLM makes subtle logical error here- the unoccupied heating temperature set point difference from occupied state suggests the set point should be lower by 10 degrees, not higher (highlighted in blue). Logical errors such this may have contributed to the slightly worse accuracy of natural language SysCaps compared to key-value SysCaps. Collaborating with domain experts able validate the LLM's outputs is important for catching these errors. We expect that using more powerful LLMs will help reduce logical errors.
C QUALITATIVE EXAMPLES OF TEST PREDICTIONS
We visualize in Figure 12 the predicted energy timeseries for one building by the SysCaps-nl and SysCaps-kv models alongside the weather timeseries and each SysCaps type (key-value, short, medium, and long). We also show test predictions for wind farm (Figure 13) alongside the key-value SysCaps and each style-augmented natural language SysCaps.
19

---

Published as a conference paper at ICLR 2025
Complex Energy System: Commercial Building
Instruction: <CES>, <STYLE>, <NUM>
[System] You are building expert who provides building descriptions with an objective tone, [User] Write building description based on the following attributes. Your answer should
sentences. Please nole your response should NOT
attributes should
entirely based on the information provided.
System Knowledge: Fabular
building subtype: strip_mail restaurant30, building type: RetailStripmall, number of stories: 2.0, sqft: 17500.0, hvac_ srco__ystem__yye _ype: PSZ-AC with coil, weekday_ becplyy__eennnggiiie _ime: 8:15 AM, weekend _eaeodng___ng_____ee simminngtmme:: 11:00 AM, unoccupied coaling temperature set point difference from occupied state: 6.0, occupied cooling temperature set point: 72.0, unoccupied heating temperature set point difference from occupied state: 7.0, occupied heating temperature set point 68.0, weekday clooiinggoongggoooeeeeee time: PM, weekend closing time: 22:30
SysCap: Natural language
The strip mall restaurant two-story building with total floor of 17,500 square feet. The building equipped with PSZ-AC system with gas coil, which provides heating and cooling. The building is open from 8:15 AM to 11:00 AM on weekdays and 11:00 AM to 22:30 PM on weekends. During unoccupied periods, the cooling temperature set point is 6.0 degrees lower than the occupied state, while the heating temperature set point is 7.0 degrees lower. The building well-insulated and energy-effciiet, with weekday opening time 8:15 AM and weekend opening 11:00 AM.
Figure 8: Natural language building SysCap. The LLM confuses the weekday closing time with the weekend open time (highlighted in blue).
Complex Energy System: Wind Farm
Instruction: <CES>, <STYLE>, <NUM>
[System] You wind energy expert describing wind plant to colleague.
[User] Write a wind plant description based on the following attributes. Your answer should be 4- sentences. Please note that response should NOT be a list of attributes and should be entirely based on the information provided.
System Knowledge: Tabular attributes
Plant Layout: cluster, Number of Turbines: 123, Mean Turbine Spacing: 5, Rotor Diameter: 130.0 meters, Turbine Rated Power: 3.4 MW
SysCan: Natural language
The wind plant is laid out in duster configuration, with 123 turbines positioned at spacing of approximately 5 times the rotor diameter. Each turbine features a rotor diameter of 130 meters and rated to produce 3.4 megawatts power. The plant's layout and turbine design work together optimize energy production minimize visual impact.
Figure 9: Natural language wind farm SysCap.
20

---

Published as a conference paper at ICLR 2025
Complex Energy System: Wind Farm
Instruction: <CES>, =>,YL <NUM>
[System] You are wind energy expert describing wind plant to colleague.
[User] Write a wind plant description based on the following attributes. Your answer should be 4- sentences. Please note that your response should NOT be a list of attributes and should be entirely based on information provided.
System Knowledge: Tabular attributes
Plant Layout: multiple strings, Number Turbines: 159, Mean Turbine Spacing:
Rotor
Diameter: 130.0 meters, Turbine Rated Power: 3.4 MW
SysCap: Natural language
The wind plant consists multiple strings of turbines, with total 159 turbines spread the landscape. Each turbine has rotor diameter of 130 meters and spaced distance of approximately 7 times the rotor diameter apart, creating sprawling landscape of towering structures. With a rated power of 3.4 megawatts each, the turbines are capable of generating significant amount electricity, making the plant a major contributor to the local energy mix.
Figure 10: Natural language wind farm SysCap.
Complex Energy System: Wind Farm
Instruction: <CES> <STYL <NUM>
[System] You wind energy expert describing wind plant to colleague.
[User] Write wind plant description based on the following attributes. Your answer should be 4- 6 sentences. Please note that your response should NOT be a list of attributes and should be entirely based on the information provided.
System Knowledge: Tabular attributes
Plant Layout: parallel strings, Number of Turbines: 127, Mean Turbine Spacing: 7, Rotor Diameter: 130.0 meters, Turbine Rated Power: 3.4 MW
SysCap: Natural language
The wind plant consists of 127 turbines arranged in parailel strings, each with a rotor diameter of 130 meters. The turbines are spaced at an average distance times the rotor diameter, resulting in a total installed capacity electricity from wind, making valuable source of
MW. With this layout, the plant is capable of
generating significant renewable energy.
Figure 11: Natural language wind farm SysCap with a logical error where it says "total installed capacity". but it just states the capacity of a single turbine (highlighted in blue). The correct total installed capacity is number of turbines times 3.4, or 431.8 MW.
21

---

Published as a conference paper at ICLR 2025
Building Weather Timeseries
275
Targer Spscuss Sescous
225
Load
175
150
100
-12
Dry Bub Temperoture
-14
18
-20
2018:02-05
2018:02.06 06
2018-02-07
2018:02.00 08
2018-02-00
2018 10
Time
total floor area of 17,500 square feet. The building is equipped with PSZ-AC system with gas cail, which provides heating and cooling to the space. The building is open from 9:15 AM to 10:45 AM on weekdays and 10:45 AM 19:15 PM on weekends. During unoccupied periods,
Medium: The strip mall restaurant a single-ston retail building
cooling temperature set paint is set at 73 degrees Fahrenheit, while the heating temperature set point is set at 68 degrees Fahrenheit. The building has moderate temperature difference between occupied and unoccupied states, with cooling temperature difference degrees Fahrenheit and heating temperature difference of 0 degrees Fahrenheit.
The strip mall restaurant is single-story building with a total area of 17,500 square feet, The HVAC system is PSZ-AC with gas coil, and the temperature set points both heating and cooling set at 68"F "F and F3*F, respectively, during unoccupied periods. The building at 9:15 AM on weekdays and 10:45 AM on weekends, and clases at 5:30 PM on weekdays and 7:15 PM on weekends.
Short: :
Long: The strip mall restaurant is single-story retail building with total floor area of approximately 17, 7,500 square feet. The building is equipped with a PSZ-AC with gas coil HVAC system, which provides heating and cooling
restaurant opens at 9:15 AM and closes at 5:30 PM, while on weekends, it opens at 10:45 AM and closes at 6:00 PM. The building is unoccupied during the night, from 17:30 PM to 10:45 AM on weekdays and from 19:15 PM on weekends. point during occupancy set
space. During weekdays,
The cooling temperatur rature heating remperature set point set at 68 degrees Fahrenheit. The building has moderate temperature difference between occupancy and unoccupancy, with a difference of degrees Fahrenheit cooling and degrees Fahrenheit heating.
degrees Fahrenheit, while the
Keyvalue: building subtype:strip, mall restaurant20lbuiddin htwpra/ooooouiinigeeeeeeeeerarrr Sype:RetaillStripmalllmmma of sto
ries:1.0|sqft:115000 1550:g::1550000000 / soocsssstmmmmmmm
wtwpssetttyeeeSSeeeeee.e. pSpe:pSZ-AAc with gas weekwwe operating Nours:8.251 weekday ope
Ling_time:9.222 htti:me::.225/weeeeee weekendd -operaiingggggggggggge hours:8. weeeeedd opening time:10.75 0.75 tstat__cll ceeiat_clg__elle cdg_delta_ft, f:3.ta_ff3...ppm f:3.0]tstat_c
g_sp f.7.0 htg delta t.O.0 htg sp_.ppff... f:68.0
Figure 12: (Top) Predicted hourly energy consumption for randomly selected week. (Second from top) We visualize out of input weather timeseries, Dry Bulb Temperature. The text boxes show medium, short, long, and key-value SysCaps created for this particular test building.
22

---

Published as a conference paper at ICLR 2025
Layout214, True: 318.93 MW, SysCaps-nl: 319.50 MW, SysCaps-kv: 315.75 MW
<TYLE
objective Creative paraphrasing acceptable
The wind plant features layout of multiple strings, with 94 turbines positioned across the site. Each turbine stands at height 130 meters, with a rotor diameter that is four times that size. The turbines are spaced optimal wind flow and energy production. With rated power of 3.4 megawatts, each turbine is capable of generating significant amount electricity, making the plant significant contributor the local energy grid.
distance approximately
from each other, allowing for
<TYLE with objective
The wind plant features layout of multiple strings, with 94 turbines installed across the site, Each turbine has rotor diameter 130 meters, and turbines are spaced at an average distance of four times the rotor diameter apart. The turbines rated produce 3.4 megawatts power, making this significant contributor to the local renewable energy mix.
<TYLE
colleague
The wind plant consists of multiple strings of turbines, with 94 turbines in total. Each turbine has a rotor diameter of 130 meters and spaced approximately times that distance apart, resulting in total installed capacity 3.4 megawatts. The plant takes advantage the consistent wind patterns in the area to generate clean and renewable energy.
<TYLE
classroom
The wind plant consists of multiple strings of turbines stretched across the landscape, with total 94 turbines standing tall and proud. Each turbine has rotor diameter of 130 meters, giving massive presence the horizon. The turbines the rotor diameter from each other, creating a uniform and efficient layout. With each turbine capable generating 3.4 megawatts power, the plant
spaced at distance of approximately four times
a total capacity over megawatts, enough to power thousands homes and businesses. As the wind blows, the turbines spin effortlessly, harnessing the natural energy of the wind to generate clean and sustainable electricity.
keyvalue
Plant Layout:multiple strings/Numbbe of Turbines:94 |Mean Turbine Spacing:4 Rotor Diameter:130.. meters] meters/Turbine Rated Power:3.4 4 MW
Figure show the LLM ----gerrraeed SysCaps using four different styles as a prompt augmentation strategy for test wind farm layout 214. The key-value SysCaps is shown at the bottom. Model predictions are shown the top next to the true ground truth value.
23


Reviews:
[{"review_id": "soJZYufbl3", "reviewer": "Reviewer_38tw", "summary": "This paper describes a set of lightweight models to model complex energy systems, using an LLM to generate prompts and a encoder and bidirectional time-series model to predict energy consumption.", "questions": "I would be interested in seeing the performance compared between time-series-centric models and current generic architectures.", "limitations": "", "rating": 5, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The authors provide a clear explanation of their setup, flow of data across multiple components and their evaluation and analysis.\n- (I do not feel sufficiently well-acquainted with this domain to evaluate the predictive contribution or performance of the models.)", "weaknesses": "- I presume the authors' choice of models is due to resource constraints and aiming for a lightweight setup, but it feels like it has multiple components when it could be a simpler setup with fewer model components. For instance, a BERT-type model could also be used for time-series prediction (as opposed to only text encoding). Similarly the two-step process of generating prompts using a separate LLM and then encoding that prompt with an encoder could be avoided by just using the LLM directly and fine-tuning it.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "C2lWY54O3J", "reviewer": "Reviewer_2gzL", "summary": "The paper discusses the important challenge of building surrogate models for the prediction of simulation data. They specifically motivate the problem for complex energy systems(CES). These surrogate models often model system features as one hot vectors. The authors propose using text based descriptions to model these so-called surrogate systems with time series data. The text data is encoded as a dense embedding obtained from language models. The embedding is then fed to a bidirectional sequence encoder along with the time series data. \n\nThe paper discusses the generation of the text pertaining to the attributes of such systems and proposes an automatic evaluation strategy for the same. \n\nFor generating the captions the authors prompt an LLM with an in-context learning-based prompt that tunes the style and number of sentences. To evaluate the SysCap quality the authors train a multi-class classifier to check the attributes covered in the description generated by the LLM, using the text embedding. \n\nThe authors show how including SysCaps along with time series data leads to improved performance against baselines that perform onehot encoding over attributes. The authors further show how training a custom embedding model can aid in improving time series prediction over just using a time series-based model. They further empirically prove how the embeddings are more robust to synonyms and missing data.", "questions": "In addition to the points in the weakness:\n\n1. Did the authors try to just templatize the sentences rather than generating them using an LLM, how would that impact performance (i.e. rather than telling an LLM to adhere to some constraint-based template, just have a sketch sentence and fill attribute values in the given sentence)?\n2. Why wasn't RFE performed for the Wind Farm Wake modeling dataset, would performing RFE improve performance ?\n3. Would the model not further improve if the SysCaps were generated using synonyms for the attributes, did the authors see the LLM generate different synonyms for the building or wind farm dataset? \n4. Do the authors believe that training on the subset of data where the caption quality assessed by the classifier model, would improve the overall model performance?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper is well-written and easy to follow. The authors motivate the problem well and empirically show improvements over 2 real-world datasets. Further, SysCaps can be used by non-expert users to understand the features of surrogate systems. The Design space exploration is insightful to show the features learned by the model.", "weaknesses": "1. The paper claims the technique uses a pretrained text encoder for generating the embeddings, but then in section 5 mentions that the models are actually finetuned. This should be explicitly mentioned in the claims that the paper makes rather than just mentioning that a pretrained embedding is used. \n2. Further, the authors do not compare with the \"said-pretrained\" embeddings but only finetuned embeddings, and other SOTA embedding models for text encoding. \n3. The paper also claims that they train a system to evaluate the caption quality, the parameters of the said multiclass classifier are omitted from the paper.\n4. The paper claims that for the CES building energy consumption dataset, the SysCaps-kv configuration works best, and for the turbine configuration the SysCaps-nl, there should be some discussion regarding the insights drawn from both cases and why the performance for both techniques are different. \n5. The authors claim that SysCaps would be useful for non-expert users, but lack the discussion if LLM-based explanations (complementary to the work done) can also aid in explaining the system attributes for surrogate models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TAL525GnH3", "reviewer": "Reviewer_Gxzq", "summary": "This paper studies the use of natural language captions as inputs to surrogate models that simulate \"complex energy systems\". These natural language captions describe the features of the system being simulated. The task is to predict a timeseries of some variable of interest that depends on these features and some other independent variable that is fed as a time series. The paper introduces an architecture that fuses the textual description with the time series data to achieve this goal.\nThe viability of the approach and its robustness to out-of-distribution perturbations are validated with a relatively extensive empirical evaluation, including different ablations of the system (such as one-hot encoding of the features, or no features), variations on the caption lengths or replacing words with synonyms.", "questions": "* Q1) What's the advantage of the proposed approach using LLMs over more traditional template-based natural language captions? (e.g. \"The building is <x> squared feet.\", etc.)\n* Q2) In Figure 1, the key-value template has only a colon to separate the key and the value. Have you tried adding a space in between? I expect \n* Q3) For the one-hot encodings, how do you deal with numeric inputs?\n* Q4) In the results in Table 3, why did you expect longer captions to have larger error? I would have had the opposite intuition as shorter captions are more likely to miss important attributes.", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* S1: Provides extensive empirical evaluation of the proposed system\n* S2: The presentation is clear.", "weaknesses": "* W1: The LightGBM baseline is underspecified. This baseline is the only one that stands as a reference point that is not an ablated version of the proposed model. However, as I understand it, LightGBM is a framework but not necessarily a model, so I don't really to which model this system is really being compared against.\n* W2: Not very clear what is the added value of the proposal of using LLMs against simply using a template-based natural language description.\n* W3: Despite the system is motivated on the potential intuitiveness of language interfaces to non-experts, no particular study is conducted on that front.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)