SysCaps: Language Interfaces for Simulation Surrogates of Complex Systems

## Abstract

Abstract Surrogate models are used to predict the behavior of complex energy systems that are too expensive to simulate with traditional numerical methods.
Our work introduces the use of language descriptions, which we call “system captions” or SysCaps, to interface with such surrogates.
We argue that interacting with surrogates through text, particularly natural language, makes these models more accessible for both experts and non-experts.
We introduce a lightweight multimodal text and timeseries regression model and a training pipeline that uses large language models (LLMs) to synthesize high-quality captions from simulation metadata.
Our experiments on two real-world simulators of buildings and wind farms show that our SysCaps-augmented surrogates have better accuracy on held-out systems than traditional methods while enjoying new generalization abilities, such as handling semantically related descriptions of the same test system.
Additional experiments also highlight the potential of SysCaps to unlock language-driven design space exploration and to regularize training through prompt augmentation.

## 1 Introduction

Data-driven surrogates enable computational scientists to efficiently predict the results of expensive numerical simulations that run on supercomputers (Lavin et al., ; Carter et al., ).
Surrogates are particularly valuable for emulating simulations of *complex energy systems* (CES), which model dynamic interactions between humans, earth systems, and infrastructure.
Examples of CES include buildings (Vazquez-Canteli et al., ; Dai et al., ; Bhavsar et al., ), electric vehicle fleets (Vepsäläinen et al., ), and microgrids (Du & Li, ).
Advancing the science of CES contributes to reducing emissions and accelerating the adoption of clean energy, which is needed to address the impacts of climate change.

These models perform a fairly standard regression task, predicting simulation output quantities of interest from *a*) an input system configuration and *b*) a deployment scenario.
For example, we might want to predict the amount of energy a building will consume given *a*) a list of building characteristics and *b*) a weather timeseries spanning an entire year.
In this case, this involves performing long sequence timeseries regression, which traditional regression techniques such as gradient-boosted decision trees have difficulty with (Bhavsar et al., ; Zhang et al., ).

Surrogate models are not only used by experts. Surrogates are also used to inform highly consequential policy and investment decisions about complex systems made by non-experts in industry and governments (Rackauckas & Abdelrehim, ), such as when planning to build and deploy a new renewable energy system (Harrison-Atlas et al., ).
In this work, we design and analyze *language interfaces* for such surrogates.
Intuitively, language interfaces make surrogate models more accessible, particularly for non-experts, by simplifying how we inspect and alter a complex system’s configuration.
Language interfaces are powerful—they ground interactions between humans and machines in the human’s preferred way (Vaithilingam et al., ).
The idea of using language to create interfaces for complex data or models is not new (Hendrix et al., ; Quamar et al., ), but interest has renewed due to the success of large language models (LLMs) and their demonstrated ability to generate high-quality synthetic natural captions (Schick & Schütze, ; Doh et al., ; Mei et al., ; Hegselmann et al., ).
Our work defines a “system caption”, or SysCap, as text-based descriptions of *knowledge about the system* being simulated.
The only available knowledge our work assumes is the system configuration found in simulator metadata files as lists of attributes.

In general, it is unknown whether textual inputs, and particularly *natural language* inputs, are suitable for real-world *tabular* regression tasks.
Tabular data, such as the system attributes in question, are sets of both discrete (categorical, binary, or string) and continuous (numeric) variables.
Previous work demonstrated inconclusive evidence when using pretrained language models to do tabular regression from text-encoded inputs, with and without modifications to the architecture (Dinh et al., ; Jablonka et al., ; Bellamy et al., ; Yan et al., ), motivating further study.
Regression with text-encoded tabular inputs is promising because *a*) language is a more intuitive and flexible user interface than traditional encoding strategies (e.g., one-hot encodings), and *b*) using pretrained language embeddings to encode system attributes unlocks the ability to exploit the semantic information contained in SysCaps to generalize across related systems.

Our paper introduces a framework for training *multimodal* surrogates for CES with text (for system attributes) and timeseries inputs (for the deployment scenario) and makes contributions towards addressing the following technical challenges:

- •
We introduce a simple and lightweight multimodal surrogate model architecture for timeseries regression that *a*) fuses text embeddings obtained from pretrained language models (LMs) with *b*) timeseries encoded by a bidirectional sequence encoder. We expect this to be insightful for future multimodal text and timeseries studies.
- •
To address the lack of human-labeled natural language descriptions of complex systems, we describe a process that uses LLMs to generate high-quality natural language SysCaps from simulation metadata. Although LLMs have previously been used to generate text captions from metadata (Doh et al., ; Mei et al., ), we believe our application to multi-modal surrogate modeling is novel.
- •
We develop an automatic evaluation strategy to assess caption quality–specifically, we estimate the rate at which ground truth attributes appear in the synthetic description with a multiclass attribute classifier.

Our experiments are based on two real-world CES simulators of buildings and wind farm wake.
We rigorously evaluate accuracy on held-out systems and show that SysCaps-augmented surrogates have better accuracy than one-hot baselines. We also show generalization beyond the capabilities of traditional regression approaches enabled by the use of text embeddings, e.g., robustness to replacing attributes names with synonyms in test captions.
We qualitatively show that text interfaces unlock system design space exploration via language.
As there are no standard benchmarks for comparing surrogate modeling performance for CES, we will open-source all data and code and contribute the generated SysCaps datasets to facilitate future work.

## 2 Related Work

Language interfaces for scientific machine learning: An increasing amount of work is exploring language interfaces for advanced scientific machine learning (SciML) models, including protein representation learning (Xu et al., ), protein design (Liu et al., ), and activity prediction for drug discovery (Seidl et al., ).
LLM-powered natural language interfaces are also being designed for complicated scientific workflows including synchrotron management (Potemkin et al., ), automated chemistry labs (Bran et al., ), and fluid dynamics workflows (Kumar et al., ).
We add to this body of work by studying language interfaces for lightweight surrogate models.

Large language models for regression: Another line of work asks whether LLMs can perform regression with both text inputs and outputs (numbers encoded as tokens), such as for tabular problems (Dinh et al., ) or black-box optimization (Song et al., ; Liu et al., ).
We do not use an LLM to do regression directly, but rather train a lightweight multimodal architecture that predicts continuous outputs instead of tokens.
Moreover, one study (Dinh et al., ) found mixed results when comparing to simple gradient-boosted tree baselines and highlighted difficulty with interpolation, raising questions about the effectiveness of LLM-based regression.

Multimodal text and timeseries forecasting: Timeseries forecasting aims to predict future values of an input timeseries given past values.
Surrogate modeling can be cast as a forecasting problem when the goal is to train a model to emulate a dynamical system and predict its future behavior (e.g., predicting future energy demand from past energy usage).
Previous work has explored multimodal timeseries forecasting where auxiliary text data is introduced as covariates to improve the forecasting accuracy (Rodrigues et al., ; Emami et al., ; Jin et al., ).
Notably, Time-LLM (Jin et al., ) “reprograms” an LLM to process both text prompts and timeseries.
However, our timeseries regression setting (Section 3) differs in that our surrogates are trained to map simulator inputs (e.g., building characteristics and a weather timeseries) to simulator outputs (e.g., energy usage).
In our problem, models critically depend on the system information encoded as text, whereas in Time-LLM the text only contains auxiliary information that slightly improves forecasting accuracy.
This critical dependence partially motivates our development of a custom architecture for fusing text and timeseries embeddings.
Recent evidence (Merrill et al., ; Tan et al., ) also brings into question whether LLMs are useful at all for reasoning about temporal data; thus, research on lightweight multimodal models is warranted.

Multimodal text and timeseries contrastive pretraining:
Various efforts have explored contrastive pretraining objectives for modeling text and timeseries data (Agostinelli et al., ; Huang et al., ; Liu et al., ; Zhou et al., ).
For example, Agostinelli et al. () aligns embeddings of captions that describe the audio.
In our setting, captions describe system attributes, and the timeseries inputs are exogenous simulator inputs.
Here, the text and timeseries inputs share no information with which to learn a shared embedding space.

Knowledge-enhanced PDE surrogates:
Numerical simulation of partial differential equations (PDEs) is extremely computationally intensive, and thus a large body of SciML work is focused on developing neural PDE surrogates which are efficient to evaluate.
Recent work has tried to encode knowledge about the physical system under study (the PDE) into a surrogate to facilitate generalization within and across families of PDEs.
Specifically, methods that embed equation parameters (i.e., the system attributes) within the architecture to generalize to unseen parameters include CAPE (Takamoto et al., ) and those explored in Gupta & Brandstetter ().
Others embed structural knowledge about the PDE into the surrogate model architecture (Rackauckas et al., ; Ye et al., ) or the loss (Raissi et al., ).
Concurrent work has explored “PDE captions” (Lorsung et al., ; Yang et al., ), which are a type of SysCaps for neural PDE surrogates where the system knowledge is PDE equations encoded as text.

## 3 Problem Statement

Our goal is to learn a surrogate $f:\mathcal{X}\times\mathcal{Z}\rightarrow\mathcal{Y}$ that regresses the outputs of a simulator $F$ directly from its inputs.
We are given a dataset $D$ of pairs of simulator inputs and outputs.
The inputs are a deployment scenario $x_{1:T}\in\mathcal{X}$ (a timeseries) and the tabular system attributes $z\in\mathcal{Z}$.
The outputs are a timeseries $y_{1:T}\in\mathcal{Y}$.
For simplicity, we consider only univariate timeseries outputs in this work ($y_{t}\in\mathbb{R}$).
However, the number of timesteps $T$ may be large ( thousands of steps), the timeseries inputs $\mathcal{X}$ are multivariate, and the mapping $f$ which approximates the simulator is highly nonlinear.

To summarize, we have a timeseries regression problem modeled as $y_{1:T}=f(x_{1:T},z)$.
By conditioning the surrogate on system knowledge $z$, it can potentially generalize to new system configurations.
However, learning transferable representations of variable-length, heterogeneous input features such as $z$ is notoriously difficult for deep neural networks, and is a key focus of tabular deep learning (see survey  (Badaro et al., )).
In our work, we develop and analyze a framework for learning multimodal surrogates where $z$ is encoded as text.

Some simulators may have inputs that are not clearly distinguishable into what is $\mathcal{X}$ and $\mathcal{Z}$, for example, if a dynamical system simulation is configured to be in steady-state or assumes fixed exogenous conditions.
In these cases, we allow $\mathcal{X}$ to be a vector of real-valued scalars (a timeseries with $T=1$), or, simply an empty set (leaving only $\mathcal{Z}$).

Example: In many CES, the timeseries $\mathcal{X}$ are exogenous inputs to the system such as weather timeseries consisting of temperature or wind speed. Attributes $\mathcal{Z}$ of a wind farm might include the number of turbines in the wind farm and turbine blade length.

## 4 Synthesizing System Captions (SysCaps) with LLMs

Figure: Figure 1: Our pipeline for augmenting multimodal simulation surrogates with language interfaces using “system captions”, or SysCaps. SysCaps are text descriptions of knowledge about the system being simulated. In our work, the SysCaps describe the system’s characteristics, as found in simulation metadata files. During training (a), we create paired datasets of temporal simulator inputs with key-value template SysCaps or LLM-generated natural language SysCaps. At test time (b), we prompt the surrogate model with one or more key-value template captions or natural language captions. LLMs are only used to generate synthetic training data; we use a lightweight BERT-style text encoder and an efficient long-sequence encoder to keep the computational cost of our surrogate low.
Refer to caption: x1.png

Our work is motivated by the idea that language interfaces for surrogates represent a path towards improving the accessibility of these models for expert and non-expert users, e.g., when using them for downstream system design tasks (Vaithilingam et al., ).
Our proposed framework for augmenting surrogates with language interfaces is visualized in Figure 1.
During training, we create SysCaps out of system attributes specified in simulation metadata.
To create large amounts of synthetic natural language SysCaps, we use LLMs.
The ultimate goal is to enable scientists to “chat" with the multi-modal surrogate model at test time via text prompting.
In this section, we describe two approaches for converting system attributes into text: key-value templates and natural language.

For the key-value approach, attributes are described as key-value pairs key:value and joined by a separator “$|$” (SysCaps-kv). For example, if a simulation has attributes A=1.0 and B=blue, we create the string A:1.0|B:blue. Generating these strings is easy to do and incurs a negligible amount of extra computational overhead.
In the natural language approach (SysCaps-nl, Figure 1), attributes are described in a conversational manner, which we believe is more flexible and expressive than key-value captions and thereby more accessible for non-experts.
However, we do not have access to large quantities of natural language descriptions for each system and simulation.
We avoid the time-consuming task of enlisting domain experts to create this data by instead prompting a powerful LLM to generate synthetic natural language descriptions given attributes.
In our work, we use the open-source LLM llama-2-7b-chat (Touvron et al., ).
The details of the prompt are provided next.

Prompt design: We append a carefully written instruction template to a list of system attributes to help guide the LLM in generating a caption via prompting (see Figure 1). The system prompt is: *You are a <CES> expert who provides <CES> descriptions <STYLE>.* The user prompt is: *Write a <CES> description based on the following attributes. Your answer should be <NUM> sentences. Please note that your response should NOT be a list of attributes and should be entirely based on the information provided*. The last part is added to discourage the LLM from changing or omitting attributes. The tags <CES>, <STYLE>, <NUM> are filled in with the CES type (e.g., *buildings*), the style of the description (e.g, *with an objective tone*), and the number of sentences to use in the description (e.g., “4-6”), respectively.

Attribute subset selection: Simulations of real-world systems may have attributes that only weakly correlate with the output quantity of interest, or have a large number of attributes, which can be challenging for deep learning approaches.
Since the length of a SysCap is proportional to the number of attributes, the computational burden incurred by text-based encodings of attributes can grow significantly in these cases.
In these cases, reducing the number of attributes can be handled with classic feature selection methods such as recursive feature elimination (RFE) (Guyon et al., ) or by recommendations from domain experts, as a pre-processing step.

## 5 Text and Timeseries Surrogate Model

Figure: Figure 2: Building blocks of our surrogate model, $f=h_{\theta}\circ g_{\psi}$, that includes a multimodal encoder, $g_{\psi}$, and a top model, $h_{\theta}$. The multimodal encoder, $g_{\psi}=g_{\psi}^{\mathsf{seq}}\circ g_{\psi}^{\mathsf{text}}$, is a composition of a text encoder, $g_{\psi}^{\mathsf{text}}$, and a bidirectional sequence encoder, $g_{\psi}^{\mathsf{seq}}$, for timeseries inputs. The text embedding vector $\hat{z}$ is broadcasted (dashed lines) to create a sequence that is concatenated with the timeseries input. This multimodal sequence is the input to the sequence encoder.
Refer to caption: x2.png

We now describe a lightweight multimodal surrogate model for timeseries regression.
The surrogate $f$ (Figure 2) is a composition of a multimodal encoder function $g_{\psi}:(\mathcal{Z},\mathcal{X})\rightarrow\{\mathbb{R}^{d}\}_{1:T}$ and a top model $h_{\theta}:\mathbb{R}^{d}\rightarrow\mathbb{R}$, where for simplicity, the model parameters $\theta$ are shared across timesteps to predict each timeseries output $y_{t}$.
The training objective is to minimize the expected mean square error averaged over simulation timesteps,

$$ $\min_{\theta,\psi}\mathbb{E}_{(z,x_{1:T},y_{1:T})\sim D}\Biggl{[}\frac{1}{T} \sum_{t=1}^{T}\bigl{(}[h_{\theta}(g_{\psi}(z,x_{1:T}))]_{t}-y_{t}\bigr{)}^{2} \Biggr{]}.$ (1) $$

Although more sophisticated loss functions than Eq. 1 could be used that account for predictive uncertainty, we left this extension for future work to simplify our exposition and experiments.

Multimodal encoder $g_{\psi}$:
A text encoder $g_{\psi}^{\mathsf{text}}$ extracts an embedding $\hat{z}$ from a SysCap $z$ then broadcasts and concatenates this embedding with the timeseries inputs to create a sequence of multimodal feature vectors.
These features get processed by a bidirectional sequence encoder $g_{\psi}^{\mathsf{seq}}$ to produce a sequence of time-dependent fused multimodal features $e_{1:T}$, $e_{1:T}=g_{\psi}^{\mathsf{seq}}(g_{\psi}^{\mathsf{text}}(z),x_{1:T})$ , which are finally used to regress outputs.

Text encoder $g_{\psi}^{\mathsf{text}}$: To encode textual inputs we use pretrained BERT (Devlin et al., ) and DistilBERT (Sanh et al., ) models that are relatively more efficient than LLMs. We use the model’s default pretrained tokenizer. Tokenized sequences are bracketed by [CLS] and [EOS] tokens, and we use the final activation at the [CLS] token position to produce a text embedding $\hat{z}\in\mathbb{R}^{d}$.
Following standard fine-tuning practices, all layers for BERT are fine-tuned while only the last layer of DistilBERT is fine-tuned.

Bidirectional sequence encoder $g_{\psi}^{\mathsf{seq}}$: We broadcast the text embedding $\hat{z}$ to create a sequence of length $T$, $\hat{z}\rightarrow\{\hat{z}_{t}\}_{t=1}^{T}$, and concatenate each $\hat{z}_{t}$ with the timeseries input $x_{1:T}$, $\{\hat{z}_{t};x_{t}\}_{1}^{T}$.
This simplifies the task of learning timestep-specific correlations between system attributes $\hat{z}$ and timeseries $x_{1:T}$ in the multimodal encoder $g_{\psi}$.
To efficiently embed long timeseries with thousands of timesteps, we explore both bidirectional LSTMs (Hochreiter & Schmidhuber, ) and bidirectional SSMs (Goel et al., ) for $g_{\psi}^{\mathsf{seq}}$.
Our bidirectional SSM uses stacks of S4 blocks (Gu et al., ) without downpooling layers.
We use the last layer’s hidden states as temporal features $e_{1:T}$ for the top model.
If $T=1$ or for non-sequential surrogate models, we instead use an MLP with residual layers (ResNet MLP) to embed each $\{\hat{z}_{t};x_{t}\}$ per-timestep to get $e_{t}$.

Top model $h_{\theta}$: The multimodal encoder $g_{\psi}$ produces $T$ feature vectors $e_{1:T}$. For simplicity, the output $\hat{y}_{t}$ at each timestep is predicted from $e_{t}$ by a shared MLP with a single hidden layer.

## 6 Experiments

This section presents our experimental results, which include: quantifying the quality of LLM-generated SysCaps (Section 6.1), accuracy on held-out systems (Section 6.2), generalization under distribution shifts (Section 6.3), a design space exploration application (Section 6.4), and a study on SysCaps prompt augmentation (Section 6.5).
These experiments are based on two real-world CES simulators for buildings and wind farms.
All SysCaps are synthetically generated in this work. We provide additional qualitative examples of SysCaps in Appendix B.

Setup:
Our main experiments focus on training building stock surrogate models for the building energy simulator EnergyPlus (Crawley et al., ).
Given an annual hourly weather timeseries ($T$ = 8,760) with 7 variables and a list of tabular building attributes, surrogates predict the building’s energy consumption at each hour of the year.
Each building initially has 17 attributes; using RFE with a tuned LightGBM (Ke et al., ) model, we selected the 13 most important attributes.
We use the commercial building split of the Buildings-900K dataset (Emami et al., ), which are building stock simulation runs representative of the United States commercial building stock.
Commercial building stock surrogates can provide significant speedups compared to EnergyPlus, e.g., 96% (Zhang et al., ).
Since this dataset only provides energy timeseries, we extract the building configuration and weather timeseries from the End-Use Load Profiles database () for each building.
Our training set is comprised of 330K buildings, and we use 100 buildings for validation, and 6K held-out buildings for testing.
We also reserved a held-out set of 10K buildings for RFE.
We carefully tune the hyperparameters of all models (details in Appendix A.2).

We created three SysCaps datasets: a “medium” caption length dataset where <NUM> $\coloneqq$“4-6”, a “short” dataset using 2-3 sentences, and a “long” dataset using 7-9 sentences.
The SSMs in our experiments are trained with medium captions.
Generating these datasets with llama-2-7b-chat used $\sim$1.5K GPU hours on a cluster with 16 NVIDIA A100-40GB GPUs.

### 6.1 Evaluating caption quality

The LLM that generates natural language SysCaps may erroneously ignore or hallucinate attributes, which may negatively impact downstream performance.
To test this, we propose evaluating generated captions by estimating the fraction of attributes which the LLM successfully includes per caption.
To compute this metric, we train a multi-class classifier to predict each categorical attribute in a SysCaps from its text embedding.
We used a held-out validation set of captions to check that the classifier was not overfitting.
The test rate of missing or incorrect attributes is around 9-12% across the “short”, “medium”, and “long” caption types, with “short” captions having the highest error (Table 3).
This increases our confidence that our LLM-based approach for generating natural language SysCaps preserves sufficient information for surrogate modeling.

Table: Table 1: Accuracy. We show the mean NRMSE across 3 random seeds. Lower NRMSE is better. Building-hourly is the NRMSE normalized per building and per hour. Stock-annual first sums the predictions and targets over all buildings and hours, before computing the NRMSE (equivalent to the normalized mean bias error—see Appendix A.1). When using a BERT encoder instead of DistilBERT, SysCaps surrogates achieve better accuracy than one-hot baselines. We trained one SSM without any attribute information (Attribute Encoding “X”) as an ablation study; the poor accuracy shows that our multimodal architecture successfully learns to fuse the text-based attribute and timeseries inputs.

Figure: Figure 3: System captions unlock text-prompt-style surrogate modeling for complex systems. We show building stock daily load profiles aggregated for Warehouse building type, created with caption templates. From left to right, we use captions with one, three, and six attributes.
Refer to caption: x3.png

**Table 2: Caption quality. We estimate the presence of each attribute in a SysCap, measured by the average test accuracy of a multi-class classifier trained to predict each categorical attribute. Our metric suggests $\sim$9-12% of attributes are missing or incorrect per SysCap, due to errors made by llama-2-7b-chat.**
| Caption length (13 attributes) | Accuracy (%) |
| --- | --- |
| Short (2-3 sent.) | 88.90 |
| Medium (4-6 sent.) | 90.90 |
| Long (7-9 sent.) | 90.38 |

Table: Table 4: Generalization to attribute synonyms. We quantify how text embeddings make our models robust to the use of attribute synonyms. The metric is the difference in NRMSE between the original caption and the modified caption. We bold the best method, i.e, closest to 0. Column 3 replaces the building type with a synonym, column 4 removes the building type and sub-type attributes from the caption, and column 5 randomly swaps the correct building type attributes with incorrect ones.

### 6.2 Accuracy On Held-Out Systems

Following Emami et al. (), we use the normalized root mean square error (NRMSE) metric to compare model accuracy, averaged across 3 random seeds.
In addition to comparing the ResNet MLP, LSTM, and SSM, we also trained a tuned LightGBM baseline.
Does the sequential architecture matter?
*Yes*—Table 1 shows that the LSTM and SSM encoders outperform both the ResNet and our carefully tuned LightGBM baseline, and the SSM outperforms the LSTM.
How do different system attribute encoding approaches compare?
Surprisingly, the best SSM and LSTM with key-value SysCaps achieve comparable building-hourly test accuracy to one-hot baselines and the *best* accuracy at the stock-annual aggregation level (equivalent to the normalized mean bias error—see Appendix A.1).
The SSM and LSTM with natural language SysCaps has slightly worse accuracy than with the key-value SysCaps, yet they comfortably outperform the non-sequential models (including LightGBM) and the one-hot baselines at the stock-annual aggregation level.
We initially expected to see a non-negligible drop in regression accuracy for SysCaps models, even key-value SysCaps, because the text encoder has to compress the caption into a single embedding vector.
However, the BERT encoder is sufficiently expressive to mitigate this.
We observe that the stock-level NRMSE reduces by about half when switching from DistilBERT to BERT.
We believe the performance gap between key-value templates and natural language SysCaps is mostly explained by the caption quality (Table 3).
We ablate the importance of the system attributes by training an SSM baseline with these inputs removed (SSM with Attribute Encoding “X”); this model is unable to learn this task.

Figure 3 qualitatively shows how a SysCaps model performs with natural language captions (created with a sentence template) provided to the model.
Note that we did not train our models on any captions with missing attributes.
Prediction accuracy improves as more information is given; notably, there is a large jump in accuracy once the building square footage is known.

### 6.3 Caption Generalization

Length generalization: We assess how accuracy varies when surrogates are provided with natural language SysCaps having different lengths than seen during training.
We evaluate zero-shot generalization to the short and long captions.
The results (Table 3) show a small increase in error for shorter captions with a larger increase in error for longer captions, as might be expected.
The error on long captions remains lower than the error achieved by our tuned LightGBM baseline.

Attribute synonyms: To quantitatively evaluate the extent to which natural language SysCaps surrogates gain a level of robustness to distribution shifts such as word order changes, synonyms, or writing style (Hendrycks et al., ), we created captions for the held-out systems where the “building type” attribute is replaced by a synonym.
We avoid biasing the choice of building type synonym by 5-shot prompting llama-2 to suggest the synonyms.
There are two baselines we compare the synonym caption accuracy against: 1) accuracy when testing the model on captions with the building type attribute removed, and 2) accuracy when testing the model on captions with a random building type.
Examples and results are shown in Table 4, where for 11/14 building type synonyms the increase in NRMSE is less than 13%, while the average increase for the two baselines is 54% and 90%, respectively.

### 6.4 Design Space Exploration Application Using Language

Figure: (a)
Refer to caption: x4.png

We visualize in Figure 4(a) a use of SysCaps to conduct a sensitivity analysis on two system attributes, as might be performed for an early-stage design space exploration task.
We use a simple template to create a caption for each test building that enumerates all combinations of the number of stories and square footage attributes, totaling 160 configurations; the entire analysis requires simulating 960K buildings, and took 1 hour on a single NVIDIA A100 GPU.
We observe that the model has indeed learned physically plausible relationships between these two attributes.
However, the model fails to predict the energy usage for buildings over 100K square feet—such buildings are in the “long tail” of the training data distribution.
Figure 4(b) also shows a failure case where the model underestimates energy usage at unseen numeric building square footage attribute values.

### 6.5 Prompt Augmentation: Wind Farm Wake

This experiment uses the Wind Farm Wake Modeling Dataset (Ramos et al., ), made with the FLORIS simulator, to train a surrogate to predict a wind farm’s power generation in steady-state atmospheric conditions.
The speed-up provided by surrogate models for downstream optimization use-cases is $\sim$700$\times$ (Harrison-Atlas et al., ).
The difficulty of this task is in modeling losses due to wake effects, given only a coarse description of the wind farm layout.
There are three numeric simulator inputs $x$ specifying atmospheric conditions, and five system attributes which include categorical variables indicating wind farm shape (four different layout types), number of turbines, and average turbine spacing. We do not use RFE.
In this dataset, there are only 500 unique system configurations (split 3:1:1 for train, val, test), although each configuration is simulated under 500 distinct atmospheric conditions.

**Table 5: Wind farm surrogate accuracy. The base architecture is ResNet. Average across 3 random seeds.**
| Model | NRMSE |
| --- | --- |
| LightGBM | 0.196$\pm 0.000$ |
| one-hot | 0.212$\pm$ 0.009 |
| SysCaps-kv | 0.054$\pm$ 0.024 |
| SysCaps-nl | 0.036$\pm$ 0.001 |

We explore generating multiple captions for each system configuration through prompt augmentation to increase diversity.
Specifically, we replace the <STYLE> tag in the prompt with phrases encouraging different description styles, e.g., *with an objective tone*, *with an objective tone (creative paraphrasing is acceptable)*, *to a colleague*, and *to a classroom*.
The simulation is run assuming steady-state conditions (i.e., time-independent), so we tune hyperparameters for and train the non-sequential ResNet models.
The ResNet baseline with one-hot encoded attributes suffers from severe overfitting (Table 5), likely due to the small number (300) of training systems, whereas the SysCaps models generalize better to unseen systems.
This suggests SysCaps can have a regularizing effect in small data settings.
Notably, the prompt augmentation helps the natural language SysCaps model to achieve the lowest NRMSE.

## 7 Conclusion

In this work, we introduced a lightweight, multimodal text and timeseries surrogate models for complex energy systems such as buildings and wind farms, and described a process for using LLMs to synthesize natural language descriptions of such systems, which we call SysCaps.
Our experiments showcase SysCaps-augmented surrogates that achieve better accuracy than standard feature engineering (e.g., one-hot encoding) while also enjoying the advantages of using text embeddings such as robustness to caption paraphrasing (e.g., synonyms of attributes).
For a problem with only a small number of training systems available, we showed that SysCaps prompt augmentation has a regularizing effect that helps mitigate overfitting.
Overall, these results underscore that language is a viable interface for interacting with real-world surrogate models.

Limitations:
Current BERT-style tokenizers struggle with numerical values (Wallace et al., ); for one example, they interpolate poorly to unseen numbers (Figure 4(b)).
For another, because llama-2-7b-chat tends to add a comma to large numbers (e.g., $200,000$) when generating SysCaps, we found that our surrogates failed to understand large numbers without commas (had high error).
Orthogonal research on improving number encodings for language model inputs (Golkar et al., ; Yan et al., ) can benefit our framework.
Another potential concern is with creating SysCaps for simulators with a large number of attributes (e.g., over 100).
A more powerful LLM than llama-2-7b-chat with a longer context window may be needed in this case.
In general, we expect that more powerful LLMs will further improve the quality of the training captions.

Broader impacts: Improving surrogate models for CES has the potential to accelerate the transition to cleaner energy sources. To avoid unfair outcomes from decisions made with CES surrogates, care should be taken when deciding what simulation runs to use as training data and when selecting an LLM to use for caption generation.

Future work: A future extension of this work might explore how to use language to also interface with the timeseries simulator inputs, possibly through summary statistics.
For example, to study how the complex system behaves when the average exogenous temperature is increased by five degrees.
Another promising extension is to conduct a series of human studies with domain scientists to evaluate the quality of SysCaps in the context of, e.g., system design optimization.
Moreover, an important question is how we might create *surrogate foundation models* that generalize not only across system configurations for a single simulator, but also generalize across different simulators.

## Acknowledgments

This work was authored by the National Renewable Energy Laboratory (NREL), operated by Alliance for Sustainable Energy, LLC, for the U.S. Department of Energy (DOE) under Contract No. DE-AC36-08GO28308. This work was supported by the Laboratory Directed Research and Development (LDRD) Program at NREL. The views expressed in the article do not necessarily represent the views of the DOE or the U.S. Government. The U.S. Government retains and the publisher, by accepting the article for publication, acknowledges that the U.S. Government retains a nonexclusive, paid-up, irrevocable, worldwide license to publish or reproduce the published form of this work, or allow others to do so, for U.S. Government purposes.
The research was performed using computational resources sponsored by the Department of Energy’s Office of Energy Efficiency and Renewable Energy and located at the National Renewable Energy Laboratory.

## Appendix A Additional Experiment Details

### A.1 Metrics

We use the normalized root mean square error (NRMSE) to capture the accuracy of the surrogate model. NRMSE is also known as (CV)RMSE.

The building-hour NRMSE is where the NRMSE is normalized by building and by hour, where $B$ is the number of buildings in the building stock and $T$ is the number of hours in a year:

$$ $:=\frac{1}{\frac{1}{BT}\sum y^{b}_{t}}\sqrt{\frac{1}{BT}\sum_{b=1,t=1}^{B,T}(y ^{b}_{t}-\hat{y}^{b}_{t})^{2}}.$ (2) $$

The stock-annual NRMSE, is where the NRMSE is normalized by the annual stock energy consumption:

$$ $:=\frac{1}{\sum y_{t}^{b}}\sqrt{\bigg{(}(\sum_{b=1,t=1}^{B,T}y^{b}_{t})-(\sum_ {b=1,t=1}^{B,T}\hat{y}^{b})\bigg{)}^{2}}.$ (3) $$

Notice that the square and square root cancel, making the stock-annual NRMSE equivalent to the normalized mean bias error.

We use the AdamW Loshchilov & Hutter () optimizer with $\beta_{1}$ = 0.9, $\beta_{2}$ = 0.98, $\epsilon$ = 1e-9, and weight decay of 0.01 for all experiments. The early stopping patience is 50 for all experiments.
All models are trained with a single NVIDIA A100-40GB GPU. The longest training runs take 1-2 days and the shortest 2-3 hours.

### A.2 Hyperparameters

Table: Table 6: Buildings: hyperparameters.

**Table 7: Wind: hyperparameters.**
| Model | Hyperparameter | Grid search space | Best values |
| --- | --- | --- | --- |
| LightGBM | Learning rate | From 0.01 to 0.1 | 0.039 |
| Number of leaves | From 40 to 120 | 108 |  |
| Subsample | From 0.6 to 1.0 | 0.963 |  |
| Feature fraction | From 0.6 to 1.0 | 0.997 |  |
| Min number of data in one leaf | From 20 to 100 | 96 |  |
| ResNet (one-hot) | Hidden layers size | [256,1024] | 256 |
| Number of layers | [2,8] | 2 |  |
| Batch size | [128,256] | 128 |  |
| Learning rate | [1e-5, 3e-4, 1e-3] | 1e-5 |  |
| ResNet (SysCaps-kv) | Hidden layers size | [256,1024] | 1024 |
| Number of layers | [2,8] | 8 |  |
| Batch size | [128,256] | 256 |  |
| Learning rate | [1e-5,3e-4,1e-3] | 3e-4 |  |
| ResNet (SysCaps-nl) | Hidden layers size | [256,1024] | 1024 |
| Number of layers | [2,8] | 8 |  |
| Batch size | [128,256] | 256 |  |
| Learning rate | [1e-5,3e-4,1e-3] | 1e-5 |  |

See Table 6 and Table 7.

#### A.2.1 Buildings

Figure: Figure 5: A visualization of the sequential surrogate model baseline with one-hot encoded system attributes for the buildings experiment.
Refer to caption: x6.png

There are 13 attributes after RFE, which are one-hot encoded into a 336-dimensional feature vector that gets embedded into 128 dimensions, whereas the text embeddings are 768-dimensional. We concatenate cyclically encoded calendar features to 7 weather variables, creating a 103-dimensional vector (Emami et al., ). For the one-hot models (Figure 5), this creates a 128 + 103 = 231 dimensional input for the bidirectional sequence encoder, and for the text models, it is a 768 + 103 = 871 dimensional input.

LightGBM: As LightGBM does not support batch training out of the box, the entire training data needs to be loaded into the memory to train a LightGBM model. With the train dataset containing 340k buildings, each with 8759 hours and 347 features, we randomly extract 438 hours per building (which is about 5% of total hours) to limit memory usage. This results in $340,000\times 438$ hours in total for the train dataset, which consumes about 380 GB of memory when being loaded into a NumPy object. For the validation and test splits, we retain the full number of hours per building.
The LightGBM model is tuned with Optuna (Akiba et al., ) across 30 trials and achieves the best validation NRMSE of 0.667.

#### A.2.2 Wind farm

LightGBM: The training, validation, and test split for the wind dataset gives us datasets of size 148,650, 49,600, and 49,250 respectively with 190 features after one-hot encoding. The training dataset is loaded into memory to train the LightGBM model. We use Optuna (Akiba et al., ) to tune the hyperparameters and the best validation NMRSE achieved is 0.189.

## Appendix B Additional SysCaps examples

In this section we visualize examples of prompts and the corresponding SysCaps outputs for the buildings simulator (Figures 6, 7, and 8) and the wind farm simulator (Figures 9, 10, and 11).

Figure: Figure 6: Natural language building SysCap. The instruction and list of key-value attributes (first two paragraphs) are provided to the LLM as the prompt. We observe that the LLM automatically converts numbers to more human-interpretable descriptions (e.g., the number of stories is changed from $1.0$ in the prompt to “single-story”). The LLM also succinctly (and correctly) states that “there is no difference in the temperature set points when the space is unoccupied” in the output by summarizing the “unoccupied heating/cooling temperature set point difference from occupied state: 0.0” attributes.
Refer to caption: x7.png

Figure: Figure 7: Natural language building SysCap. The LLM makes a subtle logical error here—the unoccupied heating temperature set point difference from occupied state suggests the set point should be *lower* by 10 degrees, not higher (highlighted in blue). Logical errors such as this may have contributed to the slightly worse accuracy of natural language SysCaps compared to key-value SysCaps. Collaborating with domain experts able to validate the LLM’s outputs is important for catching these errors. We expect that using more powerful LLMs will help reduce logical errors.
Refer to caption: x8.png

Figure: Figure 8: Natural language building SysCap. The LLM confuses the weekday closing time with the weekend open time (highlighted in blue).
Refer to caption: x9.png

Figure: Figure 9: Natural language wind farm SysCap.
Refer to caption: x10.png

Figure: Figure 10: Natural language wind farm SysCap.
Refer to caption: x11.png

Figure: Figure 11: Natural language wind farm SysCap with a logical error where it says "total installed capacity", but it just states the capacity of a single turbine (highlighted in blue). The correct total installed capacity is number of turbines times 3.4, or 431.8 MW.
Refer to caption: x12.png

## Peer Reviews

confidence: 2
contribution: 2
initial_rating: 5
presentation: 3
questions: I would be interested in seeing the performance compared between time-series-centric models and current generic architectures.
soundness: 2
strengths: - The authors provide a clear explanation of their setup, flow of data across multiple components and their evaluation and analysis.
- (I do not feel sufficiently well-acquainted with this domain to evaluate the predictive contribution or performance of the models.)
summary: This paper describes a set of lightweight models to model complex energy systems, using an LLM to generate prompts and a encoder and bidirectional time-series model to predict energy consumption.
weaknesses: - I presume the authors' choice of models is due to resource constraints and aiming for a lightweight setup, but it feels like it has multiple components when it could be a simpler setup with fewer model components. For instance, a BERT-type model could also be used for time-series prediction (as opposed to only text encoding). Similarly the two-step process of generating prompts using a separate LLM and then encoding that prompt with an encoder could be avoided by just using the LLM directly and fine-tuning it.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: In addition to the points in the weakness:

1. Did the authors try to just templatize the sentences rather than generating them using an LLM, how would that impact performance (i.e. rather than telling an LLM to adhere to some constraint-based template, just have a sketch sentence and fill attribute values in the given sentence)?
2. Why wasn't RFE performed for the Wind Farm Wake modeling dataset, would performing RFE improve performance ?
3. Would the model not further improve if the SysCaps were generated using synonyms for the attributes, did the authors see the LLM generate different synonyms for the building or wind farm dataset? 
4. Do the authors believe that training on the subset of data where the caption quality assessed by the classifier model, would improve the overall model performance?
soundness: 3
strengths: The paper is well-written and easy to follow. The authors motivate the problem well and empirically show improvements over 2 real-world datasets. Further, SysCaps can be used by non-expert users to understand the features of surrogate systems. The Design space exploration is insightful to show the features learned by the model.
summary: The paper discusses the important challenge of building surrogate models for the prediction of simulation data. They specifically motivate the problem for complex energy systems(CES). These surrogate models often model system features as one hot vectors. The authors propose using text based descriptions to model these so-called surrogate systems with time series data. The text data is encoded as a dense embedding obtained from language models. The embedding is then fed to a bidirectional sequence encoder along with the time series data. 

The paper discusses the generation of the text pertaining to the attributes of such systems and proposes an automatic evaluation strategy for the same. 

For generating the captions the authors prompt an LLM with an in-context learning-based prompt that tunes the style and number of sentences. To evaluate the SysCap quality the authors train a multi-class classifier to check the attributes covered in the description generated by the LLM, using the text embedding. 

The authors show how including SysCaps along with time series data leads to improved performance against baselines that perform onehot encoding over attributes. The authors further show how training a custom embedding model can aid in improving time series prediction over just using a time series-based model. They further empirically prove how the embeddings are more robust to synonyms and missing data.
weaknesses: 1. The paper claims the technique uses a pretrained text encoder for generating the embeddings, but then in section 5 mentions that the models are actually finetuned. This should be explicitly mentioned in the claims that the paper makes rather than just mentioning that a pretrained embedding is used. 
2. Further, the authors do not compare with the "said-pretrained" embeddings but only finetuned embeddings, and other SOTA embedding models for text encoding. 
3. The paper also claims that they train a system to evaluate the caption quality, the parameters of the said multiclass classifier are omitted from the paper.
4. The paper claims that for the CES building energy consumption dataset, the SysCaps-kv configuration works best, and for the turbine configuration the SysCaps-nl, there should be some discussion regarding the insights drawn from both cases and why the performance for both techniques are different. 
5. The authors claim that SysCaps would be useful for non-expert users, but lack the discussion if LLM-based explanations (complementary to the work done) can also aid in explaining the system attributes for surrogate models.

---

confidence: 2
contribution: 2
initial_rating: 6
presentation: 3
questions: * Q1) What's the advantage of the proposed approach using LLMs over more traditional template-based natural language captions? (e.g. "The building is <x> squared feet.", etc.)
* Q2) In Figure 1, the key-value template has only a colon to separate the key and the value. Have you tried adding a space in between? I expect 
* Q3) For the one-hot encodings, how do you deal with numeric inputs?
* Q4) In the results in Table 3, why did you expect longer captions to have larger error? I would have had the opposite intuition as shorter captions are more likely to miss important attributes.
soundness: 3
strengths: * S1: Provides extensive empirical evaluation of the proposed system
* S2: The presentation is clear.
summary: This paper studies the use of natural language captions as inputs to surrogate models that simulate "complex energy systems". These natural language captions describe the features of the system being simulated. The task is to predict a timeseries of some variable of interest that depends on these features and some other independent variable that is fed as a time series. The paper introduces an architecture that fuses the textual description with the time series data to achieve this goal.
The viability of the approach and its robustness to out-of-distribution perturbations are validated with a relatively extensive empirical evaluation, including different ablations of the system (such as one-hot encoding of the features, or no features), variations on the caption lengths or replacing words with synonyms.
weaknesses: * W1: The LightGBM baseline is underspecified. This baseline is the only one that stands as a reference point that is not an ablated version of the proposed model. However, as I understand it, LightGBM is a framework but not necessarily a model, so I don't really to which model this system is really being compared against.
* W2: Not very clear what is the added value of the proposal of using LLMs against simply using a template-based natural language description.
* W3: Despite the system is motivated on the potential intuitiveness of language interfaces to non-experts, no particular study is conducted on that front.

Meta-review: Reviewers liked that the paper solves the well-motivated problem of interacting with complex energy system (CES) using natural language and the strength of the results. They did not evaluate much of the technical aspects and provided lower confidence reviews: 38tw: could use BERT to directly predict the time series. 2gzL: unclear if fine-tuned or not. Gxzq: how LightGBM was used as baseline is not clear. It is clear that the paper focuses on specialized area that could be important, but also very few reviewers understand. This is also out of the domain for me, so I defer to reviewers on the importance of the problem and recommend accept to err on the positive side.

Decision: Accept (Poster)