SysCaps: Language Interfaces for Simulation Surrogates of Complex Systems

Authors: Patrick Emami, Zhaonan Li, Saumya Sinha, Truc Nguyen
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)
Keywords: surrogate models, multimodal text and timeseries models, language-interfaced regression

Abstract:
Surrogate models are used to predict the behavior of complex energy systems that are too expensive to simulate with traditional numerical methods. 
Our work introduces the use of language descriptions, which we call "system captions" or SysCaps, to interface with such surrogates. 
We argue that interacting with surrogates through text, particularly natural language, makes these models more accessible for both experts and non-experts.
We introduce a lightweight multimodal text and timeseries regression model and a training pipeline that uses large language models (LLMs) to synthesize high-quality captions from simulation metadata. 
Our experiments on two real-world simulators of buildings and wind farms show that our SysCaps-augmented surrogates have better accuracy on held-out systems than traditional methods while enjoying new generalization abilities, such as handling semantically related descriptions of the same test system.
Additional experiments also highlight the potential of SysCaps to unlock language-driven design space exploration and to regularize training through prompt augmentation.

Decision: accept (poster)
Reviewer scores: 5, 8, 8

Reviews and discussion:
<thread>
creation date:
1730379432088

thread_type:
official_review

<thread_object>
creator: reviewer_gxzq

<content>
summary:
This paper studies the use of natural language captions as inputs to surrogate models that simulate "complex energy systems". These natural language captions describe the features of the system being simulated. The task is to predict a timeseries of some variable of interest that depends on these features and some other independent variable that is fed as a time series. The paper introduces an architecture that fuses the textual description with the time series data to achieve this goal.
The viability of the approach and its robustness to out-of-distribution perturbations are validated with a relatively extensive empirical evaluation, including different ablations of the system (such as one-hot encoding of the features, or no features), variations on the caption lengths or replacing words with synonyms.

strengths:
* S1: Provides extensive empirical evaluation of the proposed system
* S2: The presentation is clear.

weaknesses:
* W1: The LightGBM baseline is underspecified. This baseline is the only one that stands as a reference point that is not an ablated version of the proposed model. However, as I understand it, LightGBM is a framework but not necessarily a model, so I don't really to which model this system is really being compared against.
* W2: Not very clear what is the added value of the proposal of using LLMs against simply using a template-based natural language description.
* W3: Despite the system is motivated on the potential intuitiveness of language interfaces to non-experts, no particular study is conducted on that front.

questions:
* Q1) What's the advantage of the proposed approach using LLMs over more traditional template-based natural language captions? (e.g. "The building is <x> squared feet.", etc.)
* Q2) In Figure 1, the key-value template has only a colon to separate the key and the value. Have you tried adding a space in between? I expect 
* Q3) For the one-hot encodings, how do you deal with numeric inputs?
* Q4) In the results in Table 3, why did you expect longer captions to have larger error? I would have had the opposite intuition as shorter captions are more likely to miss important attributes.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730692012320

thread_type:
official_review

<thread_object>
creator: reviewer_2gzl

<content>
summary:
The paper discusses the important challenge of building surrogate models for the prediction of simulation data. They specifically motivate the problem for complex energy systems(CES). These surrogate models often model system features as one hot vectors. The authors propose using text based descriptions to model these so-called surrogate systems with time series data. The text data is encoded as a dense embedding obtained from language models. The embedding is then fed to a bidirectional sequence encoder along with the time series data. 

The paper discusses the generation of the text pertaining to the attributes of such systems and proposes an automatic evaluation strategy for the same. 

For generating the captions the authors prompt an LLM with an in-context learning-based prompt that tunes the style and number of sentences. To evaluate the SysCap quality the authors train a multi-class classifier to check the attributes covered in the description generated by the LLM, using the text embedding. 

The authors show how including SysCaps along with time series data leads to improved performance against baselines that perform onehot encoding over attributes. The authors further show how training a custom embedding model can aid in improving time series prediction over just using a time series-based model. They further empirically prove how the embeddings are more robust to synonyms and missing data.

strengths:
The paper is well-written and easy to follow. The authors motivate the problem well and empirically show improvements over 2 real-world datasets. Further, SysCaps can be used by non-expert users to understand the features of surrogate systems. The Design space exploration is insightful to show the features learned by the model.

weaknesses:
1. The paper claims the technique uses a pretrained text encoder for generating the embeddings, but then in section 5 mentions that the models are actually finetuned. This should be explicitly mentioned in the claims that the paper makes rather than just mentioning that a pretrained embedding is used. 
2. Further, the authors do not compare with the "said-pretrained" embeddings but only finetuned embeddings, and other SOTA embedding models for text encoding. 
3. The paper also claims that they train a system to evaluate the caption quality, the parameters of the said multiclass classifier are omitted from the paper.
4. The paper claims that for the CES building energy consumption dataset, the SysCaps-kv configuration works best, and for the turbine configuration the SysCaps-nl, there should be some discussion regarding the insights drawn from both cases and why the performance for both techniques are different. 
5. The authors claim that SysCaps would be useful for non-expert users, but lack the discussion if LLM-based explanations (complementary to the work done) can also aid in explaining the system attributes for surrogate models.

questions:
In addition to the points in the weakness:

1. Did the authors try to just templatize the sentences rather than generating them using an LLM, how would that impact performance (i.e. rather than telling an LLM to adhere to some constraint-based template, just have a sketch sentence and fill attribute values in the given sentence)?
2. Why wasn't RFE performed for the Wind Farm Wake modeling dataset, would performing RFE improve performance ?
3. Would the model not further improve if the SysCaps were generated using synonyms for the attributes, did the authors see the LLM generate different synonyms for the building or wind farm dataset? 
4. Do the authors believe that training on the subset of data where the caption quality assessed by the classifier model, would improve the overall model performance?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730795624836

thread_type:
official_review

<thread_object>
creator: reviewer_38tw

<content>
summary:
This paper describes a set of lightweight models to model complex energy systems, using an LLM to generate prompts and a encoder and bidirectional time-series model to predict energy consumption.

strengths:
- The authors provide a clear explanation of their setup, flow of data across multiple components and their evaluation and analysis.
- (I do not feel sufficiently well-acquainted with this domain to evaluate the predictive contribution or performance of the models.)

weaknesses:
- I presume the authors' choice of models is due to resource constraints and aiming for a lightweight setup, but it feels like it has multiple components when it could be a simpler setup with fewer model components. For instance, a BERT-type model could also be used for time-series prediction (as opposed to only text encoding). Similarly the two-step process of generating prompts using a separate LLM and then encoding that prompt with an encoder could be avoided by just using the LLM directly and fine-tuning it.

questions:
I would be interested in seeing the performance compared between time-series-centric models and current generic architectures.
</content>
</thread_object>

</thread>


Metareview:
metareview:
Reviewers liked that the paper solves the well-motivated problem of interacting with complex energy system (CES) using natural language and the strength of the results. They did not evaluate much of the technical aspects and provided lower confidence reviews: 38tw: could use BERT to directly predict the time series. 2gzL: unclear if fine-tuned or not. Gxzq: how LightGBM was used as baseline is not clear. It is clear that the paper focuses on specialized area that could be important, but also very few reviewers understand. This is also out of the domain for me, so I defer to reviewers on the importance of the problem and recommend accept to err on the positive side.