ConfigSense: Utilizing LLMs for the Efficient Identification of Performance-Sensitive Configurations in Software Systems
Authors: Zehao Wang, Dong Jae Kim, Tse-Hsun Chen


Reviewer paper summaries: This paper presents ConfigSense, an LLM-based agent system to identify performance-sensitive configurations in software systems. ConfigSense uses two agents: (1) DevAgent to conduct performance-aware code review and (2) PerfAgent to iteratively gather information and classify the configuration. The evaluation found the system has higher accuracy than baselines, with additional ablation analysis and error analysis results.
Summary of strengths: Overall, I found this paper well-written and clearly structured. The paper builds an LLM-based system to solve a real-world problem, identifying performance-sensitive configuration. Its design and analysis can inform other LLM application builders in better leveraging LLM capabilities.
Summary of weaknesses: My major concerns for this paper are (1) a lack of details on implementation and (2) not well-justified evaluation design on metrics, datasets, and baselines.

1. After reading Section 3, I was fairly confused about how the paper exactly implements (1) the retrieval system and (2) memory mechanism. The appendix A.1 mentions LangGraph but did not explicitly says whether the retrieval and memory implementations are entirely based on LangGraph APIs. Adding a quick implementation overview in the main paper would make everything much more clearer.
2. The evaluation claims that ConfigSense is better than the ChatGPT baseline as it has a higher accuracy and recall. However, it seems that the ChatGPT baseline has a higher precision and F1-score (even though F1 is unreported in the paper). The paper does argue that “our primary objective with ConfigSense is to maintain high recall alongside reasonable precision”, but has no criteria of what precision being reasonable (61.47% of ConfigSense is but 49.58% of ChatGPT is not?). 
3. The paper uses “seven open-source Java applications” as the evaluation dataset, but never justifies this choice. It seems that this dataset is introduced by a previous work from 2023 – is this dataset a well-used benchmark for identifying performance-sensitive configurations? What are the benchmarks out in the community and why did the paper choose this specific one? It is particularly concerning, as for this dataset, there is only one existing baseline DiaConfig, with results on a small fraction of it. 

Comparing to more established baselines with better-justified metrics and criteria would make the claims this paper makes (i.e., LLM-based agent system helps identify performance-sensitive configurations) much more convincing.
Comments, suggestions and typos: ##### Evaluation

The descriptions on metrics (“accuracy, precision, and recall metrics”) are unnecessarily verbose for this audience.

Table 2 could have shown the aggregated results for readers to compare the ablations more easily.

The first reason in RQ3 “Most misclassifications (54.1%) occur in configurations without clear evidence to support the performance sensitivity.” is confusing. At first glance, I thought this indicates a ground truth issue – i.e., there is no enough information to conclude performance (in-)sensitivity, but what it really means is that ConfigSense has a wrong prediction and there is no specific reason why?


##### References
For prompt chaining, please cite some papers on this direction. For example,

- Wu, Tongshuang, et al. "Promptchainer: Chaining large language model prompts through visual programming." CHI Conference on Human Factors in Computing Systems Extended Abstracts. 2022.
- Madaan, Aman, et al. "Self-refine: Iterative refinement with self-feedback." Advances in Neural Information Processing Systems (2023).


##### Typos
- Line 136: “reduces” -> “requires”?
- Line 137: “identical” -> “identify”
Overall assessment scores: 3