Evaluating LLM Decision-Making in Security Vulnerability Testing: Correlation with Tokenized Source Code Length
Authors: Jie Lin, David Mohaisen


Reviewer paper summaries: This paper studies how tokenized source code length impacts LLM’s vulnerability detection abilities. The paper found that some more advanced models are more robust to code length while others are more sensitive.
Summary of strengths: + The paper’s analysis can be useful for understanding various LLM’s long-context understanding capabilities.
Summary of weaknesses: The paper is poorly written and has limited contribution that can warrant publication at *ACL venues. 
- Essentially the paper evaluates different models on a very small dataset (n=140) and presents a very rudimentary analysis that at best can be one or two paragraphs in a full paper.
- The dataset used is very small (n=140)  and extremely biased with only positive examples. This makes any claims made in this paper much less convincing.
- The writing is confusing, with no motivation for the problem studied, poorly described related work, unnecessarily verbose methodologies, and hard-to-parse analysis results.
Comments, suggestions and typos: ##### Terminology
- Where does the term “vulnerability testing” come from? I suspect the authors want to study “vulnerability detection”?
- “Decision making” is a very inaccurate description of what is simply classification.

##### Introduction
What motivates the paper to study “how the tokenized length of input source code affects the decision-making accuracy of LLM responses in vulnerability testing”? I assume it is about long-context capabilities. If that is the case, please make it explicit and provide more background information.

##### Related work
The entire section doesn’t have useful information. The two works in the “Understanding” paragraph are barely connected. The detailed descriptions of “models” and “more applications”  are simply irrelevant to this work. The authors might want to list some related work on lone-context LLMs and relevant evaluation and analysis papers here instead.

##### Methodologies
- The author did a manual analysis of the LLM responses, which can limit the scale of evaluation they can do. As an alternative, consider using another LLM to extract classification results, or instructing LLMs to output JSON-style structured responses for programmatic extraction.
- The evaluation used the same tokenizer for all models. How and why is this decision made, instead of using each model’s own tokenizer?
- It is unclear why the paper wants to study “explicitness”. Please motivate better.
- The hypotheses are poorly described. H0_1 talks only about indicating vulnerabilities, which is different from correctness. H0_2 does not have any references to “explicitness”.
- The authors claimed that they determined the setting for Chi-Square Test “upon various attempts”, which is not a great description of a supposedly scientific procedure.

##### Results
There are lots of findings for each model family, but what are the high-level findings? I can see some efforts in the summary paragraph but the results are still hard to parse.
Overall assessment scores: 1