Generate-then-Test: Automated Test Case Generation for WebAssembly Using Large Language Models

Authors: Weike Fang, Kaicheng Wang, Weihang Wang
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: WebAssembly, Large Language Models, Test Case Generation

Abstract:
The reliability and security of WebAssembly (Wasm) binaries are crucial for modern web development, yet effective testing methodologies remain undeveloped. This paper addresses the gap in Wasm binary testing by proposing a novel approach for test cases generation, leveraging Large Language Models (LLMs) to enhance test coverage and bug detection. Traditional testing approaches typically require access to source code, which is often unavailable for Wasm binaries. Our generate-then-test methodology overcomes this limitation by generating equivalent C++ code to simulate expected Wasm behavior, creating and mutating test cases in C++, and compiling these tests to evaluate them against the Wasm binary. Key contributions include automated test case generation using LLMs and improved code coverage through type-aware mutations, with comprehensive evaluation demonstrating the effectiveness of our approach in detecting subtle bugs in Wasm binaries, thereby ensuring more reliable Wasm applications.

Decision: reject
Reviewer scores: 3, 3, 3, 3

Reviews and discussion:
<thread>
creation date:
1729254748330

thread_type:
official_review

<thread_object>
creator: reviewer_xrym

<content>
summary:
This paper proposes WasmTest, a tool to automatically detect bugs in Wasm binaries leveraging the power of LLMs. WasmTest converts the Wasm binaries into functionally equivalent C++ code by LLMs, and then generate testing inputs based on the C++ code by combined efforts from LLMs and type-aware mutations to detect bugs in the Wasm binaries. The authors evaluated WasmTest on 2 benchmarks, the HumanEval-X and MBXP. The evaluation demonstrated that WasmTest can generate C++ code that has high pass rate; the generated test cases can achieve high code coverage and high bug detection rate in Wasm binaries on manually injected bugs.

strengths:
1. The presentation of the paper is clear and easy to follow.
2. The testing inputs generation of WasmTest can generate test cases that achieve high code coverage.
3. Evaluation on multiple LLMs demonstrate the robustness against the choice of LLMs.

weaknesses:
## Summarization of major weaknesses

1. The use cases of WasmTest is not clear.
2. The use of "functionally equivalent" C++ code as the testing oracle, i.e., the ground truth of testing outcomes, is dubious. WasmTest cannot guarantee the functional equivalence of the generated C++ code with the original Wasm binaries.
3. Lack of comparison with other binary-lifting tools for Wasm like wasm2c.
4. Lack of comparison with directly generating testing inputs for Wasm binaries.

## Detailed comments

**Significance**:

The use cases of WasmTest is not particularly clear. This paper targets testing the functional bugs of Wasm binaries where the original source code is not available. The paper does not clearly state the intended audiences (or users) of the proposed tool:

1. If WasmTest is intended for attackers, then targeting functional bugs may not be interesting to them, since functional bugs typically have less severe consequences and cannot be easily exploited. Also, attackers may not have access to the natural language descriptions of the Wasm binaries, which are necessary for WasmTest to work.
2. WasmTest is unlikely to be of interests to developers, as they typically have access to the original source code and can directly test the source code.

In either cases, the WasmTest is not particularly useful.

**Originality**:

The idea of using LLMs for both binary lifting and generating test cases are already explored, as mentioned in the related work section of this paper. The novelty of this paper is the combination of them into an end-to-end testing pipeline for Wasm binaries. Yet the usability of the lifting-then-testing approach is not clear. Relying on the pass rates and the majority voting to ensure the functional equivalence of the generated C++ code is not convincing. There can be cases where the generated C++ code passes all example test cases, but still not functionally equivalent to the original Wasm binaries. Without ensuring the correctness of the C++ code, employing it as the testing oracle can lead to false positives.

**Soundness**:

The evaluation of this paper requires significant improvement:

1. The authos do not mention the false positive rate of the testing outcomes. Since the generated C++ code may not be equivalent to the original Wasm binaries, a disagreement between the outcomes from the C++ code and the Wasm binaries may not necessarily indicate a bug in the Wasm binaries, but rather a functional difference.
2. The HumanEval-X and MBXP benchmarks are not complex enough. For real-world projects, the Wasm binaries can be much more complex than a few hundred lines of code. Testing on more complex Wasm binaries is necessary to demonstrate the performance of WasmTest on more realistic scenarios.
3. The evaluation does not compare generating testing inputs directly for Wasm binaries. Without such comparison, the effectiveness of lifting to C++ code is not clear.
4. The evaluation does not compare the performance of WasmTest with other binary-lifting tools for Wasm like wasm2c. Since traditional binary-lifting tools can also lift Wasm binaries to C/C++ code, the use of LLMs in WasmTest may not be necessary.
5. The authors do not evaluate the performance of WasmTest when the natural language descriptions of the Wasm binaries are not available. In cases where only the binaries are available, which can be common in practice, the performance of WasmTest is unknown.

In sum, the evaluation of this paper is not sufficient to demonstrate the effectiveness of WasmTest.

questions:
1. What are the main intended audiences and use cases of WasmTest?
2. What is the false positive rate of the generated testing results?
3. How to ensure that the generated C++ code is functionally equivalent to the original Wasm binaries? Passing example tests or majority voting cannot guarantee that.
4. How does the performance of WasmTest compare with directly generating testing inputs for Wasm binaries?
5. How does WasmTest perform when the natural language descriptions of the Wasm binaries are not available?
6. What is the performance of WasmTest on more complex Wasm binaries other than the ones presented in the benchmarks?
7. How does the Wasm-to-C++ conversion compare with other binary-lifting tools for Wasm like wasm2c?
</content>
</thread_object>

</thread>
<thread>
creation date:
1729410252251

thread_type:
official_review

<thread_object>
creator: reviewer_52kk

<content>
summary:
- This paper propose a new approach, named WasmTest, that leverages LLMs to automate the generation of intermediate C++ code and test cases from high-level descriptions of Wasm functionalities.
- This paper includes a type-aware mutation strategy for mutation testing
- The evaluation results of this paper indicates the effectiveness of WasmTest through compile rate, code coverage, test correctness and bug detection rate

strengths:
- This paper is clearly written and easy to comprehend
- This paper tries to address an important and difficult problem, Wasm test generation

weaknesses:
Generally, I think this paper tries to address a typical software engineering task, automated test generation; however, it lacks a comprehensive empirical study about recent work, which needs to be mentioned and compared, and thus incorrectly claims that "effective testing methods remain underdeveloped" (Line 47).

Specifically, I have the following concerns:
- This paper overlook a large number of existing Wasm test generation approaches, e.g., WASMaker [1], wasm-smith [2], Wasm-mutate [3]
- The motivation of translating Wasm code into C++ code and then generate test cases is not clearly explained. Now that you utilize LLMs to comprehend the semantics of Wasm code, why not directly use LLMs to generate Wasm tests?
- This technical contribution of this paper is limited as each component in its approach has been proposed before. 1) Wasm has existing reverse engineering tool, wasm2c [5], so you need to explain why to use LLM for code translation and add comparison evaluation between your approach and wasm2c. 2) The step 3 in your approach (type-aware mutation). There exists a large portion of mutation testing approaches, even for c/c++ programs, e.g., CCmutator [4]. Additionally, the SOTA mutation testing approaches also include LLM, e.g., LEAM [6], which I suggest to consider.
- The evaluation section needs to be improved with more details about its setup. 1) Now that you leverage LLMs for code translation, you need further evaluation to show the semantic-preserving of your approach; 2) As for bug detection, you need to explain more details about your manually crafted bugs (Line 447), e.g., the fault types, the background of your involved engineers, which is important in the area of software engineering as this paper is an AI4SE one; 3).



These reference might be considered:

[1] Cao S, He N, She X, et al. WASMaker: Differential Testing of WebAssembly Runtimes via Semantic-Aware Binary Generation[C]//Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis. 2024: 1262-1273.

[2] Bytecode Alliance. 2023. Github wasm-tools repository. https://github.com/bytecodealliance/wasm-tools/tree/main/crates/wasm-smith

[3] Arteaga J C, Fitzgerald N, Monperrus M, et al. Wasm-mutate: Fuzzing WebAssembly compilers with e-graphs[C]//E-Graph Research, Applications, Practices, and Human-factors Symposium. 2022.

[4] Kusano M, Wang C. CCmutator: A mutation generator for concurrency constructs in multithreaded C/C++ applications[C]//2013 28th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 2013: 722-725.

[5] https://github.com/WebAssembly/wabt/wasm2c

[6] Tian Z, Chen J, Zhu Q, et al. Learning to construct better mutation faults[C]//Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering. 2022: 1-13.

questions:
1. What is the difference between your WasmTest approach and existing approaches, e.g., WASMaker, wasm-smith, Wasm-mutate?
2. What is the difference between your step 1 and existing Wasm reverse engineering approaches, especially wasm2c?
3. What is the difference between your step 3 and existing mutation testing tools, e.g., CCmutator (actually mutation testing is also a matured research area in software engineering)?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730224053601

thread_type:
official_review

<thread_object>
creator: reviewer_m8cy

<content>
summary:
The paper introduces a testing methodology for WebAssembly (Wasm) binaries that uses Large Language Models (LLMs) to overcome the absence of source code. By generating equivalent C++ code from a high-level understanding of Wasm functionalities, the methodology facilitates the creation of seed test cases via LLMs and type-aware mutation to improve coverage and detect bugs. The approach is validated through metrics like compile rate, code coverage, and bug detection rate. Two benchmarks are used in the evaluation. The paper showcases a practical application of LLMs in addressing challenges associated with Wasm binary testing.

strengths:
- Originality

Wasm binary testing is an interesting and important problem. The primary innovation of the "Generate-Then-Test" methodology is its application of LLMs to generate equivalent C++ code based on abstract descriptions of Wasm functionalities. 

- Quality

The proposed approach appears to work, validated using two benchmark datasets and evaluated through several standard metrics
- Clarity

The paper is overall clear and well-organised. 

- Significance:

This paper offers a new way in the testing binaries distributed without source code and this might even go beyond its immediate application in testing Wasm binaries.

weaknesses:
Using LLMs to generate equivalent source code from binaries constitutes the major contribution of this paper, yet it also the bottleneck of the work. It is an interesting idea but it is also a challenging problem, isn't it? This method presents difficulties in effectively translating high-level descriptions into executable test cases, and the paper does not convincingly demonstrate the efficacy of this translation.

The experimental validation is somewhat limited due to the nature of the benchmarks used. The study employs only two source code datasets that are compiled to WebAssembly before applying the proposed methodology. This approach can create an artificial scenario tailored to the solution presented in the paper, rather than addressing the more complex and varied real-world challenges found in typical WebAssembly applications.

While the methodology is shown to be effective within the constraints of the selected benchmarks, the paper does not provide a comparison with existing binary testing approaches. This raises concerns about its advantages or improvements over traditional testing techniques.

The success of the "Generate-Then-Test" methodology relies heavily on the quality and accuracy of the high-level descriptions of Wasm functionalities. Inadequate or imprecise descriptions can lead to derived C++ code and test cases that do not accurately reflect the intended functionality of the Wasm binaries. The paper should clarify how the quality of these descriptions impacts the effectiveness of the generated tests and propose strategies to ensure the reliability of these descriptions in producing meaningful test cases.

questions:
- Validation of Description Quality
 
What methods or criteria could be used to ensure the quality and accuracy of the high-level descriptions used for generating C++ code? Can you provide detailed methodologies that validate the adequacy of these descriptions?

- Comparison with Existing Methods

How does the proposed testing approach compare with traditional binary testing methods 


- Generalizability and Scalability

How would the proposed testing perform on real-world applications?


- Impact of Inadequate Descriptions

What are the observed impacts or specific examples where inadequate descriptions have led to ineffective test generation? Can the authors provide case studies or examples?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730655200713

thread_type:
official_review

<thread_object>
creator: reviewer_2aew

<content>
summary:
This paper introduces WasmTest, an approach for testing of WebAssembly (Wasm) binaries. To overcome the challenges of testing Wasm binaries without source code, WasmTest leverages a generate-then-test methodology. It utilizes large language models (LLMs) to automatically generate equivalent C++ code, create and mutate test cases, and compile these tests for execution against the Wasm binary. Results demonstrate that WasmTest  generates comprehensive test suites.

strengths:
1. Novel Approach: Introduces a novel generate-then-test approach for black-box testing of WebAssembly (Wasm) binaries, leveraging the power of large language models (LLMs).
2. Effective Coverage: Demonstrates high test coverage for simple, single-program benchmarks.
3. Ease of Understanding: The proposed technique is straightforward and easy to comprehend.

weaknesses:
- **Limited Literature Review:** The paper lacks a comprehensive review of related work, particularly in the areas of LLM-based automated test generation (e.g., CODAMOSA, ChatUniTest, MuTAP) and LLM-based fuzzing (e.g., TitanFuzz, FuzzGPT, Fuzz4All). Additionally, the review of existing Wasm testing techniques is insufficient.
- **Inadequate Test Oracle:** The reliance on LLM-generated oracles, which exhibit approximately 50% accuracy, limits the practical applicability of the approach. The black-box testing paradigm may not be suitable for developer-focused scenarios, and general fuzzing techniques with sanitizers might be more effective for attacker-focused use cases.
- **Insufficient Evaluation of Generate-Then-Test:** The paper lacks a thorough evaluation of the generate-then-test approach. It's unclear whether directly generating tests for Wasm binary code using LLMs or applying simple fuzzing techniques like AFL or libfuzzer would yield comparable or superior coverage results. Additionally, it lacks an ablation study exploring the substitution of the first component with a simpler reverse engineering approach.
- **Lack of Novelty in Mutation:** The mutation techniques employed in the paper do not appear to be significantly novel.
- **Limited Evaluation Scope:** The evaluation is restricted to a single-program dataset, which may not accurately reflect real-world application scenarios

questions:
- What is the result of directly generating tests for Wasm binaries?
- Could you elaborate on the applications of this black-box setting? Specifically, who is expected to test the Wasm binaries in such a context?
- Could you provide more details on how equivalence with the C++ implementation is verified? Additionally, what if we apply reverse engineering to the Wasm binary and use the (likely non-equivalent) results to guide test case generation?
- How can the generated tests be used to detect bugs? The current 50% correct rate of the oracles generated by LLMs can definitely not be applied.
</content>
</thread_object>

</thread>


Metareview:
metareview:
> This paper introduces WasmTest, an approach for testing of WebAssembly (Wasm) binaries. To overcome the challenges of testing Wasm binaries without source code, WasmTest leverages a generate-then-test methodology. It utilizes large language models (LLMs) to automatically generate equivalent C++ code, create and mutate test cases, and compile these tests for execution against the Wasm binary. Results demonstrate that WasmTest generates comprehensive test suites.

The reviewers agree on experimental and applicability weaknesses. Even though the paper is interesting, its current scope is too limited for ICLR.