Kati8/ICLR_2023_2025 huggingface dataset:
authors: Weike Fang
authors: Kaicheng Wang
authors: Weihang Wang
reviews: 
Comments: The paper introduces a testing methodology for WebAssembly (Wasm) binaries that uses Large Language Models (LLMs) to overcome the absence of source code. By generating equivalent C++ code from a high-level understanding of Wasm functionalities, the methodology facilitates the creation of seed test cases via LLMs and type-aware mutation to improve coverage and detect bugs. The approach is validated through metrics like compile rate, code coverage, and bug detection rate. Two benchmarks are used in the evaluation. The paper showcases a practical application of LLMs in addressing challenges associated with Wasm binary testing.
Strengths: - Originality

Wasm binary testing is an interesting and important problem. The primary innovation of the "Generate-Then-Test" methodology is its application of LLMs to generate equivalent C++ code based on abstract descriptions of Wasm functionalities. 

- Quality

The proposed approach appears to work, validated using two benchmark datasets and evaluated through several standard metrics
- Clarity

The paper is overall clear and well-organised. 

- Significance:

This paper offers a new way in the testing binaries distributed without source code and this might even go beyond its immediate application in testing Wasm binaries.
Weaknesses: Using LLMs to generate equivalent source code from binaries constitutes the major contribution of this paper, yet it also the bottleneck of the work. It is an interesting idea but it is also a challenging problem, isn't it? This method presents difficulties in effectively translating high-level descriptions into executable test cases, and the paper does not convincingly demonstrate the efficacy of this translation.

The experimental validation is somewhat limited due to the nature of the benchmarks used. The study employs only two source code datasets that are compiled to WebAssembly before applying the proposed methodology. This approach can create an artificial scenario tailored to the solution presented in the paper, rather than addressing the more complex and varied real-world challenges found in typical WebAssembly applications.

While the methodology is shown to be effective within the constraints of the selected benchmarks, the paper does not provide a comparison with existing binary testing approaches. This raises concerns about its advantages or improvements over traditional testing techniques.

The success of the "Generate-Then-Test" methodology relies heavily on the quality and accuracy of the high-level descriptions of Wasm functionalities. Inadequate or imprecise descriptions can lead to derived C++ code and test cases that do not accurately reflect the intended functionality of the Wasm binaries. The paper should clarify how the quality of these descriptions impacts the effectiveness of the generated tests and propose strategies to ensure the reliability of these descriptions in producing meaningful test cases.
Recommendation: 3
Reviewer confidence: 3
Soundness/correctness: 2
Clarity: 2
reviews: 
Comments: This paper introduces WasmTest, an approach for testing of WebAssembly (Wasm) binaries. To overcome the challenges of testing Wasm binaries without source code, WasmTest leverages a generate-then-test methodology. It utilizes large language models (LLMs) to automatically generate equivalent C++ code, create and mutate test cases, and compile these tests for execution against the Wasm binary. Results demonstrate that WasmTest  generates comprehensive test suites.
Strengths: 1. Novel Approach: Introduces a novel generate-then-test approach for black-box testing of WebAssembly (Wasm) binaries, leveraging the power of large language models (LLMs).
2. Effective Coverage: Demonstrates high test coverage for simple, single-program benchmarks.
3. Ease of Understanding: The proposed technique is straightforward and easy to comprehend.
Weaknesses: - **Limited Literature Review:** The paper lacks a comprehensive review of related work, particularly in the areas of LLM-based automated test generation (e.g., CODAMOSA, ChatUniTest, MuTAP) and LLM-based fuzzing (e.g., TitanFuzz, FuzzGPT, Fuzz4All). Additionally, the review of existing Wasm testing techniques is insufficient.
- **Inadequate Test Oracle:** The reliance on LLM-generated oracles, which exhibit approximately 50% accuracy, limits the practical applicability of the approach. The black-box testing paradigm may not be suitable for developer-focused scenarios, and general fuzzing techniques with sanitizers might be more effective for attacker-focused use cases.
- **Insufficient Evaluation of Generate-Then-Test:** The paper lacks a thorough evaluation of the generate-then-test approach. It's unclear whether directly generating tests for Wasm binary code using LLMs or applying simple fuzzing techniques like AFL or libfuzzer would yield comparable or superior coverage results. Additionally, it lacks an ablation study exploring the substitution of the first component with a simpler reverse engineering approach.
- **Lack of Novelty in Mutation:** The mutation techniques employed in the paper do not appear to be significantly novel.
- **Limited Evaluation Scope:** The evaluation is restricted to a single-program dataset, which may not accurately reflect real-world application scenarios
Recommendation: 3
Reviewer confidence: 4
Soundness/correctness: 1
Clarity: 2