Repository: google/oss-fuzz-gen
Pull request #811: Reuse/Replicate SemanticAnalyzer and Enhancer to Agent workflow
Author: login: DonggeLiu
Created: 2025-02-21T00:30:21Z | Merged: 2025-03-03T12:44:22Z

1. [x] Reuse SemanticAnalyzer from the one-prompt workflow
2. [x] Replicate Enhancer from OnePromptEnhancer
3. [x] Last result can still be `BuildResult` if build failure.
4. [x] When Execution Fails, still record `BuildResult` without `RunResult`.
5. [x] Download [`xs` benchmarks](https://llm-exp.oss-fuzz.com/Result-reports/ofg-pr/2025-02-23-811-dg2-comparison/benchmark/output-xs-fxawaitimport/index.html) to understand why its successful `BuildResult` was not recorded on final experiment report after its run failure.
6. [x] Clean up commented code.
7. [x] Do not print `chat_histroy` when logging `Results`, which makes it unreadable.
8. [x] Stringify `WorDir` and `Benchmark` in `Result`s.
9. [x] Maybe `write_result()` at the end of every stage, so that the report is always up-to-date.

Next:
1. [ ] [Fix log type](#812) for easier debugging.
2. [ ] Understand and fix prompt [truncation logic error](https://llm-exp.oss-fuzz.com/Result-reports/ofg-pr/2025-02-23-811-dg2-comparison/sample/output-xs-fxawaitimport/04.html).
3. [ ] Re-run cloud build if a flaky step failed (e.g., `apt install`).


Diff:
diff --git a/agent/enhancer.py b/agent/enhancer.py
new file mode 100644
index 0000000000..9f37fc5820
--- /dev/null
+++ b/agent/enhancer.py
@@ -0,0 +1,60 @@
+# Copyright 2025 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""An LLM agent to improve a fuzz target's runtime performance.
+Use it as a usual module locally, or as script in cloud builds.
+"""
+import logger
+from agent.prototyper import Prototyper
+#from experiment.workdir import WorkDirs
+from llm_toolkit.prompt_builder import (DefaultTemplateBuilder,
+                                        JvmErrorFixingBuilder)
+from llm_toolkit.prompts import Prompt
+from results import AnalysisResult, Result
+
+
+class Enhancer(Prototyper):
+  """The Agent to refine a compilable fuzz target for higher coverage."""
+
+  def _initial_prompt(self, results: list[Result]) -> Prompt:
+    """Constructs initial prompt of the agent."""
+    last_result = results[-1]
+    benchmark = last_result.benchmark
+
+    if not isinstance(last_result, AnalysisResult):
+      logger.error('The last result in Enhancer is not AnalysisResult: %s',
+                   results,
+                   trial=self.trial)
+      return Prompt()
+
+    if benchmark.language == 'jvm':
+      # TODO: Do this in a separate agent for JVM coverage.
+      jvm_coverage_fix = True
+      error_desc, errors = '', []
+      builder = JvmErrorFixingBuilder(self.llm, benchmark,
+                                      last_result.run_result.fuzz_target_source,
+                                      errors, jvm_coverage_fix)
+      prompt = builder.build([], None, None)
+    else:
+      error_desc, errors = last_result.semantic_result.get_error_info()
+      builder = DefaultTemplateBuilder(self.llm)
+      prompt = builder.build_fixer_prompt(benchmark,
+                                          last_result.fuzz_target_source,
+                                          error_desc,
+                                          errors,
+                                          context='',
+                                          instruction='')
+      # TODO: A different file name/dir.
+      prompt.save(self.args.work_dirs.prompt)
+
+    return prompt
diff --git a/agent/prototyper.py b/agent/prototyper.py
index b683a60d8a..214ff338b8 100644
--- a/agent/prototyper.py
+++ b/agent/prototyper.py
@@ -26,6 +26,7 @@
 from data_prep import project_targets
 from data_prep.project_context.context_introspector import ContextRetriever
 from experiment.benchmark import Benchmark
+from experiment.workdir import WorkDirs
 from llm_toolkit import prompt_builder
 from llm_toolkit.prompts import Prompt
 from results import BuildResult, Result
@@ -393,8 +394,11 @@ def _container_handle_conclusion(self, cur_round: int, response: str,
         self._validate_fuzz_target_and_build_script(cur_round, build_result))
 
     # Updates build_result with _alt or _ori, depending on their status.
-    build_result, prompt_final = self._generate_prompt_from_build_result(
+    final_build_result, prompt_final = self._generate_prompt_from_build_result(
         build_result_alt, build_result_ori, build_result, prompt, cur_round)
+    # Ensure build_result is consistent with the one selected.
+    if final_build_result is not None:
+      build_result.__dict__.update(final_build_result.__dict__)
 
     return prompt_final
 
@@ -421,8 +425,9 @@ def _container_tool_reaction(self, cur_round: int, response: str,
 
   def execute(self, result_history: list[Result]) -> BuildResult:
     """Executes the agent based on previous result."""
+    WorkDirs(self.args.work_dirs.base)
     last_result = result_history[-1]
-    logger.info('Executing Prototyper', trial=last_result.trial)
+    logger.info('Executing %s', self.name, trial=last_result.trial)
     benchmark = last_result.benchmark
     self.inspect_tool = ProjectContainerTool(benchmark, name='inspect')
     self.inspect_tool.compile(extra_commands=' && rm -rf /out/* > /dev/null')
diff --git a/agent/semantic_analyzer.py b/agent/semantic_analyzer.py
index 381b4f4364..d41932295a 100644
--- a/agent/semantic_analyzer.py
+++ b/agent/semantic_analyzer.py
@@ -68,7 +68,7 @@ def execute(self, result_history: list[Result]) -> AnalysisResult:
           fuzzer_log, last_result.benchmark.project)
 
     analysis_result = AnalysisResult(
-        author=repr(self),
+        author=self,
         run_result=last_result,
         semantic_result=semantic_result,
         chat_history={self.name: semantic_result.to_dict()})
diff --git a/experiment/benchmark.py b/experiment/benchmark.py
index d100b37bdb..52e80a9fa7 100644
--- a/experiment/benchmark.py
+++ b/experiment/benchmark.py
@@ -211,7 +211,7 @@ def __init__(self,
       # Special handling of benchmark_id is needed to avoid this situation.
       self.id = self.id.replace('::', '-')
 
-  def __str__(self):
+  def __repr__(self):
     return (f'Benchmark<id={self.id}, project={self.project}, '
             f'language={self.language}, '
             f'function_signature={self.function_signature}, '
diff --git a/experiment/workdir.py b/experiment/workdir.py
index 72495771e6..d6d83951fe 100644
--- a/experiment/workdir.py
+++ b/experiment/workdir.py
@@ -42,6 +42,9 @@ def __init__(self, base_dir, keep: bool = False):
     os.makedirs(self.dills, exist_ok=True)
     os.makedirs(self.fuzz_targets, exist_ok=True)
 
+  def __repr__(self) -> str:
+    return self._base_dir
+
   @property
   def base(self):
     return self._base_dir
diff --git a/logger.py b/logger.py
index c39caebaaf..69399c2181 100644
--- a/logger.py
+++ b/logger.py
@@ -23,7 +23,7 @@
 
 from google.cloud import storage
 
-from results import Result, RunResult
+from results import Result, RunResult, TrialResult
 
 FINAL_RESULT_JSON = 'result.json'
 
@@ -54,7 +54,7 @@ def write_build_script(self, result: Result) -> None:
                                      f'{result.trial:02d}.build_script')
     self.write_to_file(build_script_path, result.build_script_source)
 
-  def write_result(self, result_status_dir: str, result: Result) -> None:
+  def write_result(self, result_status_dir: str, result: TrialResult) -> None:
     """Writes the final result into JSON for report generation."""
     trial_result_dir = os.path.join(result_status_dir, f'{result.trial:02d}')
     os.makedirs(trial_result_dir, exist_ok=True)
diff --git a/pipeline.py b/pipeline.py
index e2baecccae..84b62375b4 100644
--- a/pipeline.py
+++ b/pipeline.py
@@ -17,7 +17,7 @@
 
 import logger
 from agent.base_agent import BaseAgent
-from results import AnalysisResult, BuildResult, Result, RunResult
+from results import AnalysisResult, BuildResult, Result, RunResult, TrialResult
 from stage.analysis_stage import AnalysisStage
 from stage.execution_stage import ExecutionStage
 from stage.writing_stage import WritingStage
@@ -57,50 +57,69 @@ def _terminate(self, result_history: list[Result], cycle_count: int) -> bool:
     if not cycle_count:
       return False
 
-    last_result = result_history[-1]
     if cycle_count > 5:
-      self.logger.warning('[Cycle %d] Terminate after 5 cycles: %s',
-                          cycle_count, result_history)
+      self.logger.info('[Cycle %d] Terminate after 5 cycles: %s', cycle_count,
+                       result_history)
       return True
 
-    if not isinstance(last_result, AnalysisResult):
-      self.logger.warning('[Cycle %d] Last result is not AnalysisResult: %s',
-                          cycle_count, result_history)
+    last_result = result_history[-1]
+    if isinstance(last_result, BuildResult) and not last_result.success:
+      self.logger.debug('[Cycle %d] Last result is failed BuildResult: %s',
+                        cycle_count, last_result)
       return True
 
-    if last_result.success:
+    if isinstance(last_result, AnalysisResult) and last_result.success:
       self.logger.info('[Cycle %d] Generation succeeds: %s', cycle_count,
                        result_history)
       return True
 
-    self.logger.info('[Cycle %d] Generation continues: %s', cycle_count,
-                     result_history)
-    return False
+    if isinstance(last_result, AnalysisResult) and not last_result.success:
+      self.logger.info('[Cycle %d] Generation continues: %s', cycle_count,
+                       result_history)
+      return False
+
+    self.logger.warning('[Cycle %d] Last result is unexpected: %s', cycle_count,
+                        last_result)
+    return True
+
+  def _update_status(self, result_history: list[Result]) -> None:
+    trial_result = TrialResult(benchmark=result_history[-1].benchmark,
+                               trial=self.trial,
+                               work_dirs=result_history[-1].work_dirs,
+                               result_history=result_history)
+    self.logger.write_result(
+        result_status_dir=trial_result.best_result.work_dirs.status,
+        result=trial_result)
 
   def _execute_one_cycle(self, result_history: list[Result],
                          cycle_count: int) -> None:
     """Executes the stages once."""
     self.logger.info('[Cycle %d] Initial result is %s', cycle_count,
                      result_history[-1])
+    # Writing stage.
     result_history.append(
         self.writing_stage.execute(result_history=result_history))
+    self._update_status(result_history=result_history)
     if (not isinstance(result_history[-1], BuildResult) or
         not result_history[-1].success):
       self.logger.warning('[Cycle %d] Build failure, skipping the rest steps',
                           cycle_count)
       return
 
+    # Execution stage.
     result_history.append(
         self.execution_stage.execute(result_history=result_history))
+    self._update_status(result_history=result_history)
     if (not isinstance(result_history[-1], RunResult) or
         not result_history[-1].log_path):
       self.logger.warning('[Cycle %d] Run failure, skipping the rest steps',
                           cycle_count)
       return
 
+    # Analysis stage.
     result_history.append(
         self.analysis_stage.execute(result_history=result_history))
-
+    self._update_status(result_history=result_history)
     self.logger.info('[Cycle %d] Analysis result %s: %s', cycle_count,
                      result_history[-1].success, result_history[-1])
 
@@ -121,8 +140,4 @@ def execute(self, result_history: list[Result]) -> list[Result]:
       cycle_count += 1
       self._execute_one_cycle(result_history=result_history,
                               cycle_count=cycle_count)
-
-    final_result = result_history[-1]
-    self.logger.write_result(result_status_dir=final_result.work_dirs.status,
-                             result=final_result)
     return result_history
diff --git a/results.py b/results.py
index ac777f2db1..b301b94ed5 100644
--- a/results.py
+++ b/results.py
@@ -30,6 +30,7 @@ class Result:
   build_script_source: str
   author: Any
   chat_history: dict
+  _repr_exclude = {'_repr_exclude', 'chat_history'}
 
   def __init__(self,
                benchmark: Benchmark,
@@ -48,8 +49,11 @@ def __init__(self,
     self.chat_history = chat_history or {}
 
   def __repr__(self) -> str:
-    return (f'{self.__class__.__name__}'
-            f'({", ".join(f"{k}={v!r}" for k, v in vars(self).items())})')
+    attributes = [
+        f'{k}={v!r}' for k, v in vars(self).items()
+        if k not in self._repr_exclude
+    ]
+    return f'{self.__class__.__name__}({", ".join(attributes)})'
 
   def to_dict(self) -> dict:
     return {
@@ -60,7 +64,7 @@ def to_dict(self) -> dict:
         'trial': self.trial,
         'fuzz_target_source': self.fuzz_target_source,
         'build_script_source': self.build_script_source,
-        'author': str(self.author),
+        'author': self.author.name,
         'chat_history': self.chat_history,
     }
 
@@ -74,6 +78,7 @@ class BuildResult(Result):
   compile_log: str  # Build full output.
   binary_exists: bool  # Fuzz target binary generated successfully.
   is_function_referenced: bool  # Fuzz target references function-under-test.
+  _repr_exclude = Result._repr_exclude | {'compile_log', 'compile_error'}
 
   def __init__(self,
                benchmark: Benchmark,
@@ -127,6 +132,7 @@ class RunResult(BuildResult):
   coverage_report_path: str
   cov_pcs: int
   total_pcs: int
+  _repr_exclude = BuildResult._repr_exclude | {'textcov_diff'}
 
   def __init__(
       self,
@@ -230,7 +236,7 @@ class AnalysisResult(Result):
   coverage_result: Optional[CoverageResult]
 
   def __init__(self,
-               author: str,
+               author: Any,
                run_result: RunResult,
                semantic_result: SemanticCheckResult,
                crash_result: Optional[CrashResult] = None,
@@ -273,6 +279,85 @@ def __init__(self,
     self.work_dirs = work_dirs
     self.result_history = result_history or []
 
+  @property
+  def function_signature(self) -> str:
+    """Function signature of the benchmark."""
+    return self.benchmark.function_signature
+
+  @property
+  def project(self) -> str:
+    """Project name of the benchmark."""
+    return self.benchmark.project
+
+  @property
+  def project_commit(self) -> str:
+    """Project commit of the benchmark."""
+    return self.benchmark.commit or ''
+
+  @property
+  def project_language(self) -> str:
+    """Project language of the benchmark."""
+    return self.benchmark.language
+
+  @property
+  def best_result(self) -> Result:
+    """Best result in trial."""
+    # Preference order:
+    #   1. Highest coverage diff (RunResult)
+    #   2. Highest coverage (RunResult)
+    #   3. Last Build success (BuildResult)
+    #   4. Last Result
+    best_result = None
+
+    max_cov_diff = 0
+    for result in self.result_history:
+      if (isinstance(result, RunResult) and
+          result.line_coverage_diff > max_cov_diff):
+        max_cov_diff = result.line_coverage_diff
+        best_result = result
+    if best_result:
+      return best_result
+
+    max_cov = 0
+    for result in self.result_history:
+      if (isinstance(result, RunResult) and result.coverage > max_cov):
+        max_cov = result.coverage
+        best_result = result
+    if best_result:
+      return best_result
+
+    for result in self.result_history[::-1]:
+      if isinstance(result, BuildResult) and result.success:
+        return result
+
+    return self.result_history[-1]
+
+  @property
+  def fuzz_target_source(self) -> str:
+    """The best fuzz target source code."""
+    result = self.best_result
+    if isinstance(result, AnalysisResult):
+      return result.run_result.fuzz_target_source
+    return self.best_result.fuzz_target_source
+
+  @property
+  def build_script_source(self) -> str:
+    """The best build script source code."""
+    result = self.best_result
+    if isinstance(result, AnalysisResult):
+      return result.run_result.build_script_source
+    return self.best_result.build_script_source
+
+  @property
+  def author(self) -> Any:
+    """The author of the best result."""
+    return self.best_result.author
+
+  @property
+  def chat_history(self) -> dict:
+    """The chat history of the best result."""
+    return self.best_result.chat_history
+
   @property
   def build_success(self) -> bool:
     """True if there is any build success."""
@@ -303,6 +388,22 @@ def line_coverage_diff(self) -> float:
                 if isinstance(result, RunResult)),
                default=0)
 
+  @property
+  def cov_pcs(self) -> int:
+    """Log path of the best result if it is RunResult."""
+    return max((result.cov_pcs
+                for result in self.result_history
+                if isinstance(result, RunResult)),
+               default=0)
+
+  @property
+  def total_pcs(self) -> int:
+    """Log path of the best result if it is RunResult."""
+    return max((result.total_pcs
+                for result in self.result_history
+                if isinstance(result, RunResult)),
+               default=0)
+
   @property
   def line_coverage_report(self) -> str:
     """Max line coverage diff report."""
@@ -322,6 +423,80 @@ def textcov_diff(self) -> textcov.Textcov:
         all_textcov.merge(result.textcov_diff)
     return all_textcov
 
+  @property
+  def run_error(self) -> str:
+    """Run error of the best result if it is RunResult."""
+    result = self.best_result
+    if isinstance(result, RunResult):
+      return result.run_error
+    if isinstance(result, AnalysisResult):
+      return result.run_result.run_error
+    return ''
+
+  @property
+  def run_log(self) -> str:
+    """Run log of the best result if it is RunResult."""
+    result = self.best_result
+    if isinstance(result, RunResult):
+      return result.run_log
+    if isinstance(result, AnalysisResult):
+      return result.run_result.run_log
+    return ''
+
+  @property
+  def log_path(self) -> str:
+    """Log path of the best result if it is RunResult."""
+    result = self.best_result
+    if isinstance(result, RunResult):
+      return result.log_path
+    if isinstance(result, AnalysisResult):
+      return result.run_result.log_path
+    return ''
+
+  def to_dict(self) -> dict:
+    return {
+        'trial':
+            self.trial,
+        'function_signature':
+            self.function_signature,
+        'project':
+            self.project,
+        'project_commit':
+            self.project_commit,
+        'project_language':
+            self.project_language,
+        'fuzz_target_source':
+            self.fuzz_target_source,
+        'build_script_source':
+            self.build_script_source,
+        'author':
+            self.author.name,
+        'chat_history':
+            self.chat_history,
+        'compiles':
+            self.build_success,
+        'crash':
+            self.crash,
+        'coverage':
+            self.coverage,
+        'line_coverage_diff':
+            self.line_coverage_diff,
+        'cov_pcs':
+            self.cov_pcs,
+        'total_pcs':
+            self.total_pcs,
+        'line_coverage_report':
+            self.line_coverage_report,
+        'textcov_diff':
+            dataclasses.asdict(self.textcov_diff) if self.textcov_diff else '',
+        'run_error':
+            self.run_error,
+        'run_log':
+            self.run_log,
+        'log_path':
+            self.log_path,
+    }
+
 
 class BenchmarkResult:
   """All trial results for a benchmark in an experiment."""
diff --git a/run_all_experiments.py b/run_all_experiments.py
index 84bb20f40e..2fc7a79b21 100755
--- a/run_all_experiments.py
+++ b/run_all_experiments.py
@@ -377,7 +377,7 @@ def _setup_logging(verbose: str = 'info', is_cloud: bool = False) -> None:
       client.setup_logging()
     except Exception as e:
       # For local runs we continue
-      pass
+      logger.warning('Error setting up cloud logging client: %s', e)
 
   if verbose == "debug":
     log_level = logging.DEBUG
diff --git a/run_one_experiment.py b/run_one_experiment.py
index 4caa0ad43f..aea965e651 100644
--- a/run_one_experiment.py
+++ b/run_one_experiment.py
@@ -24,6 +24,7 @@
 
 import logger
 import pipeline
+from agent.enhancer import Enhancer
 from agent.one_prompt_enhancer import OnePromptEnhancer
 from agent.one_prompt_prototyper import OnePromptPrototyper
 from agent.prototyper import Prototyper
@@ -241,10 +242,17 @@ def _fuzzing_pipeline(benchmark: Benchmark, model: models.LLM,
   trial_logger = logger.get_trial_logger(trial=trial, level=logging.DEBUG)
   trial_logger.info('Trial Starts')
   if args.agent:
-    p = pipeline.Pipeline(
-        args=args,
-        trial=trial,
-        writing_stage_agents=[Prototyper(trial=trial, llm=model, args=args)])
+    p = pipeline.Pipeline(args=args,
+                          trial=trial,
+                          writing_stage_agents=[
+                              Prototyper(trial=trial, llm=model, args=args),
+                              Enhancer(trial=trial, llm=model, args=args),
+                          ],
+                          analysis_stage_agents=[
+                              SemanticAnalyzer(trial=trial,
+                                               llm=model,
+                                               args=args),
+                          ])
   else:
     p = pipeline.Pipeline(args=args,
                           trial=trial,
@@ -259,17 +267,21 @@ def _fuzzing_pipeline(benchmark: Benchmark, model: models.LLM,
                           analysis_stage_agents=[
                               SemanticAnalyzer(trial=trial,
                                                llm=model,
-                                               args=args)
+                                               args=args),
                           ])
 
   results = p.execute(result_history=[
       Result(benchmark=benchmark, trial=trial, work_dirs=work_dirs)
   ])
 
-  return TrialResult(benchmark=benchmark,
-                     trial=trial,
-                     work_dirs=work_dirs,
-                     result_history=results)
+  trial_result = TrialResult(benchmark=benchmark,
+                             trial=trial,
+                             work_dirs=work_dirs,
+                             result_history=results)
+  trial_logger.write_result(
+      result_status_dir=trial_result.best_result.work_dirs.status,
+      result=trial_result)
+  return trial_result
 
 
 def _fuzzing_pipelines(benchmark: Benchmark, model: models.LLM,
diff --git a/stage/base_stage.py b/stage/base_stage.py
index a6e5f78657..193f886872 100644
--- a/stage/base_stage.py
+++ b/stage/base_stage.py
@@ -28,11 +28,13 @@ class BaseStage(ABC):
   def __init__(self,
                args: argparse.Namespace,
                trail: int,
-               agents: Optional[list[BaseAgent]] = None) -> None:
+               agents: Optional[list[BaseAgent]] = None,
+               name: str = '') -> None:
     self.args = args
     self.trial = trail
     self.agents: list[BaseAgent] = agents or []
     self.logger = logger.get_trial_logger(trial=trail)
+    self.name: str = name or self.__class__.__name__
 
   def __repr__(self) -> str:
     return self.__class__.__name__
diff --git a/stage/execution_stage.py b/stage/execution_stage.py
index 11e0e65391..98013881d8 100644
--- a/stage/execution_stage.py
+++ b/stage/execution_stage.py
@@ -129,7 +129,7 @@ def execute(self, result_history: list[Result]) -> Result:
           fuzz_target_source=last_result.fuzz_target_source,
           build_script_source=last_result.build_script_source,
           chat_history=last_result.chat_history,
-          author=repr(self),
+          author=self,
           compiles=last_result.compiles,
           compile_error=last_result.compile_error,
           compile_log=last_result.compile_log,
@@ -158,7 +158,7 @@ def execute(self, result_history: list[Result]) -> Result:
           fuzz_target_source=last_result.fuzz_target_source,
           build_script_source=last_result.build_script_source,
           chat_history=last_result.chat_history,
-          author=repr(self),
+          author=self,
           compiles=last_result.compiles,
           compile_error=last_result.compile_error,
           compile_log=last_result.compile_log,


Reviews:
user: login: mihaimaruseac
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-02-24T14:21:21Z
user: login: oliverchang
state: APPROVED
author_association: COLLABORATOR
submitted_at: 2025-02-25T08:26:17Z
user: login: DonggeLiu
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-01T21:16:31Z
user: login: DonggeLiu
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-01T21:17:12Z
user: login: DonggeLiu
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-01T21:17:18Z

Review comments:
diff_hunk: @@ -0,0 +1,65 @@
+# Copyright 2025 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""An LLM agent to improve a fuzz target's runtime performance.
+Use it as a usual module locally, or as script in cloud builds.
+"""
+import logger
+from agent.prototyper import Prototyper
+#from experiment.workdir import WorkDirs
+from llm_toolkit.prompt_builder import (DefaultTemplateBuilder,
+                                        JvmErrorFixingBuilder)
+from llm_toolkit.prompts import Prompt
+from results import AnalysisResult, Result
+
+# from results import AnalysisResult, BuildResult, Result
+# from tool.container_tool import ProjectContainerTool
+
+MAX_ROUND = 10
+
+
+class Enhancer(Prototyper):
+  """The Agent to generate a simple but valid fuzz target from scratch."""
path: agent/enhancer.py
user: login: oliverchang
body: nit: fix docstring. 
created_at: 2025-02-25T08:23:26Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,65 @@
+# Copyright 2025 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""An LLM agent to improve a fuzz target's runtime performance.
+Use it as a usual module locally, or as script in cloud builds.
+"""
+import logger
+from agent.prototyper import Prototyper
+#from experiment.workdir import WorkDirs
+from llm_toolkit.prompt_builder import (DefaultTemplateBuilder,
+                                        JvmErrorFixingBuilder)
+from llm_toolkit.prompts import Prompt
+from results import AnalysisResult, Result
+
+# from results import AnalysisResult, BuildResult, Result
+# from tool.container_tool import ProjectContainerTool
+
+MAX_ROUND = 10
path: agent/enhancer.py
user: login: oliverchang
body: where is this used? 
created_at: 2025-02-25T08:23:39Z
author_association: COLLABORATOR
diff_hunk: @@ -57,50 +57,69 @@ def _terminate(self, result_history: list[Result], cycle_count: int) -> bool:
     if not cycle_count:
       return False
 
-    last_result = result_history[-1]
     if cycle_count > 5:
-      self.logger.warning('[Cycle %d] Terminate after 5 cycles: %s',
-                          cycle_count, result_history)
+      self.logger.info('[Cycle %d] Terminate after 5 cycles: %s', cycle_count,
+                       result_history)
       return True
 
-    if not isinstance(last_result, AnalysisResult):
-      self.logger.warning('[Cycle %d] Last result is not AnalysisResult: %s',
-                          cycle_count, result_history)
+    last_result = result_history[-1]
+    if isinstance(last_result, BuildResult) and not last_result.success:
+      self.logger.debug('[Cycle %d] Last result is failed BuildResult: %s',
+                        cycle_count, last_result)
       return True
 
-    if last_result.success:
+    if isinstance(last_result, AnalysisResult) and last_result.success:
       self.logger.info('[Cycle %d] Generation succeeds: %s', cycle_count,
                        result_history)
       return True
 
-    self.logger.info('[Cycle %d] Generation continues: %s', cycle_count,
-                     result_history)
-    return False
+    if isinstance(last_result, AnalysisResult) and not last_result.success:
+      self.logger.info('[Cycle %d] Generation continues: %s', cycle_count,
+                       result_history)
+      return False
+
+    self.logger.warning('[Cycle %d] Last result is unexpeceted: %s',
path: pipeline.py
user: login: oliverchang
body: nit: fix spelling of unexpected. 
created_at: 2025-02-25T08:24:32Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,65 @@
+# Copyright 2025 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""An LLM agent to improve a fuzz target's runtime performance.
+Use it as a usual module locally, or as script in cloud builds.
+"""
+import logger
+from agent.prototyper import Prototyper
+#from experiment.workdir import WorkDirs
+from llm_toolkit.prompt_builder import (DefaultTemplateBuilder,
+                                        JvmErrorFixingBuilder)
+from llm_toolkit.prompts import Prompt
+from results import AnalysisResult, Result
+
+# from results import AnalysisResult, BuildResult, Result
+# from tool.container_tool import ProjectContainerTool
+
+MAX_ROUND = 10
+
+
+class Enhancer(Prototyper):
+  """The Agent to generate a simple but valid fuzz target from scratch."""
path: agent/enhancer.py
user: login: DonggeLiu
body: Done, thanks!
created_at: 2025-03-01T21:16:31Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,65 @@
+# Copyright 2025 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""An LLM agent to improve a fuzz target's runtime performance.
+Use it as a usual module locally, or as script in cloud builds.
+"""
+import logger
+from agent.prototyper import Prototyper
+#from experiment.workdir import WorkDirs
+from llm_toolkit.prompt_builder import (DefaultTemplateBuilder,
+                                        JvmErrorFixingBuilder)
+from llm_toolkit.prompts import Prompt
+from results import AnalysisResult, Result
+
+# from results import AnalysisResult, BuildResult, Result
+# from tool.container_tool import ProjectContainerTool
+
+MAX_ROUND = 10
path: agent/enhancer.py
user: login: DonggeLiu
body: Remove.
It was used by its parent class to control the #rounds for LLM interaction.
created_at: 2025-03-01T21:17:12Z
author_association: COLLABORATOR
diff_hunk: @@ -57,50 +57,69 @@ def _terminate(self, result_history: list[Result], cycle_count: int) -> bool:
     if not cycle_count:
       return False
 
-    last_result = result_history[-1]
     if cycle_count > 5:
-      self.logger.warning('[Cycle %d] Terminate after 5 cycles: %s',
-                          cycle_count, result_history)
+      self.logger.info('[Cycle %d] Terminate after 5 cycles: %s', cycle_count,
+                       result_history)
       return True
 
-    if not isinstance(last_result, AnalysisResult):
-      self.logger.warning('[Cycle %d] Last result is not AnalysisResult: %s',
-                          cycle_count, result_history)
+    last_result = result_history[-1]
+    if isinstance(last_result, BuildResult) and not last_result.success:
+      self.logger.debug('[Cycle %d] Last result is failed BuildResult: %s',
+                        cycle_count, last_result)
       return True
 
-    if last_result.success:
+    if isinstance(last_result, AnalysisResult) and last_result.success:
       self.logger.info('[Cycle %d] Generation succeeds: %s', cycle_count,
                        result_history)
       return True
 
-    self.logger.info('[Cycle %d] Generation continues: %s', cycle_count,
-                     result_history)
-    return False
+    if isinstance(last_result, AnalysisResult) and not last_result.success:
+      self.logger.info('[Cycle %d] Generation continues: %s', cycle_count,
+                       result_history)
+      return False
+
+    self.logger.warning('[Cycle %d] Last result is unexpeceted: %s',
path: pipeline.py
user: login: DonggeLiu
body: Done, thanks
created_at: 2025-03-01T21:17:18Z
author_association: COLLABORATOR

PR comments:
user: login: DonggeLiu
created_at: 2025-02-21T00:30:43Z
body: /gcbrun exp -n dg -ag
author_association: COLLABORATOR
user: login: DonggeLiu
created_at: 2025-02-21T03:14:28Z
body: /gcbrun exp -n dg -ag
author_association: COLLABORATOR
user: login: DonggeLiu
created_at: 2025-02-22T22:36:58Z
body: /gcbrun exp -n dg -ag
author_association: COLLABORATOR
user: login: DonggeLiu
created_at: 2025-02-23T02:31:45Z
body: /gcbrun exp -n dg1 -ag


author_association: COLLABORATOR
user: login: DonggeLiu
created_at: 2025-02-23T08:19:12Z
body: /gcbrun exp -n dg2 -ag
author_association: COLLABORATOR
user: login: DonggeLiu
created_at: 2025-02-24T11:29:49Z
body: /gcbrun exp -n dg -ag
author_association: COLLABORATOR
user: login: DonggeLiu
created_at: 2025-02-24T21:26:34Z
body: /gcbrun exp -n dg -ag
author_association: COLLABORATOR
user: login: DonggeLiu
created_at: 2025-02-25T02:18:19Z
body: /gcbrun exp -n dg1 -ag
author_association: COLLABORATOR
user: login: DonggeLiu
created_at: 2025-02-25T03:48:03Z
body: /gcbrun exp -n dg2 -ag
author_association: COLLABORATOR
user: login: DonggeLiu
created_at: 2025-03-01T21:20:52Z
body: /gcbrun exp -n dg -ag
author_association: COLLABORATOR
user: login: DonggeLiu
created_at: 2025-03-02T12:29:13Z
body: /gcbrun exp -n dg -ag
author_association: COLLABORATOR
user: login: DonggeLiu
created_at: 2025-03-02T13:13:54Z
body: /gcbrun exp -n dg -ag
author_association: COLLABORATOR
user: login: DonggeLiu
created_at: 2025-03-02T14:03:27Z
body: /gcbrun exp -n dg -ag
author_association: COLLABORATOR
user: login: DonggeLiu
created_at: 2025-03-03T01:05:31Z
body: /gcbrun exp -n dg1 -ag
author_association: COLLABORATOR
user: login: DonggeLiu
created_at: 2025-03-03T12:32:49Z
body: This build rate is crazy:
<img width="386" alt="image" src="https://github.com/user-attachments/assets/ac763ea4-5b91-41c8-8d42-fef0e2a5cc68" />

It's (one of) the highest I've ever seen, and this is the selected set of 'difficult' benchmarks.


For some reasons, I think there are still rooms for improvements. E.g., [some trials are missing](https://llm-exp.oss-fuzz.com/Result-reports/ofg-pr/2025-03-03-811-dg1-comparison/benchmark/output-bluez-g_obex_get_req/index.html).
~~I will figure that out tmr.~~

Yes there is definitely room for improvements: LLM prompt truncation failed for some cases.
I will fix it in a another PR.
author_association: COLLABORATOR