Repository: google/fuzzbench

Issue:
Only trigger experiments when a comment starts with `/gcbrun`
Currently, a `GCB` experiment will be triggered if our comment **_mentions_** `/gcbrun`.

This is problematic because experiments will be launched unintendedly (e.g., when we quote previous `/gcb` commands, or even when we mention `/gcb` in any form).

## Root cause
1. `GCB` trigger simply [checks for `/gcb` without any constraints](https://pantheon.corp.google.com/cloud-build/triggers/edit/fff67372-9e94-4eeb-ab8c-4566ca73badf?project=fuzzbench). 
2. Our `gcbrun_experiment.py` [iterates all previous comments](https://github.com/google/fuzzbench/blob/56caa83e81bc59a1389367c6bd29d46fd35d03e6/service/gcbrun_experiment.py#L57) to find the latest valid `/gcb` command, which might have already been run.

## Proofs
1. [This example](https://github.com/google/fuzzbench/pull/1893#issuecomment-1716770892) quotes an old command and modifies its experiment name. However, it triggers an experiment with the old name, indicating we launch the latest valid experiment command.
2. [This example](https://github.com/google/fuzzbench/pull/1893#issuecomment-1716780709) includes a random `/gcb` command in the middle of the sentence. It also triggers the same old experiment, confirming the same conclusion.

## Propose fix
Change https://github.com/google/fuzzbench/blob/56caa83e81bc59a1389367c6bd29d46fd35d03e6/service/gcbrun_experiment.py#L56-L57
to
```python
 if not body.startswith(RUN_EXPERIMENT_COMMAND_STR): 
     return None 
```

## Justification
1. Based on our setting on Google Cloud, all `/gcb` commands will immediately trigger an experiment, there is no need to search back in history for an old command.
2. If we need to run an old command, we can always add a new comment for it.

@jonathanmetzman: Did I miss anything?
If it looks good to you, I will implement the fix proposed above.
Only trigger experiments when a comment starts with `/gcbrun`
Currently, a `GCB` experiment will be triggered if our comment **_mentions_** `/gcbrun`.

This is problematic because experiments will be launched unintendedly (e.g., when we quote previous `/gcb` commands, or even when we mention `/gcb` in any form).

## Root cause
1. `GCB` trigger simply [checks for `/gcb` without any constraints](https://pantheon.corp.google.com/cloud-build/triggers/edit/fff67372-9e94-4eeb-ab8c-4566ca73badf?project=fuzzbench). 
2. Our `gcbrun_experiment.py` [iterates all previous comments](https://github.com/google/fuzzbench/blob/56caa83e81bc59a1389367c6bd29d46fd35d03e6/service/gcbrun_experiment.py#L57) to find the latest valid `/gcb` command, which might have already been run.

## Proofs
1. [This example](https://github.com/google/fuzzbench/pull/1893#issuecomment-1716770892) quotes an old command and modifies its experiment name. However, it triggers an experiment with the old name, indicating we launch the latest valid experiment command.
2. [This example](https://github.com/google/fuzzbench/pull/1893#issuecomment-1716780709) includes a random `/gcb` command in the middle of the sentence. It also triggers the same old experiment, confirming the same conclusion.

## Propose fix
Change https://github.com/google/fuzzbench/blob/56caa83e81bc59a1389367c6bd29d46fd35d03e6/service/gcbrun_experiment.py#L56-L57
to
```python
 if not body.startswith(RUN_EXPERIMENT_COMMAND_STR): 
     return None 
```

## Justification
1. Based on our setting on Google Cloud, all `/gcb` commands will immediately trigger an experiment, there is no need to search back in history for an old command.
2. If we need to run an old command, we can always add a new comment for it.

@jonathanmetzman: Did I miss anything?
If it looks good to you, I will implement the fix proposed above.


Discussion:
you mean `[/]gcbrun` run right?
I think this is a good idea.
you mean `[/]gcbrun` run right?
I think this is a good idea.

Resolving patch:
diff --git a/service/gcbrun_experiment.py b/service/gcbrun_experiment.py
--- a/service/gcbrun_experiment.py
+++ b/service/gcbrun_experiment.py
@@ -23,7 +23,6 @@
 
 # pytype: disable=import-error
 import github  # pylint: disable=import-error
-
 from experiment import run_experiment
 
 TRIGGER_COMMAND = '/gcbrun'
@@ -38,38 +37,41 @@ def get_comments(pull_request_number):
     repo = github_obj.get_repo('google/fuzzbench')
     pull = repo.get_pull(pull_request_number)
     pull_comments = list(pull.get_comments())
+    last_pull_comment = pull_comments[-1] if pull_comments else None
     issue = repo.get_issue(pull_request_number)
     issue_comments = list(issue.get_comments())
+    last_issue_comment = issue_comments[-1] if issue_comments else None
     # Github only returns comments if from the pull object when a pull request
     # is open. If it is a draft, it will only return comments from the issue
     # object.
-    return pull_comments + issue_comments
+    return last_pull_comment, last_issue_comment
 
 
-def get_latest_gcbrun_command(comments):
+def get_latest_gcbrun_command(comment):
     """Gets the last /gcbrun comment from comments."""
-    for comment in reversed(comments):
-        # This seems to get comments on code too.
-        body = comment.body
-        if body.startswith(SKIP_COMMAND_STR):
-            return None
-        if not body.startswith(RUN_EXPERIMENT_COMMAND_STR):
-            continue
-        if len(body) == len(RUN_EXPERIMENT_COMMAND_STR):
-            return None
-        command = body[len(RUN_EXPERIMENT_COMMAND_STR):].strip().split(' ')
-        # Items that only contain space are redundant and will confuse
-        # `run_experiment_main()` in `experiment/run_experiment.py`
-        return [word for word in command if word.strip()]
-    return None
+    # This seems to get comments on code too.
+    if comment is None:
+        return None
+    body = comment.body
+    if body.startswith(SKIP_COMMAND_STR):
+        return None
+    if not body.startswith(RUN_EXPERIMENT_COMMAND_STR):
+        return None
+    if len(body) == len(RUN_EXPERIMENT_COMMAND_STR):
+        return None
+    command = body[len(RUN_EXPERIMENT_COMMAND_STR):].strip().split(' ')
+    # Items that only contain space are redundant and will confuse
+    # `run_experiment_main()` in `experiment/run_experiment.py`
+    return [word for word in command if word.strip()]
 
 
 def exec_command_from_github(pull_request_number):
     """Executes the gcbrun command for run_experiment.py in the most recent
   command on |pull_request_number|."""
-    comments = get_comments(pull_request_number)
-    print(comments)
-    command = get_latest_gcbrun_command(comments)
+    pull_cmt, issue_cmt = get_comments(pull_request_number)
+    print(f'Pull comment: {pull_cmt}\nIssue comment: {issue_cmt}')
+    command = (get_latest_gcbrun_command(pull_cmt) or
+               get_latest_gcbrun_command(issue_cmt))
     if command is None:
         logging.info('Experiment not requested.')
         return None


Test patch: