internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: google/clusterfuzz

Issue:
Don't create empty testcases for timeouts/ooms
When REPORT_OOMS_AND_HANGS is false, we report unreproducible empty crashes, because we see that there is a `Test unit written to...` even if the crash is not something we consider.


Discussion:
Thanks for noticing this, I was about to send you this example: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=17853
I think that one is a different issue, since it's unreproducible and if you look at the crash stats it's been happening for at least 2 weeks (hence why we filed).
Oh, from the issue description I somehow concluded that we wouldn't file an issue like that previously, but did file it now. Even if it's different, I guess we still prefer to ignore it for an MSan build?

Resolving patch:
diff --git a/src/python/bot/tasks/fuzz_task.py b/src/python/bot/tasks/fuzz_task.py
--- a/src/python/bot/tasks/fuzz_task.py
+++ b/src/python/bot/tasks/fuzz_task.py
@@ -231,6 +231,9 @@ def get_error(self):
     if self.is_archived() and not self.fuzzed_key:
       return 'Unable to store testcase in blobstore: %s' % self.crash_state
 
+    if not self.crash_state or not self.crash_type:
+      return 'Empty crash state or type'
+
     return None
 
 
@@ -996,8 +999,8 @@ def filter_crashes(crashes):
 
   for crash in crashes:
     if not crash.is_valid():
-      logs.log('Ignore crash (reason=%s, state=%s).' % (crash.get_error(),
-                                                        crash.crash_state))
+      logs.log('Ignore crash (reason=%s, type=%s, state=%s).' %
+               (crash.get_error(), crash.crash_type, crash.crash_state))
       continue
 
     filtered.append(crash)


Test patch:
Repository: google/clusterfuzz

Issue:
get_all_project_names doesn't include projects for which there is no testcases
E.g. on fuzzer stats page, the project filter only includes projects for which there is at least one Testcase.


Discussion:
Simple fix, now we can switch project field projection on Job entity.

Resolving patch:
diff --git a/src/python/datastore/data_handler.py b/src/python/datastore/data_handler.py
--- a/src/python/datastore/data_handler.py
+++ b/src/python/datastore/data_handler.py
@@ -94,10 +94,9 @@
 @memoize.wrap(memoize.Memcache(MEMCACHE_TTL_IN_SECONDS))
 def get_all_project_names():
   """Return all project names."""
-  query = data_types.Testcase.query(
-      projection=[data_types.Testcase.project_name],
-      distinct=True).order(data_types.Testcase.project_name)
-  return [testcase.project_name for testcase in query if testcase.project_name]
+  query = data_types.Job.query(
+      projection=[data_types.Job.project], distinct=True)
+  return sorted([job.project for job in query])
 
 
 def get_domain():
@@ -1491,7 +1490,7 @@ def get_all_job_type_names(project=None):
   query = data_types.Job.query(projection=['name'])
   if project:
     query = query.filter(data_types.Job.project == project)
-  return sorted([_.name for _ in query])
+  return sorted([job.name for job in query])
 
 
 def get_coverage_information(fuzzer_name, date, create_if_needed=False):


Test patch:
diff --git a/src/python/tests/appengine/handlers/cron/cleanup_test.py b/src/python/tests/appengine/handlers/cron/cleanup_test.py
--- a/src/python/tests/appengine/handlers/cron/cleanup_test.py
+++ b/src/python/tests/appengine/handlers/cron/cleanup_test.py
@@ -1000,9 +1000,10 @@ def setUp(self):
     ]
     self.mock.get.return_value = (1, self.top_crashes_rows)
     self.mock.get_last_successful_hour.return_value = 10000
-
-    data_types.Job(name='job', platform='LINUX', environment_string=('')).put()
-    test_utils.create_generic_testcase()
+    data_types.Job(
+        name='job',
+        platform='LINUX',
+        environment_string=('PROJECT_NAME = project')).put()
 
   def test(self):
     """Test."""
Repository: google/clusterfuzz

Issue:
Adding Label field in the Clusterfuzz form fill
To get due credit for the bugs found from Microsoft Fuzzing, a label needs to be added while submitting the job in Clusterfuzz. It can be on the form where one can add the label and it auto adds on the bug created by the Clusterfuzz.


Discussion:
Re-opening as we need to relax the restrictions here to let non-privileged users add this in ClusterFuzz. However, we can't do this in places like OSS-Fuzz since labels there control deadlines.

Resolving patch:
diff --git a/src/appengine/handlers/upload_testcase.py b/src/appengine/handlers/upload_testcase.py
--- a/src/appengine/handlers/upload_testcase.py
+++ b/src/appengine/handlers/upload_testcase.py
@@ -173,6 +173,18 @@ def find_fuzz_target(engine, target_name, job_name):
   return None, None
 
 
+def _allow_unprivileged_metadata(testcase_metadata):
+  """Returns whether or not the provided testcase metadata can be set by an
+  unprivileged user."""
+  if utils.is_oss_fuzz():
+    # Labels in OSS-Fuzz are privileged and control things like disclosure
+    # deadlines. Do not let these be editable.
+    return False
+
+  # Allow *only* issue labels to be set.
+  return len(testcase_metadata) == 1 and 'issue_labels' in testcase_metadata
+
+
 class Handler(base_handler.Handler):
   """Handler for the testcase uploads page."""
 
@@ -390,7 +402,8 @@ def do_post(self):
         raise helpers.EarlyExitException(
             'You are not privileged to run arbitary launch commands.', 400)
 
-      if testcase_metadata:
+      if (testcase_metadata and
+          not _allow_unprivileged_metadata(testcase_metadata)):
         raise helpers.EarlyExitException(
             'You are not privileged to set testcase metadata.', 400)
 


Test patch:
Repository: google/clusterfuzz

Issue:
Command field empty in OSS-Fuzz testcases
See https://oss-fuzz.com/testcase-detail/5204819744915456 for example.


Discussion:

Resolving patch:
diff --git a/src/python/bot/untrusted_runner/tasks_impl.py b/src/python/bot/untrusted_runner/tasks_impl.py
--- a/src/python/bot/untrusted_runner/tasks_impl.py
+++ b/src/python/bot/untrusted_runner/tasks_impl.py
@@ -151,6 +151,7 @@ def engine_reproduce(request, _):
                                              request.testcase_path,
                                              request.arguments, request.timeout)
   return untrusted_runner_pb2.EngineReproduceResult(
+      command=result.command,
       return_code=result.return_code,
       time_executed=result.time_executed,
       output=result.output)


Test patch:
diff --git a/src/python/tests/core/bot/testcase_manager_test.py b/src/python/tests/core/bot/testcase_manager_test.py
--- a/src/python/tests/core/bot/testcase_manager_test.py
+++ b/src/python/tests/core/bot/testcase_manager_test.py
@@ -23,7 +23,9 @@
 
 from bot import testcase_manager
 from bot.fuzzers import engine
+from bot.fuzzers.libFuzzer import constants as libfuzzer_constants
 from bot.fuzzers.libFuzzer import engine as libfuzzer_engine
+from bot.untrusted_runner import file_host
 from build_management import build_manager
 from crash_analysis.crash_result import CrashResult
 from crash_analysis.stack_parsing import stack_analyzer
@@ -758,6 +760,7 @@ def setUp(self):
     """Set up."""
     super(UntrustedEngineReproduceTest, self).setUp()
     environment.set_value('JOB_NAME', 'libfuzzer_asan_job')
+    environment.set_value('USE_MINIJAIL', False)
 
     job = data_types.Job(
         name='libfuzzer_asan_job',
@@ -789,6 +792,14 @@ def test_reproduce(self):
         libfuzzer_engine.LibFuzzerEngine(), 'test_fuzzer', testcase_file_path,
         [], 30)
 
+    self.assertEqual([
+        os.path.join(environment.get_value('BUILD_DIR'), 'test_fuzzer'),
+        '-runs=100',
+        file_host.rebase_to_worker_root(testcase_file_path)
+    ], result.command)
+    self.assertEqual(result.return_code,
+                     libfuzzer_constants.TARGET_ERROR_EXITCODE)
+    self.assertGreater(result.time_executed, 0)
     self.assertIn('Running 1 inputs 100 time(s) each', result.output)
     self.assertIn('AddressSanitizer: SEGV on unknown address 0x000000000000',
                   result.output)
Repository: google/clusterfuzz

Issue:
_update_issue_metadata in progression task fails on OSS-Fuzz
This is due to use of untrusted runner on OSS-Fuzz. Is this even needed there, currently causing exceptions. Should this be a simple bailout ?


Discussion:
Causing "Failed to find target path for fb2fuzzer" exceptions

Resolving patch:
diff --git a/src/python/bot/tasks/progression_task.py b/src/python/bot/tasks/progression_task.py
--- a/src/python/bot/tasks/progression_task.py
+++ b/src/python/bot/tasks/progression_task.py
@@ -140,6 +140,10 @@ def _check_fixed_for_custom_binary(testcase, job_type, testcase_file_path):
 
 def _update_issue_metadata(testcase):
   """Update issue metadata."""
+  if environment.is_trusted_host():
+    # Not applicable.
+    return
+
   fuzz_target = testcase.get_fuzz_target()
   if not fuzz_target:
     return


Test patch:
Repository: google/clusterfuzz

Issue:
Add security flag for recently added UBSan checks
As we enabled new UBSan checks and added new signatures, some of those (maybe even all?) should be reported as non security bugs. By default CF reports "unknown" crash types as security crashes (to be extra cautious), example: https://bugs.chromium.org/p/chromium/issues/detail?id=1029052


Discussion:
Probably just need to expand list here - https://github.com/google/clusterfuzz/blob/master/src/python/crash_analysis/crash_analyzer.py#L69

Resolving patch:
diff --git a/src/python/crash_analysis/crash_analyzer.py b/src/python/crash_analysis/crash_analyzer.py
--- a/src/python/crash_analysis/crash_analyzer.py
+++ b/src/python/crash_analysis/crash_analyzer.py
@@ -73,8 +73,14 @@
     # being used in LLVM and C++, as there is no "signed int" type.
     'Integer-overflow',
     'Invalid-bool-value',
+    'Invalid-builtin-use',
+    'Incorrect-function-pointer-type',
+    'Invalid-enum-value',
+    'Invalid-null-argument',
+    'Invalid-null-return',
     'Misaligned-address',
     'No-return-value',
+    'Non-positive-vla-bound-value',
     'Pointer-overflow',
     'Potential-null-reference',
     'Undefined-shift',
@@ -83,6 +89,7 @@
     # issues may cause some bugs, unless it's not an intended overflow, e.g. in
     # math of crpto libraries.
     'Unsigned-integer-overflow',
+    'Unreachable code',
 ]
 UBSAN_CRASH_TYPES_SECURITY = [
     'Bad-cast',


Test patch:
Repository: google/clusterfuzz

Issue:
Support authentication with Cloud IAP



Discussion:

Resolving patch:
diff --git a/src/appengine/libs/auth.py b/src/appengine/libs/auth.py
--- a/src/appengine/libs/auth.py
+++ b/src/appengine/libs/auth.py
@@ -13,12 +13,17 @@
 # limitations under the License.
 """Authentication helpers."""
 
+from builtins import str
 import collections
+import jwt
 
 from firebase_admin import auth
 from google.cloud import ndb
+from googleapiclient.discovery import build
+import requests
 import webapp2
 
+from base import memoize
 from base import utils
 from config import local_config
 from datastore import data_types
@@ -54,6 +59,68 @@ def is_current_user_admin():
   return bool(key.get())
 
 
+@memoize.wrap(memoize.FifoInMemory(1))
+def _project_number_from_id(project_id):
+  """Get the project number from project ID."""
+  resource_manager = build('cloudresourcemanager', 'v1')
+  result = resource_manager.projects().get(projectId=project_id).execute()
+  if 'projectNumber' not in result:
+    raise AuthError('Failed to get project number.')
+
+  return result['projectNumber']
+
+
+@memoize.wrap(memoize.FifoInMemory(1))
+def _get_iap_key(key_id):
+  """Retrieves a public key from the list published by Identity-Aware Proxy,
+  re-fetching the key file if necessary.
+  """
+  resp = requests.get('https://www.gstatic.com/iap/verify/public_key')
+  if resp.status_code != 200:
+    raise AuthError('Unable to fetch IAP keys: {} / {} / {}'.format(
+        resp.status_code, resp.headers, resp.text))
+
+  result = resp.json()
+  key = result.get(key_id)
+  if not key:
+    raise AuthError('Key {!r} not found'.format(key_id))
+
+  return key
+
+
+def _validate_iap_jwt(iap_jwt):
+  """Validate JWT assertion."""
+  project_id = utils.get_application_id()
+  expected_audience = '/projects/{}/apps/{}'.format(
+      _project_number_from_id(project_id), project_id)
+
+  try:
+    key_id = jwt.get_unverified_header(iap_jwt).get('kid')
+    if not key_id:
+      raise AuthError('No key ID.')
+
+    key = _get_iap_key(key_id)
+    decoded_jwt = jwt.decode(
+        iap_jwt,
+        key,
+        algorithms=['ES256'],
+        issuer='https://cloud.google.com/iap',
+        audience=expected_audience)
+    return decoded_jwt['email']
+  except (jwt.exceptions.InvalidTokenError,
+          requests.exceptions.RequestException) as e:
+    raise AuthError('JWT assertion decode error: ' + str(e))
+
+
+def get_iap_email(current_request):
+  """Get Cloud IAP email."""
+  jwt_assertion = current_request.headers.get('X-Goog-IAP-JWT-Assertion')
+  if not jwt_assertion:
+    return None
+
+  return _validate_iap_jwt(jwt_assertion)
+
+
 def get_current_user():
   """Get the current logged in user, or None."""
   if environment.is_local_development():
@@ -64,6 +131,10 @@ def get_current_user():
     return User(loas_user + '@google.com')
 
   current_request = get_current_request()
+  iap_email = get_iap_email(current_request)
+  if iap_email:
+    return User(iap_email)
+
   oauth_email = getattr(current_request, '_oauth_email', None)
   if oauth_email:
     return User(oauth_email)


Test patch:
Repository: google/clusterfuzz

Issue:
Check Android integration tests under Python 3
Per title.


Discussion:
@inferno-chromium When you get the chance, could you please check if the Android tests pass under Python 3? I'm not able to run them as I don't have a test device.
(Or anyone else who can help test with a device).
Will work on it later in the week.

Resolving patch:
diff --git a/src/python/platforms/android/adb.py b/src/python/platforms/android/adb.py
--- a/src/python/platforms/android/adb.py
+++ b/src/python/platforms/android/adb.py
@@ -108,14 +108,16 @@ def directory_exists(directory_path):
 def execute_command(cmd, timeout=None, log_error=True):
   """Spawns a subprocess to run the given shell command."""
   so = []
-  output_dest = tempfile.TemporaryFile(bufsize=0)
+  output_dest = tempfile.TemporaryFile()
+  # pylint: disable=subprocess-popen-preexec-fn
   pipe = subprocess.Popen(
       cmd,
       executable='/bin/bash',
       stdout=output_dest,
       stderr=subprocess.STDOUT,
       shell=True,
-      preexec_fn=lambda: signal.signal(signal.SIGPIPE, signal.SIG_DFL))
+      preexec_fn=lambda: signal.signal(signal.SIGPIPE, signal.SIG_DFL),
+      bufsize=0)
 
   def run():
     """Thread target function that waits for subprocess to complete."""
@@ -132,7 +134,7 @@ def run():
       if log_error:
         logs.log_warn(
             '%s returned %d error code.' % (cmd, pipe.returncode),
-            output=''.join(so).strip())
+            output=output)
 
   thread = threading.Thread(target=run)
   thread.start()
@@ -146,8 +148,8 @@ def run():
 
     return None
 
-  output = ''.join(so)
-  return output.strip()
+  bytes_output = b''.join(so)
+  return bytes_output.strip().decode('utf-8', errors='ignore')
 
 
 def factory_reset():
@@ -493,7 +495,7 @@ def reset_usb():
     return True
 
   # App Engine does not let us import this.
-  import fcntl
+  import fcntl  # pylint: disable=import-outside-toplevel
 
   # We need to get latest device path since it could be changed in reboots or
   # adb root restarts.


Test patch:
Repository: google/clusterfuzz

Issue:
in local development, zip uploads are broken
```
|   File "/build/clusterfuzz/src/appengine/handlers/jobs.py", line 132, in post
|     blob_info = self.get_upload()
|   File "/build/clusterfuzz/src/appengine/handlers/base_handler.py", line 317, in get_upload
|     blob_info = storage.GcsBlobInfo.from_key(upload_key)
|   File "/build/clusterfuzz/src/python/google_cloud_utils/storage.py", line 556, in from_key
|     logs.log_error('Failed to get blob from key %s.' % key)
| LogError: Failed to get blob from key 0e3179ad-31b4-4ba5-a45c-86a610e065c7.
| Traceback (most recent call last):
|   File "/build/clusterfuzz/src/python/google_cloud_utils/storage.py", line 554, in from_key
|     return GcsBlobInfo(blobs_bucket(), key)
|   File "/build/clusterfuzz/src/python/google_cloud_utils/storage.py", line 536, in __init__
|     self.filename = gcs_object['metadata'].get(BLOB_FILENAME_METADATA_KEY)
| TypeError: 'NoneType' object is not subscriptable
```


Discussion:
Is the go gcs server running while `run_server` is running? 
I just tested this and it worked. If you're accessing the web interface via SSH tunnelling, make sure port 9008 (for the mock GCS server) is forwarded as well.
I think this was an issue with TEST_BLOBS_BUCKET set for integration tests. This is yet another time this bites me. Removing it works. Any thoughts on fixing this, not a blocker for other people, but creates confusion for us running integration tests

Resolving patch:
diff --git a/src/local/butler/run_server.py b/src/local/butler/run_server.py
--- a/src/local/butler/run_server.py
+++ b/src/local/butler/run_server.py
@@ -59,7 +59,12 @@ def bootstrap_gcs(storage_path):
     os.mkdir(local_gcs_buckets_path)
 
   config = local_config.ProjectConfig()
-  create_local_bucket(local_gcs_buckets_path, config.get('blobs.bucket'))
+  test_blobs_bucket = os.environ.get('TEST_BLOBS_BUCKET')
+  if test_blobs_bucket:
+    create_local_bucket(local_gcs_buckets_path, test_blobs_bucket)
+  else:
+    create_local_bucket(local_gcs_buckets_path, config.get('blobs.bucket'))
+
   create_local_bucket(local_gcs_buckets_path, config.get('deployment.bucket'))
   create_local_bucket(local_gcs_buckets_path, config.get('bigquery.bucket'))
   create_local_bucket(local_gcs_buckets_path, config.get('backup.bucket'))


Test patch:
Repository: google/clusterfuzz

Issue:
js_minimizer is not callable
The following run function of JSMinimizer class is not callable as `js_minimizer` is an object and not a function, thus it should not be callable.

```
    js_tokenizer = AntlrTokenizer(JavaScriptLexer)

    js_minimizer = JSMinimizer(
        utils.test,
        max_threads=thread_count,
        tokenizer=js_tokenizer.tokenize,
        token_combiner=js_tokenizer.combine,
        file_extension=file_extension)

    result = line_minimizer.minimize(data)
    result = js_minimizer(result)
    result = js_minimizer(result)
    result = line_minimizer.minimize(result)
```

Location in codebase [here](https://github.com/google/clusterfuzz/blob/master/src/python/bot/minimizer/js_minimizer.py#L187).

Also, does js_minimizer should be called two times?
I guess `js_minimizer(result)` should be replaced with `js_minimizer.minimize(data)`, but I am not really sure. Can someone please confirm if it is a bug or not?


Discussion:
How are you running this code such that you see this error? I don't recall exactly how we've managed to set this up, but objects can be callable. We repeat the calls because we actually want to run the minimizer twice and take the second result, as there are certain types of issues that are not caught by a single pass.
I got the error while using pylint to check this file for any linting issues:

```
************* Module src.python.bot.minimizer.js_minimizer
bot/minimizer/js_minimizer.py:187:13: E1102: js_minimizer is not callable (not-callable)
bot/minimizer/js_minimizer.py:188:13: E1102: js_minimizer is not callable (not-callable)
```
> How are you running this code such that you see this error? I don't recall exactly how we've managed to set this up, but objects can be callable. We repeat the calls because we actually want to run the minimizer twice and take the second result, as there are certain types of issues that are not caught by a single pass.

I don't see `__call__` defined in https://github.com/google/clusterfuzz/blob/master/src/python/bot/minimizer/js_minimizer.py#L35 or the base class (https://github.com/google/clusterfuzz/blob/master/src/python/bot/minimizer/minimizer.py#L145). @mbarbella-chromium can you please confirm how these are working?

Resolving patch:
diff --git a/src/python/bot/minimizer/js_minimizer.py b/src/python/bot/minimizer/js_minimizer.py
--- a/src/python/bot/minimizer/js_minimizer.py
+++ b/src/python/bot/minimizer/js_minimizer.py
@@ -184,8 +184,8 @@ def run(data,
         file_extension=file_extension)
 
     result = line_minimizer.minimize(data)
-    result = js_minimizer(result)
-    result = js_minimizer(result)
+    result = js_minimizer.minimize(result)
+    result = js_minimizer.minimize(result)
     result = line_minimizer.minimize(result)
 
     return result


Test patch:
Repository: google/clusterfuzz

Issue:
Gracefully handle tasks when build isn't found. 
For cases when a bucket has a lifecycle policy, if a target is removed/renamed we will repeatedly try tasks on the same testcase even though we'll never get a build.


Discussion:

Resolving patch:
diff --git a/src/python/bot/tasks/progression_task.py b/src/python/bot/tasks/progression_task.py
--- a/src/python/bot/tasks/progression_task.py
+++ b/src/python/bot/tasks/progression_task.py
@@ -273,10 +273,8 @@ def find_fixed_range(testcase_id, job_type):
   revision_list = build_manager.get_revisions_list(
       build_bucket_path, testcase=testcase)
   if not revision_list:
-    testcase = data_handler.get_testcase_by_id(testcase_id)
-    data_handler.update_testcase_comment(testcase, data_types.TaskState.ERROR,
-                                         'Failed to fetch revision list')
-    tasks.add_task('progression', testcase_id, job_type)
+    data_handler.close_testcase_with_error(testcase_id,
+                                           'Failed to fetch revision list')
     return
 
   # Use min, max_index to mark the start and end of revision list that is used
diff --git a/src/python/bot/tasks/regression_task.py b/src/python/bot/tasks/regression_task.py
--- a/src/python/bot/tasks/regression_task.py
+++ b/src/python/bot/tasks/regression_task.py
@@ -239,10 +239,8 @@ def find_regression_range(testcase_id, job_type):
   revision_list = build_manager.get_revisions_list(
       build_bucket_path, testcase=testcase)
   if not revision_list:
-    testcase = data_handler.get_testcase_by_id(testcase_id)
-    data_handler.update_testcase_comment(testcase, data_types.TaskState.ERROR,
-                                         'Failed to fetch revision list')
-    tasks.add_task('regression', testcase_id, job_type)
+    data_handler.close_testcase_with_error(testcase_id,
+                                           'Failed to fetch revision list')
     return
 
   # Don't burden NFS server with caching these random builds.
diff --git a/src/python/datastore/data_handler.py b/src/python/datastore/data_handler.py
--- a/src/python/datastore/data_handler.py
+++ b/src/python/datastore/data_handler.py
@@ -1522,3 +1522,12 @@ def get_coverage_information(fuzzer_name, date, create_if_needed=False):
         fuzzer=fuzzer_name, date=date)
 
   return coverage_info
+
+
+def close_testcase_with_error(testcase_id, error_message):
+  """Close testcase (fixed=NA) with an error message."""
+  testcase = get_testcase_by_id(testcase_id)
+  update_testcase_comment(testcase, data_types.TaskState.ERROR, error_message)
+  testcase.fixed = 'NA'
+  testcase.open = False
+  testcase.put()


Test patch: