Repository: google/clusterfuzz

Issue:
Endpoints "/fuzz-strategy-selection" and "/fuzzer-and-job-weights" under cron-service failed. 
Hi,
Deploy was done from the master, not from the latest tag. 


The reason in both cases is the same:
**"Table afl_stats.TestcaseRun does not have a schema.">. Retrying again.**

Cron-jobs:

Below details for an endpoint "/fuzz-strategy-selection":
![image](https://user-images.githubusercontent.com/60612203/165520715-937f4ed5-3787-4a3c-8100-7d2ef620fbbb.png)

Stacktrace:
```
<HttpError 400 when requesting https://bigquery.googleapis.com/bigquery/v2/projects/sta-np-11528163/queries?alt=json returned "Table afl_stats.TestcaseRun does not have a schema.">
Traceback (most recent call last):
  File "/srv/handlers/base_handler.py", line 277, in dispatch_request
    return super(Handler, self).dispatch_request(*args, **kwargs)
  File "third_party/flask/views.py", line 163, in dispatch_request
    return meth(*args, **kwargs)
  File "/srv/libs/handler.py", line 100, in wrapper
    result = func(self)
  File "/srv/handlers/cron/fuzz_strategy_selection.py", line 218, in get
    _query_and_upload_strategy_probabilities(engine)
  File "/srv/handlers/cron/fuzz_strategy_selection.py", line 185, in _query_and_upload_strategy_probabilities
    data = _query_multi_armed_bandit_probabilities(engine)
  File "/srv/handlers/cron/fuzz_strategy_selection.py", line 151, in _query_multi_armed_bandit_probabilities
    return client.query(query=formatted_query).rows
  File "/srv/clusterfuzz/_internal/google_cloud_utils/big_query.py", line 247, in query
    result = self.raw_query(query, max_results=0)
  File "/srv/clusterfuzz/_internal/base/retry.py", line 88, in _wrapper
    result = func(*args, **kwargs)
  File "/srv/clusterfuzz/_internal/google_cloud_utils/big_query.py", line 190, in raw_query
    projectId=self.project_id, body=body).execute()
  File "third_party/googleapiclient/_helpers.py", line 134, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "third_party/googleapiclient/http.py", line 907, in execute
    raise HttpError(resp, content, uri=self.uri)
googleapiclient.errors.HttpError: <HttpError 400 when requesting https://bigquery.googleapis.com/bigquery/v2/projects/sta-np-11528163/queries?alt=json returned "Table afl_stats.TestcaseRun does not have a schema.">
```

**Is it normal behaviour?**

**Do you have any idea what can be wrong? How to fix it?**

So far we have only tested libfuzzer, afl wasn't touched yet.

Thanks,


Discussion:
Were you able to get this fixed?  We get the same error.  Could we just copy the schema from a different fuzzer?
Or can we just directly delete the bigquery dataset for the ones that do not have a schema?  Is there a proper way to do this?

Resolving patch:
diff --git a/src/appengine/handlers/cron/load_bigquery_stats.py b/src/appengine/handlers/cron/load_bigquery_stats.py
--- a/src/appengine/handlers/cron/load_bigquery_stats.py
+++ b/src/appengine/handlers/cron/load_bigquery_stats.py
@@ -77,7 +77,7 @@ def _create_dataset_if_needed(self, bigquery, dataset_id):
 
     return self._execute_insert_request(dataset_insert)
 
-  def _create_table_if_needed(self, bigquery, dataset_id, table_id):
+  def _create_table_if_needed(self, bigquery, dataset_id, table_id, schema):
     """Create a new table if needed."""
     project_id = utils.get_application_id()
     table_body = {
@@ -91,6 +91,9 @@ def _create_table_if_needed(self, bigquery, dataset_id, table_id):
         },
     }
 
+    if schema is not None:
+      table_body['schema'] = schema
+
     table_insert = bigquery.tables().insert(
         projectId=project_id, datasetId=dataset_id, body=table_body)
     return self._execute_insert_request(table_insert)
@@ -110,14 +113,16 @@ def _load_data(self, bigquery, fuzzer):
     for kind in STATS_KINDS:
       kind_name = kind.__name__
       table_id = kind_name
-      if not self._create_table_if_needed(bigquery, dataset_id, table_id):
-        continue
 
       if kind == fuzzer_stats.TestcaseRun:
         schema = fuzzer_stats_schema.get(fuzzer)
       else:
         schema = kind.SCHEMA
 
+      if not self._create_table_if_needed(bigquery, dataset_id, table_id,
+                                          schema):
+        continue
+
       gcs_path = fuzzer_stats.get_gcs_stats_path(kind_name, fuzzer, timestamp)
       load = {
           'destinationTable': {


Test patch:
diff --git a/src/clusterfuzz/_internal/tests/appengine/handlers/cron/load_bigquery_stats_test.py b/src/clusterfuzz/_internal/tests/appengine/handlers/cron/load_bigquery_stats_test.py
--- a/src/clusterfuzz/_internal/tests/appengine/handlers/cron/load_bigquery_stats_test.py
+++ b/src/clusterfuzz/_internal/tests/appengine/handlers/cron/load_bigquery_stats_test.py
@@ -20,6 +20,7 @@
 import webtest
 
 from clusterfuzz._internal.datastore import data_types
+from clusterfuzz._internal.metrics import fuzzer_stats
 from clusterfuzz._internal.tests.test_libs import helpers as test_helpers
 from clusterfuzz._internal.tests.test_libs import test_utils
 from handlers.cron import load_bigquery_stats
@@ -76,6 +77,7 @@ def test_execute(self):
                     'tableId': 'JobRun',
                     'datasetId': 'fuzzer_stats',
                 },
+                'schema': fuzzer_stats.JobRun.SCHEMA,
             },
             datasetId='fuzzer_stats',
             projectId='test-clusterfuzz'),