Repository: GoogleCloudPlatform/professional-services

Issue:
[dataflow-data-generator]  if fields else self.schema['fields'] TypeError: list indices must be integers or slices, not str [while running 'Generate Data']
I've tried different alternatives:loading the schema from the file, from the table and everytime I try it I get the same error. 
If I look to the schema generated by $bq show --format=json --schema bank-bigquery:datamart.Table1 I don't see fields, maybe something has changed ? and this is the reason of the error ?
 **Finally I've executed this command that works:**
But I needed to modify the schema generated by the command "bq show --format=json --schema bank-bigquery:datamart.Table1 " and I've added the {"fields":XXX}

$python3 data-generator-pipeline/data_generator_pipeline.py \
--project=bank-bigquery --setup_file=data-generator-pipeline/setup.py \
--worker_machine_type=n1-highcpu-32 --runner=DataflowRunner \
--staging_location=gs://bank-raw-data/test --temp_location=gs://bank-raw-data/temp \
--schema_file=gs://bank-raw-data/schemas/Table1.json  --output_bq_table=datamart.Table1 \
--p_null=0.05 \
--num_records=10000 --save_main_session

**This is the command I've executed with error:**
$python3 data-generator-pipeline/data_generator_pipeline.py \
--project=bank-bigquery --setup_file=data-generator-pipeline/setup.py \
--worker_machine_type=n1-highcpu-32 --runner=DataflowRunner \
--staging_location=gs://bank-raw-data/test --temp_location=gs://bank-raw-data/temp \
--input_bq_table=datamart.Table1  --output_bq_table=datamart.Table1 \
--p_null=0.05 \
--num_records=10000 --save_main_session

**This is the output of the error I got:**
OutputProcessor.process_outputs File "/usr/local/lib/python3.7/site-packages/data_generator/PrettyDataGenerator.py", line 565, in process faker_schema = self.data_gen.get_faker_schema() File "/usr/local/lib/python3.7/site-packages/data_generator/PrettyDataGenerator.py", line 248, in get_faker_schema this_call_schema = fields if fields else self.schema['fields'] TypeError: list indices must be integers or slices, not str [while running 'Generate Data']
[dataflow-data-generator]  if fields else self.schema['fields'] TypeError: list indices must be integers or slices, not str [while running 'Generate Data']
I've tried different alternatives:loading the schema from the file, from the table and everytime I try it I get the same error. 
If I look to the schema generated by $bq show --format=json --schema bank-bigquery:datamart.Table1 I don't see fields, maybe something has changed ? and this is the reason of the error ?
 **Finally I've executed this command that works:**
But I needed to modify the schema generated by the command "bq show --format=json --schema bank-bigquery:datamart.Table1 " and I've added the {"fields":XXX}

$python3 data-generator-pipeline/data_generator_pipeline.py \
--project=bank-bigquery --setup_file=data-generator-pipeline/setup.py \
--worker_machine_type=n1-highcpu-32 --runner=DataflowRunner \
--staging_location=gs://bank-raw-data/test --temp_location=gs://bank-raw-data/temp \
--schema_file=gs://bank-raw-data/schemas/Table1.json  --output_bq_table=datamart.Table1 \
--p_null=0.05 \
--num_records=10000 --save_main_session

**This is the command I've executed with error:**
$python3 data-generator-pipeline/data_generator_pipeline.py \
--project=bank-bigquery --setup_file=data-generator-pipeline/setup.py \
--worker_machine_type=n1-highcpu-32 --runner=DataflowRunner \
--staging_location=gs://bank-raw-data/test --temp_location=gs://bank-raw-data/temp \
--input_bq_table=datamart.Table1  --output_bq_table=datamart.Table1 \
--p_null=0.05 \
--num_records=10000 --save_main_session

**This is the output of the error I got:**
OutputProcessor.process_outputs File "/usr/local/lib/python3.7/site-packages/data_generator/PrettyDataGenerator.py", line 565, in process faker_schema = self.data_gen.get_faker_schema() File "/usr/local/lib/python3.7/site-packages/data_generator/PrettyDataGenerator.py", line 248, in get_faker_schema this_call_schema = fields if fields else self.schema['fields'] TypeError: list indices must be integers or slices, not str [while running 'Generate Data']


Discussion:

Resolving patch:
diff --git a/examples/dataflow-data-generator/data-generator-pipeline/data_generator/PrettyDataGenerator.py b/examples/dataflow-data-generator/data-generator-pipeline/data_generator/PrettyDataGenerator.py
--- a/examples/dataflow-data-generator/data-generator-pipeline/data_generator/PrettyDataGenerator.py
+++ b/examples/dataflow-data-generator/data-generator-pipeline/data_generator/PrettyDataGenerator.py
@@ -245,7 +245,10 @@ def get_faker_schema(self, fields=None):
         }
 
         faker_schema = {}
-        this_call_schema = fields if fields else self.schema['fields']
+        if fields:
+            this_call_schema = fields
+        else:  # some users pass a schema wrapped in fields.
+            this_call_schema = self.schema.get('fields', self.schema)
         for obj in this_call_schema:
             is_special = False
             if obj['type'].lower() == 'record':


Test patch:
diff --git a/examples/dataflow-data-generator/bigquery-scripts/tests/test_bq_table_resizer.py b/examples/dataflow-data-generator/bigquery-scripts/tests/test_bq_table_resizer.py
--- a/examples/dataflow-data-generator/bigquery-scripts/tests/test_bq_table_resizer.py
+++ b/examples/dataflow-data-generator/bigquery-scripts/tests/test_bq_table_resizer.py
@@ -38,6 +38,7 @@ class TestBigQueryTableResizer(unittest.TestCase):
     @classmethod
     def setUpClass(cls):
         cls._bq_cli = bigquery.Client()
+
     def setUp(self):
         self.client = TestBigQueryTableResizer._bq_cli
         # Create a unique temporary dataset.