Repository: cmu-delphi/delphi-epidata
Issue: "Compute Missing/Stale Covidcast Direction" job dies on missed data_stdevs key

The database has a 32-character limit on the length of signal names. This is irregularly enforced, and it causes Automation to fail when the limit is transgressed.

Signal names are stored in the `data_stdevs[source][signal]` dictionary exactly as they are read from an ingested filename. When they are inserted into the database, they get truncated to 32 characters. The "Compute Missing/Stale Covidcast Direction" job reads signal names out of the database and expects to find them in the `data_stdevs[source][signal]` dictionary, but the truncated names are not listed there. The job fails with a KeyError:

```python
Traceback (most recent call last):
  File "/home/automation/.pyenv/versions/3.4.10/lib/python3.4/runpy.py", line 170, in _run_module_as_main
    "__main__", mod_spec)
  File "/home/automation/.pyenv/versions/3.4.10/lib/python3.4/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/automation/driver/delphi/epidata/acquisition/covidcast/direction_updater.py", line 189, in <module>
    main(get_argument_parser().parse_args())
  File "/home/automation/driver/delphi/epidata/acquisition/covidcast/direction_updater.py", line 177, in main
    update_loop_impl(database)
  File "/home/automation/driver/delphi/epidata/acquisition/covidcast/direction_updater.py", line 137, in update_loop
    data_stdev = data_stdevs[source][signal][geo_type]
KeyError: 'wip_confirmed_7day_avg_cumulativ'
```

We should either extend the character limit in the database schema (perhaps at the same time we add in issue dates), or truncate the signal name after it gets read out of the filename and before it gets loaded into `data_stdevs[source][signal]`.

Changed files:
integrations/acquisition/covidcast/test_csv_uploading.py, src/acquisition/covidcast/csv_importer.py

diff --git a/integrations/acquisition/covidcast/test_csv_uploading.py b/integrations/acquisition/covidcast/test_csv_uploading.py
index 73ccb614..f929224c 100644
--- a/integrations/acquisition/covidcast/test_csv_uploading.py
+++ b/integrations/acquisition/covidcast/test_csv_uploading.py
@@ -82,6 +82,11 @@ class CsvUploadingTests(unittest.TestCase):
     with open(source_receiving_dir + '/hello.csv', 'w') as f:
       f.write('file name is wrong\\n')
 
+    # invalid
+    with open(source_receiving_dir + '/20200419_state_wip_really_long_name_that_will_get_truncated.csv', 'w') as f:
+      f.write('geo_id,val,se,sample_size\\n')
+      f.write('pa,100,5.4,624\\n')
+
     # upload CSVs
     args = MagicMock(data_dir=data_dir)
     main(args)
@@ -160,6 +165,16 @@ class CsvUploadingTests(unittest.TestCase):
       'message': 'success',
     })
 
+    # request CSV data from the API on the long-named signal
+    response = Epidata.covidcast(
+      'src-name', 'wip_really_long_name_that_will_g', 'day', 'state', 20200419, '*')
+
+    # verify data matches the CSV
+    # if the CSV failed correctly there should be no results
+    self.assertEqual(response, {
+      'result': -2,
+      'message': 'no results',
+    })
 
     # verify timestamps and default values are reasonable
     self.cur.execute('select timestamp1, timestamp2, direction from covidcast')
diff --git a/src/acquisition/covidcast/csv_importer.py b/src/acquisition/covidcast/csv_importer.py
index 6570dc8e..a7d5e337 100644
--- a/src/acquisition/covidcast/csv_importer.py
+++ b/src/acquisition/covidcast/csv_importer.py
@@ -121,6 +121,10 @@ class CsvImporter:
       # extract additional values, lowercased for consistency
       source = match.group(1).lower()
       signal = match.group(4).lower()
+      if len(signal) > 32:
+        print(' invalid signal name (32 char limit)',signal)
+        yield (path, None)
+        continue
 
       yield (path, (source, signal, time_type, geo_type, time_value))