Repository: PIC-IRIS/PH5 (language: python)

Issue:
[BUG] DAS table deletion seems to be affecting sort_kef_gen
**Describe the bug**
Running sort_kef_gen -a on a PH5 with DAS tables that have been deleted (and not later reloaded) results in output:
Traceback (most recent call last):
  File "/home/ph5/anaconda3/envs/ph5/bin/sort_kef_gen", line 11, in <module>
    load_entry_point('ph5', 'console_scripts', 'sort_kef_gen')()
  File "/home/ph5/PH5/ph5/utilities/sort_kef_gen.py", line 370, in main
    if not read_all_das():
  File "/home/ph5/PH5/ph5/utilities/sort_kef_gen.py", line 225, in read_all_das
    r, k = EX.ph5_g_receivers.read_das()
  File "/home/ph5/PH5/ph5/core/experiment.py", line 727, in read_das
    ignore_srm)
  File "/home/ph5/PH5/ph5/core/experiment.py", line 44, in check_srm_valid
    if 'sample_rate_multiplier_i' not in keys:
TypeError: argument of type 'NoneType' is not iterable 

The resulting sort.kef is empty.  

The deleted DAS tables seem to be in the DAS list (e.g., looking at the PH5 with kefedit) but do not open & print errors when they are selected.

**Environment (please complete the following information):**
 - OS: ph51, ph52
 - Program Version sort_kef_gen 2019.036

**To Reproduce**
Run sort_kef_gen -n master.ph5 -a > sort.kef on Monahans in Working, or on 18-016_ShaleHillsCZO/post_send_fixes/to_send in Completed.

**Expected behavior**
Can sort_kef_gen ignore (skip?) the deleted DAS tables, so it runs without error and produces a file with contents?


**Additional context**
Running ph5tokef -n master.ph5 -D <S/N of deleted DAS> now results in an error instead of an empty return.

ph5_validate and stationXML extraction run normally and do not produce errors (checked on Monahans).

sort_kef_gen still runs normally on recent archives without deleted DAS tables.


Discussion / hints:

Fix patch:
diff --git a/ph5/core/experiment.py b/ph5/core/experiment.py
index 867ecbc8..39f7dc77 100755
--- a/ph5/core/experiment.py
+++ b/ph5/core/experiment.py
@@ -39,6 +39,10 @@ def check_srm_valid(rows, keys, tablename, ignore_srm=False):
     :param tablename: name of the table (string)
     :param ignore_srm: flag to ignore checking srm when it is True (boolean)
     """
+    if keys is None:
+        LOGGER.warning("Table %s is empty. Use nuke_table > 2019.037 to "
+                       "remove the table" % tablename)
+        return rows, keys, tablename, ignore_srm
     if ignore_srm:
         return
     if 'sample_rate_multiplier_i' not in keys:


Test patch:
diff --git a/ph5/test_data/ph5_empty_das_t/README.txt b/ph5/test_data/ph5_empty_das_t/README.txt
new file mode 100644
index 00000000..3541db61
--- /dev/null
+++ b/ph5/test_data/ph5_empty_das_t/README.txt
@@ -0,0 +1,4 @@
+This PH5 has das_t of das 3X500 and 1X1111.
+Das 1X1111 has been deleted using nuke_table 2019.037. As the result, its das_t has been truncated and created error when running soft_kef_gen.
+PR 482 has been created to fix the above error with this PH5 as test data. sort_kef_gen now only give a warning for that case.
+The reason that deleting das isn’t added in the test but PH5 is added instead is because nuke_table will be fix to not truncate das_t for deleting table anymore.
\ No newline at end of file
diff --git a/ph5/test_data/ph5_empty_das_t/master.ph5 b/ph5/test_data/ph5_empty_das_t/master.ph5
new file mode 100644
index 00000000..db17774a
Binary files /dev/null and b/ph5/test_data/ph5_empty_das_t/master.ph5 differ
diff --git a/ph5/test_data/ph5_empty_das_t/miniPH5_00001.ph5 b/ph5/test_data/ph5_empty_das_t/miniPH5_00001.ph5
new file mode 100644
index 00000000..9dcb2989
Binary files /dev/null and b/ph5/test_data/ph5_empty_das_t/miniPH5_00001.ph5 differ
diff --git a/ph5/test_data/ph5_empty_das_t/sort_test_delete_das.kef b/ph5/test_data/ph5_empty_das_t/sort_test_delete_das.kef
new file mode 100644
index 00000000..ff1d55f7
--- /dev/null
+++ b/ph5/test_data/ph5_empty_das_t/sort_test_delete_das.kef
@@ -0,0 +1,35 @@
+#   sort-kef-gen Version: 2019.036 ph5 Version: 4.1.2
+#   row 1 das Das_g_3X500
+/Experiment_g/Sorts_g/Sort_t
+	array_name_s = 001
+	array_t_name_s = Array_t_001
+	description_s = Recording window 0001
+	start_time/epoch_l = 1502294400
+	start_time/micro_seconds_i = 380000
+	start_time/type_s = BOTH
+	start_time/ascii_s = Wed Aug  9 16:00:00 2017
+	end_time/epoch_l = 1502294430
+	end_time/micro_seconds_i = 0
+	end_time/ascii_s = Wed Aug  9 16:00:30 2017
+	end_time/type_s = BOTH
+	time_stamp/epoch_l = 1632415571
+	time_stamp/ascii_s = Thu Sep 23 16:46:11 2021
+	time_stamp/micro_seconds_i = 0
+	time_stamp/type_s = BOTH
+#   row 2 das Das_g_3X500
+/Experiment_g/Sorts_g/Sort_t
+	array_name_s = 001
+	array_t_name_s = Array_t_001
+	description_s = Recording window 0002
+	start_time/epoch_l = 1502294430
+	start_time/micro_seconds_i = 380000
+	start_time/type_s = BOTH
+	start_time/ascii_s = Wed Aug  9 16:00:30 2017
+	end_time/epoch_l = 1502294460
+	end_time/micro_seconds_i = 0
+	end_time/ascii_s = Wed Aug  9 16:01:00 2017
+	end_time/type_s = BOTH
+	time_stamp/epoch_l = 1632415571
+	time_stamp/ascii_s = Thu Sep 23 16:46:11 2021
+	time_stamp/micro_seconds_i = 0
+	time_stamp/type_s = BOTH
diff --git a/ph5/utilities/tests/test_soft_kef_gen.py b/ph5/utilities/tests/test_soft_kef_gen.py
new file mode 100644
index 00000000..050bd160
--- /dev/null
+++ b/ph5/utilities/tests/test_soft_kef_gen.py
@@ -0,0 +1,56 @@
+'''
+Tests for sort_kefGen
+'''
+import os
+import sys
+import logging
+import unittest
+
+from mock import patch
+from testfixtures import OutputCapture, LogCapture
+
+from ph5.utilities import sort_kef_gen
+from ph5.core.tests.test_base import LogTestCase, TempDirTestCase
+
+
+def rem_currtime_ver_info(text):
+    lines = text.split('\n')[1:]        # remove first line with version
+    lines = [line for line in lines if 'time_stamp' not in line]
+    return '\n'.join(lines)
+
+
+class TestSortKefGen_main(TempDirTestCase, LogTestCase):
+
+    def test_main(self):
+        ph5_dir = os.path.join(self.home, "ph5/test_data/ph5_empty_das_t/")
+
+        # create soft.kef
+        # Related to issue #480, if check_srm_valid isn't skipped for empty
+        # das_t, sort_kef_gen will issue an error
+        testargs = ['sort_kef_gen', '-n', 'master', '-p', ph5_dir, '-a']
+        with patch.object(sys, 'argv', testargs):
+            with LogCapture() as log:
+                log.setLevel(logging.WARNING)
+                with OutputCapture() as out:
+                    sort_kef_gen.main()
+                    output = out.captured.strip()
+                self.assertEqual(len(log.records), 1)
+                self.assertEqual(
+                    log.records[0].msg,
+                    "Table Das_t_1X1111 is empty. "
+                    "Use nuke_table > 2019.037 to remove the table"
+                )
+        output = rem_currtime_ver_info(output)
+
+        # compare with recreated sort_test_delete_das.kef
+        with open(os.path.join(
+                self.home,
+                'ph5/test_data/ph5_empty_das_t/sort_test_delete_das.kef'),
+                  'r') as content_file:
+            content = content_file.read().strip()
+        content = rem_currtime_ver_info(content)
+        self.assertEqual(output, content)
+
+
+if __name__ == "__main__":
+    unittest.main()