Repository: PIC-IRIS/PH5 (language: python)

Issue:
[FEATURE] add support for SmartSolo SEGD data to pforma
segdtoph5 has been updated to handle the SmartSolo SEGD data (PR 451). The next step in streamlining the handling of SmartSolo data ingestion into PH5 is parallelizing ingestion of SEGD data via pforma. Ingestion via command line is too slow to handle the demands of larger SmartSolo experiments. For example, ingesting 745 GB of SmartSolo data via the linear command line process took ~7 days. pforma needs to be updated to accept the SmartSolo SEGD data as a valid input format to allow for faster processing of SmartSolo data.

Current behavior of pforma:
[2021-06-09 14:56:31,274] - ph5.utilities.pformagui - ERROR: 3: File in /external/ph5/Working/test-alissa/Smart_Solo_SEGD_MSEED_Test/segd-list does not have standard name: 453005477.1.2020.10.02.17.00.00.000.E.segd



Discussion / hints:

Fix patch:
diff --git a/CHANGELOG.txt b/CHANGELOG.txt
index 7806bb11..d70dd147 100644
--- a/CHANGELOG.txt
+++ b/CHANGELOG.txt
@@ -39,6 +39,7 @@ ph5.utilities.segdtoph5
  * Fixed bug causing pforma to finish correctly on node (issue #233)
  * fixed bug in utmzone processing
  * Fixed bug in smartsolo reading causing incorrect time conversions from BCD
+ * Work with SmartSolo in which entries of same deployment (no gaps) are combined into one
 ph5.core.experiment
  * fixed bug found by Natalie (issue #231)
  * fix fast offset_table reading (issue #280)
@@ -93,6 +94,7 @@ ph5.utilities.seg2toph5
  * Add flag -r/--raw
  * Make compatible with both Obspy 1.1.1 and 1.2.2
 ph5.utilities.pforma_io
+ * add smartsolo to pforma
  * add seg2 to pforma
  * fix bug that duplicate index_t and array_t when copy tables from A/master.ph5 to Sigma/master.ph5
 ph5.utilities.125atoph5
diff --git a/create_ph5.sh b/create_ph5.sh
index a9ab7235..2271e951 100755
--- a/create_ph5.sh
+++ b/create_ph5.sh
@@ -50,7 +50,7 @@ cd ../
 mkdir availability_extent
 cd availability_extent
 initialize_ph5 -n master.ph5
-segdtoph5 -n master.ph5 -U 13N -r ../../segd/3ch.fcnt
+segdtoph5 -n master.ph5 -U 13N -r ../../segd/fairfield/3ch.fcnt
 130toph5 -n master.ph5 -r ../../rt130/2016139.9EEF.ZIP
 125atoph5 -n master.ph5 -r ../../rt125a/I2183RAW.TRD
 metadatatoph5 -n master.ph5 -f ../../metadata/station.xml
diff --git a/ph5/utilities/pforma_io.py b/ph5/utilities/pforma_io.py
index d0bfbc9a..c7071c0a 100755
--- a/ph5/utilities/pforma_io.py
+++ b/ph5/utilities/pforma_io.py
@@ -455,8 +455,8 @@ def read(self):
                      naming scheme. Please rename.")
             if tp == 'unknown':
                 raise FormaIOError(errno=3,
-                                   msg="File in {1} does not have standard\
-                                    name: {0}".format(
+                                   msg=("File in {1} does not have standard "
+                                        "name: {0}").format(
                                        raw_file, self.infile))
 
             # Save info about each raw file keyed by serial number in
@@ -759,10 +759,14 @@ def check_sum(filename):
 seg2RE = re.compile(r"(\d+)\.dat")
 # For type 'rt-130'
 rt130RE = re.compile(r"\d\d\d\d\d\d\d\.(\w\w\w\w)(\.\d\d)?\.[Zz][Ii][Pp]")
+# ============================= SEGD ============================== #
 # For type 'nodal'
 nodalRE = re.compile(r"[Rr](\d+)_(\d+)\.\d+\.\d+\.[Rr][Gg](\d+)")
 # For simpleton 'nodal'
 simpletonodalRE = re.compile(r"\d+\.fcnt")
+# For type SmartSolo (except for RE, it is still understood as 'nodal')
+martSoloRE = re.compile(r"(\d+)[\d.]+.[ENZenz].segd")
+# =========================== END SEGD ============================= #
 # For PIC rename
 picnodalRE = re.compile(r"PIC_(\d+)_(\d+)_\d+\.\d+\.\d+\.[Rr][Gg](\d+)")
 
@@ -790,6 +794,10 @@ def guess_instrument_type(filename):
     mo = simpletonodalRE.match(filename)
     if mo:
         return 'nodal', 'lllsss'
+    mo = martSoloRE.match(filename)
+    if mo:
+        das = mo.group(1)
+        return 'nodal', das
     mo = seg2RE.match(filename)
     if mo:
         das = mo.groups()[0]


Test patch:
diff --git a/ph5/utilities/tests/test_pforma_io.py b/ph5/utilities/tests/test_pforma_io.py
index d4a3180b..11b9b5e9 100644
--- a/ph5/utilities/tests/test_pforma_io.py
+++ b/ph5/utilities/tests/test_pforma_io.py
@@ -65,6 +65,23 @@ def test_unite(self):
         self.assertEqual(INDEX_T_A.rows[0],
                          INDEX_T_Sigma.rows[0])
 
+    def test_guess_instrument_type(self):
+        # SmartSolo filename
+        filename = "453005483.1.2021.03.15.16.00.00.000.E.segd"
+        ret = pforma_io.guess_instrument_type(filename)
+        self.assertEqual(ret, ('nodal', '453005483'))
+        # unsimpleton fairfield
+        filename = "PIC_1_1111_4886.0.0.rg16"
+        ret = pforma_io.guess_instrument_type(filename)
+        self.assertEqual(ret, ('nodal', '1X1111'))
+        # simpleton fairfield non standard
+        filename = "1111.0.0.fcnt"
+        ret = pforma_io.guess_instrument_type(filename)
+        self.assertEqual(ret, ('unknown', None))
+        filename = "1111.fcnt"
+        ret = pforma_io.guess_instrument_type(filename)
+        self.assertEqual(ret, ('nodal', 'lllsss'))
+
 
 if __name__ == "__main__":
     unittest.main()