Repository: obspy/obspy

Issue:
SEG2 read support for "Data format code 3" (20-bit SEG-D floating point)?
Dear Obspy users/developers,
I am opening this issue because I face an issue when I try to read some Seg2 datafiles from active seismic experiments. Here enclosed are two seg2 files, one is ok (GOOD.DAT) and the other fails (BAD.DAT).
[SEG2_ISSUE.zip](https://github.com/obspy/obspy/files/1536401/SEG2_ISSUE.zip)

Both are generated by a Smartseis (geometrics) acquisition station, but of different age.

I can read the BAD.DAT header without problem, but I face the issue when parsing the trace blockette: it looks like the bit length / data type is badly decoded.

Any help would be very much appreciated (about half of the seismic files I have to work with are of BAD.DAT type..).

Thanks again for this very helpful python tool,
Cheers

Pierre Bottelin
--
ADRGT
Grenoble, France










Discussion:
> I can read the BAD.DAT header without problem, but I face the issue when parsing the trace blockette: it looks like the bit length / data type is badly decoded.

Care to post a complete code snippet that shows the error with full error traceback?
Dear Tobias,
first of all, thank you for your concern & quick answer.

Here enclosed is the full code to execute (in the "SCRIPTS" folder) to reproduce the error. Both good and bad .dat files are in the "DATA" repository.
[SEG2_ISSUE_FULL_SNIPPET.zip](https://github.com/obspy/obspy/files/1542766/SEG2_ISSUE_FULL_SNIPPET.zip)

The error traceback comes indeed from bad trace parsing, the header being correctly read. The error raised is the following:

"NotImplementedError:
Data format code 3 (20-bit SEG-D floating point) not supported yet.
Please contact the ObsPy developers with a sample file."

Although I believe both my "good" and "bad" Smartseis (Geometrix) systems yield SEG2 seismic data.

I tried to find an appropriate parsing in the seg2.py function in Obspy by trial and error (see hereafter), without sucess.

**seg2.py function:**
line 220 and on:
elif data_format_code == 3:
           dtype = np.int32    #for example
           sample_size = 4     #for example

I tried many dtype and sample_size values, but the decoded trace has almost no sense (bit shifts when parsing?)

Best regards,



Just setting the dtype and sample size are not going to do it for some custom 20bit floating point encoding. Some more work is needed but there are already some SEG-D reader for ObsPy out there and maybe the 20bit decoding can be stolen from them? (But I see @ThomasLecocq already self-assigned this issue so I guess he is on it :) ).
Hi Lion, Thomas,
and thank you for your concern about my issue.
I might not be the only one facing this problem, related to the age of the acquisition system I use (Geometrics Smartseis 12 channels S01) that generates 20-bits or 32-bits integer encoding SEG-2 files (not sure of which one, see file enclosed).
[3551.zip](https://github.com/obspy/obspy/files/2437756/3551.zip)


How could this be solved? I spent some time trying to fix it, without success. I notice that this issue was added to 1.2.0 and 1.3.0 milestones, which means it will be fixed in the next release I guess?
Thanks agains for your help & considerable work done there.
Cheers,

Pierre
Hey Pierre,
will check the sample file and contact you directly soon,
tom
Hi Thomas,
any progress on reading 20 bit seg2 files from older smartseis models? I have the same issue as Pierre here.
Thanks
@Pbottelin @stefanazzz @ThomasLecocq do we have documentation for this custom encoding? If we have documentation we could have @DominikStr have a shot at it.

Hi Tobias!
thanks for not dropping this case ;-)

I have a few documents attached to this email, in Appendix C about the 
data format.

Looks like a strange encoding of seg2 files..

Cheers!

Pierre



Le 28/03/2019 à 12:21, Tobias Megies a écrit :
>
> @Pbottelin <https://github.com/Pbottelin> @stefanazzz 
> <https://github.com/stefanazzz> @ThomasLecocq 
> <https://github.com/ThomasLecocq> do we have documentation for this 
> custom encoding? If we have documentation we could have @DominikStr 
> <https://github.com/DominikStr> have a shot at it.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub 
> <https://github.com/obspy/obspy/issues/2022#issuecomment-477554425>, 
> or mute the thread 
> <https://github.com/notifications/unsubscribe-auth/AgugOVdwvYOCVIbCCLAn764NVw0x5TvMks5vbKXSgaJpZM4Q4YbU>.
>

-- 
Pierre BOTTELIN
-
ADRGT
2 Rue de la Condamine
ZI de Mayencin - BP 17
38610 GIERES
Tel : 04 76 44 75 72



---
L'absence de virus dans ce courrier électronique a été vérifiée par le logiciel antivirus Avast.
https://www.avast.com/antivirus

@Pbottelin email replies to github notifications are stripped of attachments, you will have to go to github and post the files here directly on the comment.
Here we go!
[GEOMETRICS_SmartSeisVersionC.pdf](https://github.com/obspy/obspy/files/3022262/GEOMETRICS_SmartSeisVersionC.pdf)

Thanks @Pbottelin  but this pdf does not include information on the encoding of the data blocks it seems..
@megies I think I can work on it this weekend based on my +- working segD code and finally merge it ... 

Alright, nice! So no need to look at this one @DominikStr.
Relevant parts of the SEG-2 documentation. That is certainty something I have not seen before 🙈 

![Screenshot 2019-04-11 at 20 39 53](https://user-images.githubusercontent.com/800487/55982800-f9127980-5c99-11e9-9523-bb17f82b74eb.png)

![Screenshot 2019-04-11 at 20 40 11](https://user-images.githubusercontent.com/800487/55982816-03cd0e80-5c9a-11e9-8560-3f80c16de1d3.png)



It should be possible to do this fairly efficiently in numpy without dropping down to C by first reshaping the data array to (10 bytes, npts/4 bytes) and then performing bit shifts.
This is pretty straightforward and I've written a pretty quick PoC, but the only issue with the sample data is all of the exponents are 0. Is there any sample data available that goes over ±32768 (and thus require an exponent) so this can be properly tested?
> a 15-bit one's complement binary integer.

OK, except for this bit; why'd they have to go and make it complicated?

Resolving patch:
diff --git a/obspy/io/seg2/seg2.py b/obspy/io/seg2/seg2.py
--- a/obspy/io/seg2/seg2.py
+++ b/obspy/io/seg2/seg2.py
@@ -16,7 +16,7 @@
 from future.utils import PY2
 
 from copy import deepcopy
-from struct import unpack
+from struct import unpack, unpack_from
 import warnings
 
 import numpy as np
@@ -102,28 +102,32 @@ def read_file_descriptor_block(self):
         file_descriptor_block = self.file_pointer.read(32)
 
         # Determine the endianness and check if the block id is valid.
-        if unpack(b'B', file_descriptor_block[0:1])[0] == 0x55 and \
-           unpack(b'B', file_descriptor_block[1:2])[0] == 0x3a:
+        if unpack_from(b'2B', file_descriptor_block) == (0x55, 0x3a):
             self.endian = b'<'
-        elif unpack(b'B', file_descriptor_block[0:1])[0] == 0x3a and \
-                unpack(b'B', file_descriptor_block[1:2])[0] == 0x55:
+        elif unpack_from(b'2B', file_descriptor_block) == (0x3a, 0x55):
             self.endian = b'>'
         else:
             msg = 'Wrong File Descriptor Block ID'
             raise SEG2InvalidFileError(msg)
 
         # Check the revision number.
-        revision_number = unpack(self.endian + b'H',
-                                 file_descriptor_block[2:4])[0]
+        revision_number, = unpack_from(self.endian + b'H',
+                                       file_descriptor_block, 2)
         if revision_number != 1:
             msg = '\nOnly SEG 2 revision 1 is officially supported. This file '
             msg += 'has revision %i. Reading it might fail.' % revision_number
             msg += '\nPlease contact the ObsPy developers with a sample file.'
             warnings.warn(msg)
-        size_of_trace_pointer_sub_block = unpack(
-            self.endian + b'H', file_descriptor_block[4:6])[0]
-        number_of_traces = unpack(
-            self.endian + b'H', file_descriptor_block[6:8])[0]
+
+        # Determine trace counts.
+        (size_of_trace_pointer_sub_block,
+         number_of_traces
+         ) = unpack_from(self.endian + b'HH', file_descriptor_block, 4)
+        if number_of_traces * 4 > size_of_trace_pointer_sub_block:
+            msg = ('File indicates %d traces, but there are only %d trace '
+                   'pointers.') % (number_of_traces,
+                                   size_of_trace_pointer_sub_block // 4)
+            raise SEG2InvalidFileError(msg)
 
         # Define the string and line terminators.
         (size_of_string_terminator,
@@ -132,7 +136,7 @@ def read_file_descriptor_block(self):
          size_of_line_terminator,
          first_line_terminator_char,
          second_line_terminator_char
-         ) = unpack(b'BccBcc', file_descriptor_block[8:14])
+         ) = unpack_from(b'BccBcc', file_descriptor_block, 8)
 
         # Assemble the string terminator.
         if size_of_string_terminator == 1:
@@ -156,12 +160,8 @@ def read_file_descriptor_block(self):
         # Read the trace pointer sub-block and retrieve all the pointers.
         trace_pointer_sub_block = \
             self.file_pointer.read(size_of_trace_pointer_sub_block)
-        self.trace_pointers = []
-        for _i in range(number_of_traces):
-            index = _i * 4
-            self.trace_pointers.append(
-                unpack(self.endian + b'L',
-                       trace_pointer_sub_block[index:index + 4])[0])
+        self.trace_pointers = unpack_from(
+            self.endian + (b'L' * number_of_traces), trace_pointer_sub_block)
 
         # The rest of the header up to where the first trace pointer points is
         # a free form section.
@@ -199,30 +199,33 @@ def parse_next_trace(self):
            0x4422:
             msg = 'Invalid trace descriptor block id.'
             raise SEG2InvalidFileError(msg)
-        size_of_this_block = unpack(self.endian + b'H',
-                                    trace_descriptor_block[2:4])[0]
-        number_of_samples_in_data_block = \
-            unpack(self.endian + b'L', trace_descriptor_block[8:12])[0]
-        data_format_code = unpack(b'B', trace_descriptor_block[12:13])[0]
+        size_of_this_block, = unpack_from(self.endian + b'H',
+                                          trace_descriptor_block, 2)
+        number_of_samples_in_data_block, = \
+            unpack_from(self.endian + b'L', trace_descriptor_block, 8)
+        data_format_code, = unpack_from(b'B', trace_descriptor_block, 12)
 
         # Parse the data format code.
         if data_format_code == 4:
-            dtype = np.float32
+            dtype = self.endian + b'f4'
             sample_size = 4
         elif data_format_code == 5:
-            dtype = np.float64
+            dtype = self.endian + b'f8'
             sample_size = 8
         elif data_format_code == 1:
-            dtype = np.int16
+            dtype = self.endian + b'i2'
             sample_size = 2
         elif data_format_code == 2:
-            dtype = np.int32
+            dtype = self.endian + b'i4'
             sample_size = 4
         elif data_format_code == 3:
-            msg = ('\nData format code 3 (20-bit SEG-D floating point) not '
-                   'supported yet.\nPlease contact the ObsPy developers with '
-                   'a sample file.')
-            raise NotImplementedError(msg)
+            dtype = self.endian + b'i2'
+            sample_size = 2.5
+            if number_of_samples_in_data_block % 4 != 0:
+                raise SEG2InvalidFileError(
+                    'Data format code 3 requires that the number of samples '
+                    'is divisible by 4, but sample count is %d' % (
+                        number_of_samples_in_data_block, ))
         else:
             msg = 'Unrecognized data format code'
             raise SEG2InvalidFileError(msg)
@@ -248,9 +251,29 @@ def parse_next_trace(self):
 
         # Unpack the data.
         data = from_buffer(
-            self.file_pointer.read(number_of_samples_in_data_block *
-                                   sample_size),
+            self.file_pointer.read(
+                int(number_of_samples_in_data_block * sample_size)),
             dtype=dtype)
+        if data_format_code == 3:
+            # Convert one's complement to two's complement by adding one to
+            # negative numbers.
+            one_to_two = (data < 0)
+            # The first two bytes (1 word) of every 10 bytes (5 words) contains
+            # a 4-bit exponent for each of the 4 remaining 2-byte (int16)
+            # samples.
+            exponents = data[0::5].view(self.endian + b'u2')
+            result = np.empty(number_of_samples_in_data_block, dtype=np.int32)
+            # Apply the negative correction, then multiply by correct exponent.
+            result[0::4] = ((data[1::5] + one_to_two[1::5]) *
+                            2**((exponents & 0x000f) >> 0))
+            result[1::4] = ((data[2::5] + one_to_two[2::5]) *
+                            2**((exponents & 0x00f0) >> 4))
+            result[2::4] = ((data[3::5] + one_to_two[3::5]) *
+                            2**((exponents & 0x0f00) >> 8))
+            result[3::4] = ((data[4::5] + one_to_two[4::5]) *
+                            2**((exponents & 0xf000) >> 12))
+            data = result
+
         # Integrate SEG2 file header into each trace header
         tmp = self.stream.stats.seg2.copy()
         tmp.update(header['seg2'])
@@ -262,52 +285,58 @@ def parse_free_form(self, free_form_str, attrib_dict):
         Parse the free form section stored in free_form_str and save it in
         attrib_dict.
         """
-        # Separate the strings.
-        strings = free_form_str.split(self.string_terminator)
-        # This is not fully according to the SEG-2 format specification (or
-        # rather the specification only speaks about on offset of 2 bytes
-        # between strings and a string_terminator between two free form
-        # strings. The file I have show the following separation between two
-        # strings: 'random offset byte', 'string_terminator',
-        # 'random offset byte'
-        # Therefore every string has to be at least 3 bytes wide to be
-        # acceptable after being split at the string terminator.
-
-        def is_good_char(c):
-            return c in (b'0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMN'
-                         b'OPQRSTUVWXYZ!"#$%&\'()*+,-./:; <=>?@[\\]^_`{|}~ ')
-
-        # A loop over a bytestring in Python 3 returns integers. This can be
-        # solved with a number of imports from the python-future module and
-        # all kinds of subtle changes throughout this file. Separating the
-        # handling for Python 2 and 3 seems the cleaner and simpler approach.
-        if PY2:
-            strings = ["".join(filter(is_good_char, _i))
-                       for _i in strings
-                       if len(_i) >= 3]
-        else:
-            strings = ["".join(map(chr, filter(is_good_char, _i)))
-                       for _i in strings
-                       if len(_i) >= 3]
+        def cleanup_and_decode_string(value):
+            # Some software/hardware produces invalid characters.
+            def is_good_char(c):
+                return c in (b'0123456789'
+                             b'abcdefghijklmnopqrstuvwxyz'
+                             b'ABCDEFGHIJKLMNOPQRSTUVWXYZ'
+                             b'!"#$%&\'()*+,-./:; <=>?@[\\]^_`{|}~ ')
+
+            # A loop over a bytestring in Python 3 returns integers. This can
+            # be solved with a number of imports from the python-future module
+            # and all kinds of subtle changes throughout this file. Separating
+            # the handling for Python 2 and 3 seems the cleaner and simpler
+            # approach.
+            if PY2:
+                return "".join(filter(is_good_char, value)).strip()
+            else:
+                return "".join(map(chr, filter(is_good_char, value))).strip()
+
+        # Separate the strings. Every string starts with a 2-byte offset to the
+        # next string, and ends with a terminator. An offset of 0 indicates the
+        # end of the strings.
+        offset = 0
+        strings = []
+        while offset + 2 < len(free_form_str):
+            strlen, = unpack_from(self.endian + b'H', free_form_str, offset)
+            if strlen == 0:
+                break
+            curstr = free_form_str[offset + 2:offset + strlen]
+            try:
+                curstrlen = curstr.index(self.string_terminator)
+            except ValueError:
+                strings.append(curstr)
+            else:
+                strings.append(curstr[:curstrlen])
+            offset += strlen
 
         # Every string has the structure OPTION<SPACE>VALUE. Write to
         # stream.stats attribute.
         for string in strings:
-            string = string.strip()
-            string = string.split(' ')
-            key = string[0].strip()
-            value = ' '.join(string[1:]).strip()
+            string = string.strip().split(b' ', 1)
+            key = cleanup_and_decode_string(string[0])
+            try:
+                value = string[1]
+            except IndexError:
+                value = ''
+            if key == 'NOTE':
+                value = [cleanup_and_decode_string(line)
+                         for line in value.split(self.line_terminator)
+                         if line]
+            else:
+                value = cleanup_and_decode_string(value)
             setattr(attrib_dict, key, value)
-        # Parse the notes string again.
-        if hasattr(attrib_dict, 'NOTE'):
-            notes = attrib_dict.NOTE.split(self.line_terminator.decode())
-            attrib_dict.NOTE = AttribDict()
-            for note in notes:
-                note = note.strip()
-                note = note.split(' ')
-                key = note[0].strip()
-                value = ' '.join(note[1:]).strip()
-                setattr(attrib_dict.NOTE, key, value)
 
 
 def _is_seg2(filename):
@@ -321,19 +350,16 @@ def _is_seg2(filename):
         file_pointer.close()
     try:
         # Determine the endianness and check if the block id is valid.
-        if unpack(b'B', file_descriptor_block[0:1])[0] == 0x55 and \
-           unpack(b'B', file_descriptor_block[1:2])[0] == 0x3a:
+        if unpack_from(b'2B', file_descriptor_block) == (0x55, 0x3a):
             endian = b'<'
-        elif unpack(b'B', file_descriptor_block[0:1])[0] == 0x3a and \
-                unpack(b'B', file_descriptor_block[1:2])[0] == 0x55:
+        elif unpack_from(b'2B', file_descriptor_block) == (0x3a, 0x55):
             endian = b'>'
         else:
             return False
     except Exception:
         return False
     # Check the revision number.
-    revision_number = unpack(endian + b'H',
-                             file_descriptor_block[2:4])[0]
+    revision_number, = unpack_from(endian + b'H', file_descriptor_block, 2)
     if revision_number != 1:
         return False
     return True


Test patch:
diff --git a/obspy/io/seg2/tests/data/20180307_031245000.0.DAT.gz b/obspy/io/seg2/tests/data/20180307_031245000.0.DAT.gz
new file mode 100644
Binary files /dev/null and b/obspy/io/seg2/tests/data/20180307_031245000.0.DAT.gz differ
diff --git a/obspy/io/seg2/tests/data/20180307_031245000.0.seg2 b/obspy/io/seg2/tests/data/20180307_031245000.0.seg2
new file mode 100644
Binary files /dev/null and b/obspy/io/seg2/tests/data/20180307_031245000.0.seg2 differ
diff --git a/obspy/io/seg2/tests/test_seg2.py b/obspy/io/seg2/tests/test_seg2.py
--- a/obspy/io/seg2/tests/test_seg2.py
+++ b/obspy/io/seg2/tests/test_seg2.py
@@ -14,10 +14,9 @@
 import numpy as np
 
 from obspy import read
-from obspy.core import AttribDict
 
 
-TRACE1_HEADER = {'ACQUISITION_DATE': '07/JAN/2013',
+TRACE2_HEADER = {'ACQUISITION_DATE': '07/JAN/2013',
                  'ACQUISITION_DATE_UTC': '07/JAN/2013',
                  'ACQUISITION_TIME': '10:30:41',
                  'ACQUISITION_TIME_MICROSECONDS': '0',
@@ -41,7 +40,7 @@
                  'LOCATION': 'LOCATION',
                  'LOW_CUT_FILTER': '10.000000 12.000000',
                  'NETWORK_NAME': 'BANK',
-                 'NOTE': AttribDict({'Comment': ''}),
+                 'NOTE': ['Comment'],
                  'OBSERVER': 'OBSERVER',
                  'REAL_TIME_AVAILABLE': 'FALSE',
                  'REGISTRATION_DIRECTION': 'X',
@@ -61,6 +60,26 @@
                  'TRIGGER_SAMPLE_NO': '8000',
                  'TRIGGER_SOURCE': '0',
                  'UNITS': 'METERS'}
+TRACE3_HEADER = {
+    'ACQUISITION_DATE': '7/MAR/2018',
+    'ACQUISITION_TIME': '3:12:45',
+    'INSTRUMENT': 'GEOMETRICS SmartSeis 0000',
+    'TRACE_SORT': 'AS_ACQUIRED',
+    'UNITS': 'METERS',
+    'NOTE': ['DISPLAY_SCALE 48'],
+    'CHANNEL_NUMBER': '1',
+    'DELAY': '-0.010',
+    'DESCALING_FACTOR': '0.001199',
+    'LINE_ID': '00-00',
+    'LOW_CUT_FILTER': '0 0',
+    'NOTCH_FREQUENCY': '0',
+    'RAW_RECORD': '1068.DAT',
+    'RECEIVER_LOCATION': '1004.00',
+    'SAMPLE_INTERVAL': '0.000125',
+    'SKEW': '-0.00001796',
+    'SOURCE_LOCATION': '1000.00',
+    'STACK': '8',
+}
 
 
 class SEG2TestCase(unittest.TestCase):
@@ -95,7 +114,31 @@ def test_read_data_format_2(self):
                                         atol=1e-7))
         # test seg2 specific header values
         # (trace headers include SEG2 file header)
-        self.assertEqual(st[0].stats.seg2, TRACE1_HEADER)
+        self.assertEqual(st[0].stats.seg2, TRACE2_HEADER)
+
+    def test_read_data_format_3(self):
+        """
+        Test reading a SEG2 data format code 3 file (20-bit floating point).
+        """
+        basename = os.path.join(self.path,
+                                '20180307_031245000.0')
+        # read SEG2 data (in counts, int32)
+        with warnings.catch_warnings(record=True) as w:
+            warnings.simplefilter('always')
+            st = read(basename + ".seg2")
+        self.assertEqual(len(w), 2)
+        self.assertIn('custom', str(w[1]))
+        # read reference ASCII data
+        with gzip.open(basename + '.DAT.gz', 'rb') as f:
+            results = np.loadtxt(f, ndmin=2).T
+        for tr, result in zip(st, results):
+            # convert raw data to unit'd
+            scaled_data = tr.data * tr.stats.calib
+            np.testing.assert_allclose(scaled_data, result,
+                                       rtol=1e-7, atol=1e-7)
+        # test seg2 specific header values
+        # (trace headers include SEG2 file header)
+        self.assertEqual(st[0].stats.seg2, TRACE3_HEADER)
 
 
 def suite():