Repository: obspy/obspy

Issue:
RT130 data encoding '32' not implemented
**GitHub should mainly be used for bug reports and code developments/discussions. For generic questions, please use either the [[obspy-users] mailing list](http://lists.swapbytes.de/mailman/listinfo/obspy-users) (for scientific questions to the general ObsPy community) or our [gitter chat](https://gitter.im/obspy/obspy) (for technical questions and direct contact to the developers).**

---

Before submitting an Issue, please review the [Issue Guidelines](https://github.com/obspy/obspy/blob/master/CONTRIBUTING.md#submitting-an-issue).

* Please check whether the bug was already reported or fixed.
* Please provide the following information:
  -  ObsPy version, Python version and Platform (Windows, OSX, Linux ...)

obspy 1.1.0
Python 3.6
macOS 10.13.1 High Sierra

  -  How did you install ObsPy and Python (pip, anaconda, from source ...)

Anaconda

  -  If possible please supply a short, self contained, correct example that
     demonstrates the issue.

    >>> from obspy import read
    >>> import obspy
    >>> a = read("/Volumes/RT130-R2D2/2018019/R2D2/1/210000005_0036EE80")

Following Error Received:

    Traceback (most recent call last):
      File "<pyshell#13>", line 1, in <module>
        a = read("/Volumes/RT130-R2D2/2018019/R2D2/1/210000005_0036EE80")
      File "<decorator-gen-31>", line 2, in read
      File "/Users/jau827/anaconda/envs/py36/lib/python3.6/site-
    packages/obspy/core/util/decorator.py", line 301, in _map_example_filename
         return func(*args, **kwargs)
      File "/Users/jau827/anaconda/envs/py36/lib/python3.6/site-packages/obspy/core/stream.py", 
    line 235, in read
        st.extend(_read(file, format, headonly, **kwargs).traces)
      File "<decorator-gen-32>", line 2, in _read
      File "/Users/jau827/anaconda/envs/py36/lib/python3.6/site-
    packages/obspy/core/util/decorator.py", line 211, in uncompress_file
        result = func(filename, *args, **kwargs)
      File "/Users/jau827/anaconda/envs/py36/lib/python3.6/site-packages/obspy/core/stream.py", 
    line 277, in _read
        headonly=headonly, **kwargs)
      File "/Users/jau827/anaconda/envs/py36/lib/python3.6/site-
    packages/obspy/core/util/base.py", line 412, in _read_from_plugin
         list_obj = read_format(filename, **kwargs)
       File "/Users/jau827/anaconda/envs/py36/lib/python3.6/site-
    packages/obspy/io/reftek/core.py", line 102, in _read_reftek130
        verbose=verbose)
      File "/Users/jau827/anaconda/envs/py36/lib/python3.6/site-
    packages/obspy/io/reftek/core.py", line 252, in to_stream
        raise NotImplementedError(msg)
    NotImplementedError: Reftek data encoding 'b'32'' not implemented yet. Please open an issue 
    on GitHub and provide a small (< 50kb) test file.

  -  If this is a regression (used to work in an earlier version of ObsPy),
     please note when it used to work.





Discussion:
Looks like this encoding is uncompressed data, so should be easy to add. Like the message states, please provide a small test file. ;-)

If you don't have a small file, we can just cut off later blocks from a bigger file, at least if the file has an event header packet at the start, like usual.
@megies Please see attached file, thank you!

[230000005_0036EE80 copy.zip](https://github.com/obspy/obspy/files/1656136/230000005_0036EE80.copy.zip)

@jau827 is there some recognizable signal in this data? do you happen to have it in a different file format?
@megies Unfortunately not, this was taken from a Reftek RT130 so it is in the reftek format. Which type of file format would help you? I could try to convert it. Were you able to read it in at all?

nevermind.. converted it with an old conversion tool I had lying around

Resolving patch:
diff --git a/obspy/io/reftek/__init__.py b/obspy/io/reftek/__init__.py
--- a/obspy/io/reftek/__init__.py
+++ b/obspy/io/reftek/__init__.py
@@ -21,9 +21,9 @@
 be supplied to :func:`~obspy.core.stream.read`.
 Currently, only event header/trailer (EH/ET) and data packets (DT) are
 implemented and any other packets will be ignored (a warning is shown if any
-other packets are encountered during reading). So far, only data encoding "C0"
-(STEIM 1 compressed data) is implemented due to the lack of test data in other
-encodings.
+other packets are encountered during reading). So far, only data encodings
+"C0", "C2", "16" and "32" are implemented (due to the lack of test data in
+other encodings).
 
 Reading
 -------
diff --git a/obspy/io/reftek/core.py b/obspy/io/reftek/core.py
--- a/obspy/io/reftek/core.py
+++ b/obspy/io/reftek/core.py
@@ -254,11 +254,19 @@ def to_stream(self, network="", location="", component_codes=None,
                 eh = EHPacket(eh_packets[0])
             else:
                 eh = EHPacket(et_packets[0])
-            # only "C0" encoding supported right now
+            # only C0, C2, 16, 32 encodings supported right now
             if eh.data_format == b"C0":
                 encoding = 'C0'
             elif eh.data_format == b"C2":
                 encoding = 'C2'
+            elif eh.data_format == b"16":
+                msg = ("Reftek130 encoding '16' is implemented but untested, "
+                       "please provide example data for testing at "
+                       "https://github.com/obspy/obspy/issues/new.")
+                warnings.warn(msg)
+                encoding = '16'
+            elif eh.data_format == b"32":
+                encoding = '32'
             else:
                 msg = ("Reftek data encoding '{}' not implemented yet. Please "
                        "open an issue on GitHub and provide a small (< 50kb) "
@@ -305,7 +313,19 @@ def to_stream(self, network="", location="", component_codes=None,
                         sample_data = np.array([], dtype=np.int32)
                         npts = packets_["number_of_samples"].sum()
                     else:
-                        sample_data = _unpack_C0_C2_data(packets_, encoding)
+                        if encoding in ('C0', 'C2'):
+                            sample_data = _unpack_C0_C2_data(packets_,
+                                                             encoding)
+                        elif encoding in ('16', '32'):
+                            dtype = {'16': np.int16, '32': np.int32}[encoding]
+                            # just fix endianness and use correct dtype
+                            sample_data = np.require(
+                                packets_['payload'],
+                                requirements=['C_CONTIGUOUS'])
+                            # either int16 or int32
+                            sample_data = sample_data.flatten().view(dtype)
+                            # switch endianness, rt130 stores in big endian
+                            sample_data = sample_data.byteswap()
                         npts = len(sample_data)
 
                     tr = Trace(data=sample_data, header=copy.deepcopy(header))


Test patch:
diff --git a/obspy/io/reftek/tests/data/230000005_0036EE80_cropped.npz b/obspy/io/reftek/tests/data/230000005_0036EE80_cropped.npz
new file mode 100644
Binary files /dev/null and b/obspy/io/reftek/tests/data/230000005_0036EE80_cropped.npz differ
diff --git a/obspy/io/reftek/tests/data/230000005_0036EE80_cropped.rt130 b/obspy/io/reftek/tests/data/230000005_0036EE80_cropped.rt130
new file mode 100644
Binary files /dev/null and b/obspy/io/reftek/tests/data/230000005_0036EE80_cropped.rt130 differ
diff --git a/obspy/io/reftek/tests/test_core.py b/obspy/io/reftek/tests/test_core.py
--- a/obspy/io/reftek/tests/test_core.py
+++ b/obspy/io/reftek/tests/test_core.py
@@ -44,6 +44,10 @@ def setUp(self):
         self.reftek_file = os.path.join(self.datapath, self.reftek_filename)
         self.reftek_file_steim2 = os.path.join(self.datapath,
                                                '104800000_000093F8')
+        self.reftek_file_32 = os.path.join(
+            self.datapath, '230000005_0036EE80_cropped.rt130')
+        self.reftek_file_32_npz = os.path.join(
+            self.datapath, '230000005_0036EE80_cropped.npz')
         self.mseed_filenames = [
             "2015282_225051_0ae4c_1_1.msd",
             "2015282_225051_0ae4c_1_2.msd", "2015282_225051_0ae4c_1_3.msd"]
@@ -554,6 +558,23 @@ def test_reading_file_with_multiple_events(self):
         for tr in st[8:]:
             self.assertEqual(tr.stats.station, 'TL02')
 
+    def test_reading_file_with_encoding_32(self):
+        """
+        Test reading a file with encoding '32' (uncompressed 32 bit integer)
+
+        Only tests unpacked sample data, everything else should be covered by
+        existing tests.
+        """
+        with warnings.catch_warnings():
+            warnings.simplefilter("ignore")
+            st = _read_reftek130(self.reftek_file_32)
+        # read expected data
+        npz = np.load(self.reftek_file_32_npz)
+        # compare it
+        self.assertEqual(len(st), 3)
+        for tr, (_, expected) in zip(st, sorted(npz.items())):
+            np.testing.assert_array_equal(expected, tr.data)
+
 
 def suite():
     return unittest.makeSuite(ReftekTestCase, "test")