Repository: obspy/obspy

Issue:
Convert byte order to native on reading data or merging traces?
### Problem

First posted on the forum: https://discourse.obspy.org/t/the-big-endian-and-little-endian-confilict-for-merging-seismogram/1890/3

Currently, we raise an exception when attempting to merge traces that would otherwise be mergeable but only differ on the byteorder of the underlying data array, e.g. coming from different files with data in differing byteorder.
Probably we should fix the byte order and enable merging instead.

```python
import numpy as np                                                              
from obspy import read                                                          
                                                                                
                                                                                
# make different endian sacfile                                                 
tr = read()[0]                                                                  
starttime = tr.stats.starttime                                                  
                                                                                
first_part = tr.copy()                                                          
first_part = first_part.trim(starttime, starttime + 5)                          
first_part.write("/tmp/first.sac", format="SAC", byteorder="<")                 
                                                                                
second_part = tr.copy()                                                         
second_part = second_part.trim(starttime + 5, starttime + 10)                   
second_part.write("/tmp/second.sac", format="SAC", byteorder=">")               
                                                                                
# merge them                                                                    
st = read("/tmp/first.sac")                                                     
st += read("/tmp/second.sac")                                                   
st.plot(automerge=False)                                                        
                                                                                
try:                                                                            
    st.merge()                                                                  
except Exception as e:                                                          
    print(str(e))                                                               
                                                                                
for tr in st:                                                                   
    dtype = tr.data.dtype.newbyteorder('native')                                
    tr.data = np.require(tr.data, dtype=dtype)                                  
                                                                                
st.merge()                                                                      
st.plot()                                                                       
```

### Proposed solution

Ideally we should probably simply convert byteorder to native on reading any data, but then we might also have to avoid keeping the original byteorder around in metadata (like we do when reading MiniSEED) or we might run into weird issues further down the road (e.g. when writing data again) and there is a mismatch between the data byteorder in the current object and the target byteorder when writing data out to file.

A more conservative approach might be to only convert byte order to native when actually performing operations that work on combining multiple traces, so probably only when merging (aka calling `Trace.__(i)add__`)

Anybody have any thoughts on this? Might be easy to overlook some implications when changing things in this regard..


Discussion:
This issue has been mentioned on **ObsPy Forum**. There might be relevant details there:

https://discourse.obspy.org/t/the-big-endian-and-little-endian-confilict-for-merging-seismogram/1890/5


Resolving patch:
diff --git a/obspy/core/stream.py b/obspy/core/stream.py
--- a/obspy/core/stream.py
+++ b/obspy/core/stream.py
@@ -1972,20 +1972,23 @@ def _merge_checks(self):
             # Check sampling rate.
             sr.setdefault(trace.id, trace.stats.sampling_rate)
             if trace.stats.sampling_rate != sr[trace.id]:
-                msg = "Can't merge traces with same ids but differing " + \
-                      "sampling rates!"
+                msg = (f"Can not merge traces with same ids ({trace.id}) but "
+                       f"differing sampling rates ({sr[trace.id]}, "
+                       f"{trace.stats.sampling_rate})!")
                 raise Exception(msg)
             # Check dtype.
             dtype.setdefault(trace.id, trace.data.dtype)
             if trace.data.dtype != dtype[trace.id]:
-                msg = "Can't merge traces with same ids but differing " + \
-                      "data types!"
+                msg = (f"Can not merge traces with same ids ({trace.id}) but "
+                       f"differing data types ({dtype[trace.id]}, "
+                       f"{trace.data.dtype})!")
                 raise Exception(msg)
             # Check calibration factor.
             calib.setdefault(trace.id, trace.stats.calib)
             if trace.stats.calib != calib[trace.id]:
-                msg = "Can't merge traces with same ids but differing " + \
-                      "calibration factors.!"
+                msg = (f"Can not merge traces with same ids ({trace.id}) but "
+                       f"differing calibration factors ({calib[trace.id]}, "
+                       f"{trace.stats.calib})!")
                 raise Exception(msg)
 
     def merge(self, method=0, fill_value=None, interpolation_samples=0,
@@ -3639,6 +3642,17 @@ def _rotate_specific_channels_to_zne(
             self.traces += traces
         return self
 
+    def newbyteorder(self, byteorder='native'):
+        """
+        Change byteorder of the data
+
+        For details see
+        :meth:`Trace.newbyteorder <obspy.core.trace.Trace.newbyteorder>`.
+        """
+        for tr in self:
+            tr.newbyteorder(byteorder)
+        return self
+
 
 def _is_pickle(filename):  # @UnusedVariable
     """
diff --git a/obspy/core/trace.py b/obspy/core/trace.py
--- a/obspy/core/trace.py
+++ b/obspy/core/trace.py
@@ -3026,6 +3026,19 @@ def remove_sensitivity(self, inventory=None):
         self.data = self.data / response.instrument_sensitivity.value
         return self
 
+    def newbyteorder(self, byteorder='native'):
+        """
+        Change byteorder of the data
+
+        :type byteorder: str
+        :param byteorder: Byte order to set on the numpy data array, e.g.
+            ``'native'``, ``'little'`` or ``'big'``. See
+            :meth:`numpy.dtype.newbyteorder`.
+        """
+        dtype = self.data.dtype.newbyteorder(byteorder)
+        self.data = np.require(self.data, dtype=dtype)
+        return self
+
 
 def _data_sanity_checks(value):
     """


Test patch:
diff --git a/obspy/core/tests/test_stream.py b/obspy/core/tests/test_stream.py
--- a/obspy/core/tests/test_stream.py
+++ b/obspy/core/tests/test_stream.py
@@ -2,6 +2,7 @@
 import io
 import pickle
 import platform
+import re
 import warnings
 from copy import deepcopy
 from unittest import mock
@@ -2862,3 +2863,68 @@ def test_filter_freq_args(self):
             stf1 = st.copy().filter(filtert, freqmin=freqmin, freqmax=freqmax)
             stf2 = st.copy().filter(filtert, freqmin, freqmax)
             assert streams_almost_equal(stf2, stf1)
+
+    def test_merge_checks_messages(self, mseed_stream):
+        """
+        Test improved error messages
+        """
+        # dtype
+        st = mseed_stream.copy()
+        st[0].data = np.require(
+            st[0].data, dtype=st[0].data.dtype.newbyteorder('<'))
+        st[1].data = np.require(
+            st[1].data, dtype=st[1].data.dtype.newbyteorder('>'))
+        msg = ('Can not merge traces with same ids (BW.BGLD..EHE) but '
+               'differing data types (float64, >f8)!')
+        with pytest.raises(Exception, match=re.escape(msg)):
+            st.merge()
+        # sampling rate
+        st = mseed_stream.copy()
+        st[0].stats.sampling_rate = 200
+        st[1].stats.sampling_rate = 20
+        msg = ('Can not merge traces with same ids (BW.BGLD..EHE) but '
+               'differing sampling rates (200.0, 20.0)!')
+        with pytest.raises(Exception, match=re.escape(msg)):
+            st.merge()
+        # calib
+        st = mseed_stream.copy()
+        st[0].stats.calib = 12345
+        st[1].stats.calib = 100
+        msg = ('Can not merge traces with same ids (BW.BGLD..EHE) but '
+               'differing calibration factors (12345, 100)!')
+        with pytest.raises(Exception, match=re.escape(msg)):
+            st.merge()
+
+    def test_newbyteorder(self, mseed_stream):
+        """
+        Test switching byteorder on trace/stream, for simplicity both things
+        tested here
+        """
+        msg = ('Can not merge traces with same ids (BW.BGLD..EHE) but '
+               'differing data types (float64, >f8)!')
+        # set different byteorders
+        st = mseed_stream
+        st[0].data = np.require(
+            st[0].data, dtype=st[0].data.dtype.newbyteorder('<'))
+        st[1].data = np.require(
+            st[1].data, dtype=st[1].data.dtype.newbyteorder('>'))
+        # check that we set up correctly this should fail now
+        assert st[0].data.dtype == np.dtype('float64')
+        assert st[1].data.dtype == np.dtype('>f8')
+        with pytest.raises(Exception, match=re.escape(msg)):
+            st.merge()
+        # check working on trace
+        st = mseed_stream.copy()
+        assert st[0].data.dtype == np.dtype('float64')
+        assert st[1].data.dtype == np.dtype('>f8')
+        st[1].newbyteorder('<')
+        assert st[1].data.dtype == np.dtype('float64')
+        st.merge()
+        # check working on stream
+        st = mseed_stream.copy()
+        assert st[0].data.dtype == np.dtype('float64')
+        assert st[1].data.dtype == np.dtype('>f8')
+        st.newbyteorder('=')
+        for tr in st:
+            assert tr.data.dtype == np.dtype('float64')
+        st.merge()