Repository: obspy/obspy

Issue:
obspy.core.trace.Trace.trim(pad=..., fill_value=...) should provide a <type 'numpy.ndarray'> object
Hey,
it looks like there is a minor issue in the case method trim() is used with a particular set of parameters, that is if padding is requested (pad=True) and a fill_value is specified as well.

Background:
- Default behaviour of trim() is to limit the resulting traces to existing data, and this works fine;
- In case padding is requested, the resulting data window is extended, a masked array is returned and non existing samples are masked; (as well the expected and documented behaviour);

From documentation and intuition, the "fill_value" parameter would serve to avoid creation of a masked array in the case padding is asked (2nd case). However, the behaviour is slightly different and a <class 'numpy.ma.core.MaskedArray'> object is still returned, but mask is set to True. See also below ...

``` python
In [31]: print tr
IV.MMN..HHZ | 2012-01-01T00:00:00.000000Z - 2012-01-01T20:50:58.990000Z | 100.0 Hz, 7505900 samples

In [32]: tr.trim(starttime=obspy.UTCDateTime("2012-01-01T20:00:00.000000Z"), endtime=obspy.UTCDateTime("2012-01-01T21:00:00.000000Z"), pad=True, fill_value=-999 )

In [33]: print tr
IV.MMN..HHZ | 2012-01-01T20:00:00.000000Z - 2012-01-01T21:00:00.000000Z | 100.0 Hz, 360001 samples

In [34]: print tr.data
[ -34.  -41. -107. ..., -999. -999. -999.]

In [35]: print type(tr.data)
<class 'numpy.ma.core.MaskedArray'>

In [36]: tr.data
Out[36]:
masked_array(data = [ -34.  -41. -107. ..., -999. -999. -999.],
            mask = False,
      fill_value = 1e+20)
```

I would argue that this "intermediate" behaviour is unexpected and not very useful. Providing a fill_value should prevent the creation of the masked array.



Discussion:
after fixing this locally and thinking a bit more about it I tend to leave the current behaviour (of returning a masked array with given fill value instead of returning a ndarray) in order to stress the point that extra data has been added

if you need to unmask your array just call self.data=self.data.filled()

other opinions?

Well, I probably do not see the point completely. Of cause, it is always possible to do many manual manipulation with the data and that's all the value of using Python, right? (I actually used the operations above as a work-arround).  

But here I would argue differently
- If you offer a optional `fill_value=...` argument, it should to something useful. Otherwise one gets the masked array back and could do all manipulation directly on it. There is no need for the optional `fill_value` argument at all.

```
tr.trim(t+1.0, t+2.5, pad=True)
tr.data=tr.data.filled(fill_value)
```
- From the documentation I believe the intended behaviour was different. For me this implies: If you ask for pading you get a masked array, but if in addition you provide a `fill_value`, the invalid samples are filled with this requested value and you get a simple `numpy.ndarray`. 
  _fill_value (int, float or None, optional):
  Fill value for gaps. Defaults to None. Traces will be converted to NumPy masked arrays if no value is given and gaps are present._
- The fact that data is added needs not to be stressed here, as it is explicitly asked by the user `tr.trim(pad=True, fill_vallue=<myvalue>)`, and if you ask for filling you do not care any more;
- The resulting array (with `pad=True, fill_value=<myvalue>`) is not a full featured masked array anyway;
- It would be particularly useful if applied on streams. Otherwise you need to cycle through you stream and make the corrections and your code becomes ugly.

I would never insist in having the behaviour changed, but I was advised to file a ticket after asking to the mailing list. Anyway, I find the current behaviour quite contra-intuitive and probably at least the documentation needs changing if you decide to leave the current behaviour. 

Note also that the resulting array will not be marked as `masked` anymore when the trace or stream is printed, and   `pad=True, fill_value=<myvalue>` was used for tr.trim().

@petrrr thanks for clarifying - I completely overlooked the "if no value is given" part in the `fill_value` docstring - you also raised many valid points, so I'm going to implement a fix


Resolving patch:
diff --git a/obspy/core/stream.py b/obspy/core/stream.py
--- a/obspy/core/stream.py
+++ b/obspy/core/stream.py
@@ -1438,7 +1438,7 @@ def trim(self, starttime=None, endtime=None, pad=False,
                 # delta is negative!
                 endtime = tr.stats.endtime + delta * tr.stats.delta
         for trace in self.traces:
-            trace.trim(starttime, endtime, pad,
+            trace.trim(starttime, endtime, pad=pad,
                        nearest_sample=nearest_sample, fill_value=fill_value)
         # remove empty traces after trimming
         self.traces = [_i for _i in self.traces if _i.stats.npts]
diff --git a/obspy/core/trace.py b/obspy/core/trace.py
--- a/obspy/core/trace.py
+++ b/obspy/core/trace.py
@@ -978,6 +978,13 @@ def trim(self, starttime=None, endtime=None, pad=False,
         if endtime:
             self._rtrim(endtime, pad, nearest_sample=nearest_sample,
                         fill_value=fill_value)
+        # if pad=True and fill_value is given convert to NumPy ndarray
+        if pad == True and fill_value is not None:
+            try:
+                self.data = self.data.filled()
+            except AttributeError:
+                # numpy.ndarray object has no attribute 'filled' - ignoring
+                pass
 
     def slice(self, starttime=None, endtime=None):
         """


Test patch:
diff --git a/obspy/core/tests/test_stream.py b/obspy/core/tests/test_stream.py
--- a/obspy/core/tests/test_stream.py
+++ b/obspy/core/tests/test_stream.py
@@ -1909,6 +1909,39 @@ def test_taper(self):
             self.assertTrue(st[1].data[i] <= 1.)
             self.assertTrue(st[1].data[i] >= 0.)
 
+    def test_issue540(self):
+        """
+        Trim with pad=True and given fill value should not return a masked
+        NumPy array.
+        """
+        # fill_value = None
+        st = read()
+        self.assertEquals(len(st[0]), 3000)
+        st.trim(starttime=st[0].stats.starttime - 0.01,
+                endtime=st[0].stats.endtime + 0.01, pad=True, fill_value=None)
+        self.assertEquals(len(st[0]), 3002)
+        self.assertTrue(isinstance(st[0].data, np.ma.masked_array))
+        self.assertTrue(st[0].data[0] is np.ma.masked)
+        self.assertTrue(st[0].data[1] is not np.ma.masked)
+        self.assertTrue(st[0].data[-2] is not np.ma.masked)
+        self.assertTrue(st[0].data[-1] is np.ma.masked)
+        # fill_value = 999
+        st = read()
+        self.assertEquals(len(st[1]), 3000)
+        st.trim(starttime=st[1].stats.starttime - 0.01,
+                endtime=st[1].stats.endtime + 0.01, pad=True, fill_value=999)
+        self.assertEquals(len(st[1]), 3002)
+        self.assertFalse(isinstance(st[1].data, np.ma.masked_array))
+        self.assertEquals(st[1].data[0], 999)
+        self.assertEquals(st[1].data[-1], 999)
+        # given fill_value but actually no padding at all
+        st = read()
+        self.assertEquals(len(st[2]), 3000)
+        st.trim(starttime=st[2].stats.starttime, endtime=st[2].stats.endtime,
+                pad=True, fill_value=-999)
+        self.assertEquals(len(st[2]), 3000)
+        self.assertFalse(isinstance(st[2].data, np.ma.masked_array))
+
 
 def suite():
     return unittest.makeSuite(StreamTestCase, 'test')
diff --git a/obspy/core/tests/test_trace.py b/obspy/core/tests/test_trace.py
--- a/obspy/core/tests/test_trace.py
+++ b/obspy/core/tests/test_trace.py
@@ -1,11 +1,11 @@
 # -*- coding: utf-8 -*-
 
 from copy import deepcopy
-import numpy as np
 from numpy.ma import is_masked
 from obspy import UTCDateTime, Trace, read, Stream
-import unittest
 import math
+import numpy as np
+import unittest
 
 
 class TraceTestCase(unittest.TestCase):
@@ -1245,6 +1245,39 @@ def test_split(self):
         self.assertFalse(isinstance(st[0].data, np.ma.masked_array))
         self.assertFalse(isinstance(st[1].data, np.ma.masked_array))
 
+    def test_issue540(self):
+        """
+        Trim with pad=True and given fill value should not return a masked
+        NumPy array.
+        """
+        # fill_value = None
+        tr = read()[0]
+        self.assertEquals(len(tr), 3000)
+        tr.trim(starttime=tr.stats.starttime - 0.01,
+                endtime=tr.stats.endtime + 0.01, pad=True, fill_value=None)
+        self.assertEquals(len(tr), 3002)
+        self.assertTrue(isinstance(tr.data, np.ma.masked_array))
+        self.assertTrue(tr.data[0] is np.ma.masked)
+        self.assertTrue(tr.data[1] is not np.ma.masked)
+        self.assertTrue(tr.data[-2] is not np.ma.masked)
+        self.assertTrue(tr.data[-1] is np.ma.masked)
+        # fill_value = 999
+        tr = read()[0]
+        self.assertEquals(len(tr), 3000)
+        tr.trim(starttime=tr.stats.starttime - 0.01,
+                endtime=tr.stats.endtime + 0.01, pad=True, fill_value=999)
+        self.assertEquals(len(tr), 3002)
+        self.assertFalse(isinstance(tr.data, np.ma.masked_array))
+        self.assertEquals(tr.data[0], 999)
+        self.assertEquals(tr.data[-1], 999)
+        # given fill_value but actually no padding at all
+        tr = read()[0]
+        self.assertEquals(len(tr), 3000)
+        tr.trim(starttime=tr.stats.starttime,
+                endtime=tr.stats.endtime, pad=True, fill_value=-999)
+        self.assertEquals(len(tr), 3000)
+        self.assertFalse(isinstance(tr.data, np.ma.masked_array))
+
 
 def suite():
     return unittest.makeSuite(TraceTestCase, 'test')