Repository: obspy/obspy

Issue:
FDSN client string decoding glitch 
### Avoid duplicates

- [X] I searched existing issues

### Bug Summary

Seems like the current version of fdsn/client.py not handling (some?) server response properly. In particular, when attempting to access restricted data with an incorrect username or password.

Doesn't like trying to decode a string-- not sure what version of python this started happening with, is this possibly a very very very old bug? Obspy is strictly Python 3 now yes?

### Code to Reproduce

```python
from obspy.clients.fdsn import Client
ap = Client('AUSPASS',user='nope',password='password')
st = ap.get_waveforms(station='AKL10',channel='BHZ',location='*',network='X5',starttime=obspy.UTCDateTime(2023,5,20,1,52),endtime=obspy.UTCDateTime(2023,5,20,2,2))
```


### Error Traceback

```Python traceback
File ~/.local/lib/python3.11/site-packages/obspy/clients/fdsn/client.py:872, in Client.get_waveforms(self, network, station, location, channel, starttime, endtime, quality, minimumlength, longestonly, filename, attach_response, **kwargs)
    867 url = self._create_url_from_parameters(
    868     "dataselect", DEFAULT_PARAMETERS['dataselect'], kwargs)
    870 # Gzip not worth it for MiniSEED and most likely disabled for this
    871 # route in any case.
--> 872 data_stream = self._download(url, use_gzip=False)
    873 data_stream.seek(0, 0)
    874 if filename:

File ~/.local/lib/python3.11/site-packages/obspy/clients/fdsn/client.py:1486, in Client._download(self, url, return_string, data, use_gzip, content_type)
   1481     headers['Content-Type'] = content_type
   1482 code, data = download_url(
   1483     url, opener=self._url_opener, headers=headers,
   1484     debug=self.debug, return_string=return_string, data=data,
   1485     timeout=self.timeout, use_gzip=use_gzip)
-> 1486 raise_on_error(code, data)
   1487 return data

File ~/.local/lib/python3.11/site-packages/obspy/clients/fdsn/client.py:1807, in raise_on_error(code, data)
   1805     server_info = None
   1806 else:
-> 1807     server_info = server_info.decode('ASCII', errors='ignore')
   1808 if server_info:
   1809     server_info = "\n".join(
   1810         line for line in server_info.splitlines() if line)

AttributeError: 'str' object has no attribute 'decode'
```


### ObsPy Version?

1.4.0

### Operating System?

Debian 12

### Python Version?

3.11

### Installation Method?

pip


Discussion:
Hmm.. I tried to reproduce but I only get the following which seems fine.. I tried Python 3.10 and 3.11 with obspy 1.4.0, maintenance branch and master branch.

```
Traceback (most recent call last):
  File "/tmp/bug.py", line 4, in <module>
    st = ap.get_waveforms(station='AKL10',channel='BHZ',location='*',network='X5',starttime=obspy.UTCDateTime(2023,5,20,1,52),endtime=obspy.UTCDateTime(2023,5,20,2,2))
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/megies/miniconda3/envs/trash/lib/python3.11/site-packages/obspy/clients/fdsn/client.py", line 872, in get_waveforms
    data_stream = self._download(url, use_gzip=False)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/megies/miniconda3/envs/trash/lib/python3.11/site-packages/obspy/clients/fdsn/client.py", line 1486, in _download
    raise_on_error(code, data)
  File "/home/megies/miniconda3/envs/trash/lib/python3.11/site-packages/obspy/clients/fdsn/client.py", line 1820, in raise_on_error
    raise FDSNUnauthorizedException("Unauthorized, authentication "
obspy.clients.fdsn.header.FDSNUnauthorizedException: Unauthorized, authentication required.
HTTP Status code: 401
Detailed response of server:


```
You're right, I tried on other machines (python3.11.4,  python3.8) and it seems to work normally.

The problem PC is a pretty brand new Debian 12 machine (but still at py 3.11.2) so will guess something in `http.client.HTTPException` or  `urllib` started decoding text automatically. Edit: thought it may have been IPython, but same in the normal python interpreter. 

Pretty low priority...
Let us know if you find out, if it has something to do with newer version of package XYZ we will have to work it out eventually anyway.
I just ran into most probably related issue with current master in Python 3.10 (Ubuntu 22.04.3 LTS):

```
import obspy.core
import obspy.clients.fdsn

client = obspy.clients.fdsn.Client("GFZ")

t1 = obspy.core.UTCDateTime(2010, 1, 1, 0, 0, 0)
t2 = obspy.core.UTCDateTime(2024, 1, 1, 0, 0, 0)

inventory = client.get_stations(
        network="NL", station="*", location="*", channel="*",
        starttime=t1, endtime=t2, level="channel")
```

In other words, I am requesting from GFZ an inventory for network NL that GFZ does _not_ have. What I get is a rather cryptic (in this context) exception:

```
Traceback (most recent call last):
  File "/tmp/test.py", line 9, in <module>
    inventory = client.get_stations(
  File "/usr/local/lib/python3.10/dist-packages/obspy/clients/fdsn/client.py", line 748, in get_stations
    data_stream = self._download(url)
  File "/usr/local/lib/python3.10/dist-packages/obspy/clients/fdsn/client.py", line 1486, in _download
    raise_on_error(code, data)
  File "/usr/local/lib/python3.10/dist-packages/obspy/clients/fdsn/client.py", line 1802, in raise_on_error
    server_info = data.decode('ASCII', errors='ignore')
AttributeError: '_io.BytesIO' object has no attribute 'decode'
```
This is certainly not the intended behavior.
When requesting the inventory from KNMI it works with the dates as above. By using dates very long ago, I can again get it to raise the same exception. Looks like it is related to empty inventory.
Regression probably caused by this commit https://github.com/obspy/obspy/commit/394387090933cc3b185be9c5e3efd552f46ce615. Does it break on all non `200` HTTP codes? The previous `try` `except` `else` works as expected.
Getting the same error on various FDSN servers, again Debian 12 / 3.11.2

Ubuntu 20.04.6 works with 3.11.4 and 3.8.10.
Just tried with Python 3.11.4 on an Ubuntu 23.04 box and the error is the same there.

It's the `data.read()` that's missing there if `data` is a `BytesIO`. Which doesn't always seem to be the case as otherwise the code would always break AFAICS, as `BytesIO` has no `decode()` method.

As a workaround, what works for me is to replace 
```
        server_info = data.decode('ASCII', errors='ignore')
```
by
```
        try:
            server_info = data.decode('ASCII', errors='ignore')
        except AttributeError:
            server_info = data.read().decode('ASCII', errors='ignore')
```
at https://github.com/obspy/obspy/blob/master/obspy/clients/fdsn/client.py#L1802
> Regression probably caused by this commit [3943870](https://github.com/obspy/obspy/commit/394387090933cc3b185be9c5e3efd552f46ce615)

Agreed. We must have some sloppiness in `fdsn.client.download_url()` resulting in varying return types bytes/string.. or that try/except that got removed and is now causing issues shouldn't even be needed..

This seems to be the culprit for some internal confusion here.. a `kwarg` `return_string` (sic!) determining whether the return type is `bytes` or `io.BytesIO` seems quite redundant and even causing trouble and confusion.

https://github.com/obspy/obspy/blob/417b047a2c13491232a3b472a9cf3c0a78785f22/obspy/clients/fdsn/client.py#L1919-L1922

I still feel the original post was about something else, since there the return type was **string** even, hence the `decode()` failing. I still can't find a reproducible case that is returning `str` there though.
I think for 1.4.1 on maintenance branch we should probably revert the first part of #3306 and then for 1.5.0 we should get rid of the `return_string` switch.

Resolving patch:
diff --git a/obspy/clients/fdsn/client.py b/obspy/clients/fdsn/client.py
--- a/obspy/clients/fdsn/client.py
+++ b/obspy/clients/fdsn/client.py
@@ -1799,7 +1799,21 @@ def raise_on_error(code, data):
     """
     # get detailed server response message
     if code != 200:
-        server_info = data.decode('ASCII', errors='ignore')
+        # let's try to resolve all the different types that `data` can sadly
+        # have..
+        # first, if it's `BytesIO` (or should it for whatever reason be
+        # `StringIO` which it shouldn't..) then break it down by reading it
+        try:
+            server_info = data.read()
+        # if there is no `read()` method it should be `bytes`
+        except AttributeError:
+            server_info = data
+        # now decode the bytes (or if for whatever weird reason we ended up
+        # with a string, then do nothing more)
+        try:
+            server_info = server_info.decode('ASCII', errors='ignore')
+        except AttributeError:
+            pass
         if server_info:
             server_info = "\n".join(
                 line for line in server_info.splitlines() if line)
@@ -1885,10 +1899,34 @@ def download_url(url, opener, timeout=10, headers={}, debug=False,
         url_obj = opener.open(request, timeout=timeout, data=data)
     # Catch HTTP errors.
     except urllib_request.HTTPError as e:
-        error_data = e.read()
+        # try hard to assemble the most details on what the problem is from the
+        # exception object
+        try:
+            error_msg = e.msg
+        except AttributeError:
+            error_msg = None
+        try:
+            error_details = e.read()
+        except Exception:
+            error_details = None
+        if error_details:
+            # looks like we get bytes back so let's decode but be robust just
+            # in case
+            try:
+                error_details = error_details.decode('UTF-8', errors='ignore')
+            except AttributeError:
+                pass
+        if error_msg and error_details:
+            error_data = f'{error_msg}\n{error_details}'
+        elif error_msg:
+            error_data = error_msg
+        elif error_details:
+            error_data = error_details
+        else:
+            error_data = None
         if debug is True:
             msg = "HTTP error %i, reason %s, while downloading '%s': %s" % \
-                  (e.code, str(e.reason), url, error_data)
+                  (e.code, str(e.reason), url, error_data or '')
             print(msg)
         return e.code, error_data
     except Exception as e:


Test patch: