Repository: obspy/obspy

Issue:
https FDSN web service end points not working
When checking the `InsecureRequestWarning` in #1779, I noticed that those FDSNWS end points we have that go to https addresses don't seem to work..

```python
In [1]: from obspy.clients.fdsn import Client

In [2]: client = Client('GFZ')
---------------------------------------------------------------------------
FDSNException                             Traceback (most recent call last)
<ipython-input-2-1d7f32e7dd73> in <module>()
----> 1 client = Client('GFZ')

/home/megies/git/obspy-master/obspy/clients/fdsn/client.pyc in __init__(self, base_url, major_versions, user, password, user_agent, debug, timeout, service_mappings, force_redirect)
    260             print("Request Headers: %s" % str(self.request_headers))
    261 
--> 262         self._discover_services()
    263 
    264     def get_events(self, starttime=None, endtime=None, minlatitude=None,

/home/megies/git/obspy-master/obspy/clients/fdsn/client.pyc in _discover_services(self)
   1479                    "be due to a temporary service outage or an invalid FDSN "
   1480                    "service address." % self.base_url)
-> 1481             raise FDSNException(msg)
   1482 
   1483         # Cache.

FDSNException: No FDSN services could be discovered at 'https://geofon.gfz-potsdam.de'. This could be due to a temporary service outage or an invalid FDSN service address.

In [3]: client = Client('SCEDC')
---------------------------------------------------------------------------
FDSNException                             Traceback (most recent call last)
<ipython-input-3-7c19ea139f48> in <module>()
----> 1 client = Client('SCEDC')

/home/megies/git/obspy-master/obspy/clients/fdsn/client.pyc in __init__(self, base_url, major_versions, user, password, user_agent, debug, timeout, service_mappings, force_redirect)
    260             print("Request Headers: %s" % str(self.request_headers))
    261 
--> 262         self._discover_services()
    263 
    264     def get_events(self, starttime=None, endtime=None, minlatitude=None,

/home/megies/git/obspy-master/obspy/clients/fdsn/client.pyc in _discover_services(self)
   1479                    "be due to a temporary service outage or an invalid FDSN "
   1480                    "service address." % self.base_url)
-> 1481             raise FDSNException(msg)
   1482 
   1483         # Cache.

FDSNException: No FDSN services could be discovered at 'https://service.scedc.caltech.edu'. This could be due to a temporary service outage or an invalid FDSN service address.

In [4]: client = Client('USGS')
---------------------------------------------------------------------------
FDSNException                             Traceback (most recent call last)
<ipython-input-4-2d8335176e68> in <module>()
----> 1 client = Client('USGS')

/home/megies/git/obspy-master/obspy/clients/fdsn/client.pyc in __init__(self, base_url, major_versions, user, password, user_agent, debug, timeout, service_mappings, force_redirect)
    260             print("Request Headers: %s" % str(self.request_headers))
    261 
--> 262         self._discover_services()
    263 
    264     def get_events(self, starttime=None, endtime=None, minlatitude=None,

/home/megies/git/obspy-master/obspy/clients/fdsn/client.pyc in _discover_services(self)
   1479                    "be due to a temporary service outage or an invalid FDSN "
   1480                    "service address." % self.base_url)
-> 1481             raise FDSNException(msg)
   1482 
   1483         # Cache.

FDSNException: No FDSN services could be discovered at 'https://earthquake.usgs.gov'. This could be due to a temporary service outage or an invalid FDSN service address.
```


Discussion:
Also there's a new end points at https://www.fdsn.org/webservices/datacenters/

 - ICGC: http://ws.icgc.cat/fdsnws/
Is this still a problem? I just tried and I can access all HTTPS services just fine.
Hmm.. strange, can't reproduce right now. So I guess it works now..? But really strange because hard to believe it was simultaneously fixed on three servers??
yea - maybe it was some kind of certificate issue? next time you encounter this: can you run with `debug=True`?

Maybe we should also revert all default services to use HTTP? Lots of Python installations have outdated crypto libraries and https will not work I guess. But I'm no expert in these manners. Opinions? @barsch @QuLogic @markcwill
@krischer @megies in my experience it's usually a cert issue. We only see crypto issues on really old libs like embedded stuff in routers & radios, or unless someone's trying to do something fancy like restrict what algos can be used. It could have been something weird like they all expired at the same time, but unlikely, since it looks like the 3 services in this thread (USGS, Caltech, etc) are signed by different root authorities.

@megies, related to #1779, my quick-and-dirty test is to do a handshake, e.g.
```shell
$ openssl s_client -connect "earthquake.usgs.gov:443" -showcerts -servername "earthquake.usgs.gov"
```
That one looks to be good for me right now. To contrast, it looks as though the service at IRIS is not set up properly, e.g. maybe not sending the full chain of certs (can't verify with intermediate certs), so the handshake is failing for me right now.
```shell
$ openssl s_client -connect "service.iris.edu:443" -showcerts -servername "service.iris.edu"
```
I think I agree with @krischer there should be a policy about this, either default to http or use https only if signed off by the datacenter and/or the http endpoint doesn't really exist (like if 80 automatically redirects to 443).
> That one looks to be good for me right now. To contrast, it looks as though the service at IRIS is not set up properly, e.g. maybe not sending the full chain of certs (can't verify with intermediate certs), so the handshake is failing for me right now.
> 
> $ openssl s_client -connect "service.iris.edu:443" -showcerts -servername "service

@chad-iris 
> I think I agree with @krischer there should be a policy about this, either default to http or use https only if signed off by the datacenter and/or the http endpoint doesn't really exist (like if 80 automatically redirects to 443).

Right now I would tend to default to HTTP. Another person also had problems connecting to the HTTPS services using ObsPy from azure cloud. Also in most cases the data is not sensitive and people could still choose to manually select HTTPS.
> That one looks to be good for me right now. To contrast, it looks as though the service at IRIS is not set up properly, e.g. maybe not sending the full chain of certs (can't verify with intermediate certs), so the handshake is failing for me right now.
>`$ openssl s_client -connect "service.iris.edu:443" -showcerts -servername "service.iris.edu"`

I've asked our admin to investigate.  Looks like that certificate is about to expire anyway so we'll probably just get a new one.

> That one looks to be good for me right now. To contrast, it looks as though the service at IRIS is not set up properly, e.g. maybe not sending the full chain of certs (can't verify with intermediate certs), so the handshake is failing for me right now.
>`$ openssl s_client -connect "service.iris.edu:443" -showcerts -servername "service.iris.edu"`

The certs for *.iris.edu are now updated and should be up to modern snuff.  Thanks for the nudge.

Well, I guess we can close this. If there's problems in the future, users can always use the explicit URL to the http endpoint anyway.
I'm reopening this as a reminder to change back to HTTP. I think we want this as just too many python installations have outdated crypto libraries/certificates and I don't see a big advantage for using HTTPS for waveform data.

Using HTTP should also not be an issue with authenticated queries as HTTP digest auth does encrypt the credentials.
Technically, FDSN specifications impose that services are available on `:80`. Defaulting to HTTP seems the best choice for now IMHO.
How about having all short URL mappings go to HTTP at port 80 and optionally we could add a `kwarg` `ensure_https=False` that raises on any attempted connection to HTTP? That could help people paranoid about their credentials to ensure they stay super safe? (But I guess thats just my paranoia coming through and most people wont care, most likely)

In any case, I'm not the HTTP/HTTPS security expert, and if HTTP digest is secure enough then this additional check is probably not necessary.

Resolving patch:
diff --git a/obspy/clients/fdsn/header.py b/obspy/clients/fdsn/header.py
--- a/obspy/clients/fdsn/header.py
+++ b/obspy/clients/fdsn/header.py
@@ -43,7 +43,7 @@ class FDSNNoDataException(FDSNException):
     "ETH": "http://eida.ethz.ch",
     "EMSC": "http://www.seismicportal.eu",
     "GEONET": "http://service.geonet.org.nz",
-    "GFZ": "https://geofon.gfz-potsdam.de",
+    "GFZ": "http://geofon.gfz-potsdam.de",
     "ICGC": "http://ws.icgc.cat",
     "INGV": "http://webservices.rm.ingv.it",
     "IPGP": "http://eida.ipgp.fr",
@@ -57,9 +57,9 @@ class FDSNNoDataException(FDSNException):
     "ODC": "http://www.orfeus-eu.org",
     "ORFEUS": "http://www.orfeus-eu.org",
     "RESIF": "http://ws.resif.fr",
-    "SCEDC": "https://service.scedc.caltech.edu",
+    "SCEDC": "http://service.scedc.caltech.edu",
     "TEXNET": "http://rtserve.beg.utexas.edu",
-    "USGS": "https://earthquake.usgs.gov",
+    "USGS": "http://earthquake.usgs.gov",
     "USP": "http://sismo.iag.usp.br"}
 
 FDSNWS = ("dataselect", "event", "station")


Test patch: