Decoding issues on Linux cause Segmentation fault for given sites

Summary.

I have tested the following on my Mac and I get both unicode errors:

For the following:

`r = requests.get('http://xn--w8jujla2a2c4e0lg4403e52qmkrfye.biz', stream=True)`

I get:

`UnicodeError: decoding with 'idna' codec failed (UnicodeError: Invalid character '\x80')`

For:

`r = requests.get('http://xn--meiner-dta.hamburg', stream=True)`

I get:

`UnicodeError: ('IDNA does not round-trip', b'xn--meiner-dta', b'meissner')`

Which I am good with.

But on linux I get:

`Segmentation fault (core dumped)`

Crashing my program.

Note the versions of python are slightly different but I guess the following is true for ubuntu.

https://forums.aws.amazon.com/thread.jspa?threadID=255687

## Expected Result

Expect it to error out like on MacOS.

## System Information

    $ python -m requests.help

On Mac

```
{
  "chardet": {
    "version": "3.0.4"
  },
  "cryptography": {
    "version": ""
  },
  "idna": {
    "version": "2.6"
  },
  "implementation": {
    "name": "CPython",
    "version": "3.6.2"
  },
  "platform": {
    "release": "16.3.0",
    "system": "Darwin"
  },
  "pyOpenSSL": {
    "openssl_version": "",
    "version": null
  },
  "requests": {
    "version": "2.18.4"
  },
  "system_ssl": {
    "version": "100020cf"
  },
  "urllib3": {
    "version": "1.22"
  },
  "using_pyopenssl": false
}
```

On Linux

```
{
  "chardet": {
    "version": "3.0.4"
  },
  "cryptography": {
    "version": ""
  },
  "idna": {
    "version": "2.6"
  },
  "implementation": {
    "name": "CPython",
    "version": "3.5.2"
  },
  "platform": {
    "release": "4.4.0-1032-aws",
    "system": "Linux"
  },
  "pyOpenSSL": {
    "openssl_version": "",
    "version": null
  },
  "requests": {
    "version": "2.18.4"
  },
  "system_ssl": {
    "version": "1000207f"
  },
  "urllib3": {
    "version": "1.22"
  },
  "using_pyopenssl": false
}
```


Hrm. Requests cannot be the cause of this segfault because it's pure-Python code. Can you use GDB or something similar to examine the coredump and check the Python stack to see what code was executing when the segfault occurred?

Lukas, thanks for getting back to me. I tried to run using GDB. I get the following:

`test.py`
```
import requests
print('Hey')
r = requests.get('http://xn--w8jujla2a2c4e0lg4403e52qmkrfye.biz', stream=True)
```

Then
```
(gdb) run test.py
Starting program: /usr/bin/python3 test.py
warning: Error disabling address space randomization: Operation not permitted
Hey
During startup program terminated with signal SIGSEGV, Segmentation fault.
```

I guess there is something wrong here because its not catching anything..

Sorry, we need a Python backtrace. See [this article](https://wiki.python.org/moin/DebuggingWithGdb)

Does this work?

```
(gdb) run test.py
Starting program: /usr/bin/python3 test.py
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Hey

Program received signal SIGSEGV, Segmentation fault.
0x00007ffff673f767 in ?? () from /usr/lib/python3.5/lib-dynload/_ssl.cpython-35m-x86_64-linux-gnu.so
```

```
(gdb) bt
#0  0x00007ffff673f767 in ?? () from /usr/lib/python3.5/lib-dynload/_ssl.cpython-35m-x86_64-linux-gnu.so
#1  0x00007ffff6739231 in ?? () from /usr/lib/python3.5/lib-dynload/_ssl.cpython-35m-x86_64-linux-gnu.so
#2  0x00000000004e9bc7 in PyCFunction_Call ()
#3  0x0000000000524414 in PyEval_EvalFrameEx ()
#4  0x000000000052e12b in PyEval_EvalCodeEx ()
#5  0x00000000004ebdd7 in ?? ()
#6  0x00000000005b7167 in PyObject_Call ()
#7  0x00000000004f413e in ?? ()
#8  0x00000000005b7167 in PyObject_Call ()
#9  0x000000000054d359 in ?? ()
#10 0x000000000055d17c in ?? ()
#11 0x00000000005b7167 in PyObject_Call ()
#12 0x0000000000528d06 in PyEval_EvalFrameEx ()
#13 0x000000000052d2e3 in ?? ()
#14 0x0000000000528eee in PyEval_EvalFrameEx ()
#15 0x000000000052d2e3 in ?? ()
#16 0x0000000000528eee in PyEval_EvalFrameEx ()
#17 0x0000000000528814 in PyEval_EvalFrameEx ()
#18 0x000000000052d82f in ?? ()
#19 0x0000000000529332 in PyEval_EvalFrameEx ()
#20 0x000000000052d2e3 in ?? ()
#21 0x0000000000528eee in PyEval_EvalFrameEx ()
#22 0x000000000052d2e3 in ?? ()
#23 0x0000000000528eee in PyEval_EvalFrameEx ()
#24 0x000000000052e12b in PyEval_EvalCodeEx ()
#25 0x00000000004ebdd7 in ?? ()
#26 0x00000000005b7167 in PyObject_Call ()
#27 0x00000000005262af in PyEval_EvalFrameEx ()
#28 0x000000000052e12b in PyEval_EvalCodeEx ()
#29 0x00000000004ebdd7 in ?? ()
#30 0x00000000005b7167 in PyObject_Call ()
#31 0x00000000005262af in PyEval_EvalFrameEx ()
#32 0x00000000004ed325 in ?? ()
```


Nope, we need the `py-bt`

hmm 

`Undefined command: "py-bt".  Try "help".`

Got it.

Needed:

`sudo apt-get install gdb python3.5-dbg`

```
(gdb) py-bt
Traceback (most recent call first):
  <built-in method _wrap_socket of SSLContext object at remote 0x7ffff4c43f98>
  File "/usr/lib/python3.5/ssl.py", line 745, in __init__
    server_hostname)
  File "/usr/lib/python3.5/ssl.py", line 377, in wrap_socket
    _context=self)
  File "/usr/lib/python3/dist-packages/urllib3/util/ssl_.py", line 305, in ssl_wrap_socket
    return context.wrap_socket(sock, server_hostname=server_hostname)
  File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 252, in connect
    ssl_version=resolved_ssl_version)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 787, in _validate_conn
    conn.connect()
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 346, in _make_request
    self._validate_conn(conn)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 560, in urlopen
    body=body, headers=headers)
  File "/usr/lib/python3/dist-packages/requests/adapters.py", line 376, in send
    timeout=timeout
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 576, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 195, in resolve_redirects
    **adapter_kwargs
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 597, in <listcomp>
    history = [resp for resp in gen] if allow_redirects else []
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 597, in send
    history = [resp for resp in gen] if allow_redirects else []
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 468, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python3/dist-packages/requests/api.py", line 53, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/api.py", line 67, in get
    return request('get', url, params=params, **kwargs)
  File "test.py", line 3, in <module>
```

Ok, this segfault is coming from your interpreter. How did you install it?

You can reproduce this if you use the following Dockerfile

```
FROM ubuntu:latest

RUN apt-get update -y
RUN apt-get install -y python3-pip python3-dev libffi-dev

RUN apt-get install -y vim

COPY requirements.txt /tmp/
RUN pip3 install --requirement /tmp/requirements.txt
```

`requirements.txt`

```
requests==2.18.4
```



I have the same problems (the unicode ones, not the segmentation fault) in Ubuntu 16.10 with Python 3.6.0b2 and aiohttp==2.2.5

This reproduces the problem:
```
import asyncio
import aiohttp


async def process():
    url = 'http://xn--x9dj.com/'
    session = aiohttp.ClientSession()
    async with session.get(url) as response:
        print(response)
    session.close()

loop = asyncio.get_event_loop()
loop.run_until_complete(
    process()
)
loop.close()
```

Traceback:
```
Traceback (most recent call last):
  File "test.py", line 14, in <module>
    process()
  File "/usr/lib/python3.6/asyncio/base_events.py", line 449, in run_until_complete
    return future.result()
  File "/usr/lib/python3.6/asyncio/tasks.py", line 239, in _step
    result = coro.send(None)
  File "test.py", line 8, in process
    async with session.get(url) as response:
  File "/myenv/lib/python3.6/site-packages/aiohttp/client.py", line 603, in __aenter__
    self._resp = yield from self._coro
  File "/myenv/lib/python3.6/site-packages/aiohttp/client.py", line 226, in _request
    session=self)
  File "/myenv/lib/python3.6/site-packages/aiohttp/client_reqrep.py", line 96, in __init__
    self.update_host(url)
  File "/myenv/lib/python3.6/site-packages/aiohttp/client_reqrep.py", line 123, in update_host
    if not url.host:
  File "/myenv/lib/python3.6/site-packages/yarl/__init__.py", line 59, in __get__
    val = self.wrapped(inst)
  File "/myenv/lib/python3.6/site-packages/yarl/__init__.py", line 433, in host
    return raw.encode('ascii').decode('idna')
  File "/myenv/lib/python3.6/encodings/idna.py", line 214, in decode
    result.append(ToUnicode(label))
  File "/myenv/lib/python3.6/encodings/idna.py", line 139, in ToUnicode
    raise UnicodeError("IDNA does not round-trip", label, label2)
UnicodeError: ('IDNA does not round-trip', b'xn--x9dj', b'xn--c09aja')
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7fe5774205f8>
```
if I use this domain instead: 

> xn---1-9ld5cqq.com

Then I get this:
```

Traceback (most recent call last):
  File "/myenv/lib/python3.6/encodings/idna.py", line 214, in decode
    result.append(ToUnicode(label))
  File "/myenv/lib/python3.6/encodings/idna.py", line 134, in ToUnicode
    label2 = ToASCII(result)
  File "/myenv/lib/python3.6/encodings/idna.py", line 76, in ToASCII
    label = nameprep(label)
  File "/myenv/lib/python3.6/encodings/idna.py", line 58, in nameprep
    raise UnicodeError("Violation of BIDI requirement 3")
UnicodeError: Violation of BIDI requirement 3

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "test.py", line 44, in <module>
    process()
  File "/usr/lib/python3.6/asyncio/base_events.py", line 449, in run_until_complete
    return future.result()
  File "/usr/lib/python3.6/asyncio/tasks.py", line 239, in _step
    result = coro.send(None)
  File "test.py", line 12, in process
    async with session.get(url) as response:
  File "/myenv/lib/python3.6/site-packages/aiohttp/client.py", line 603, in __aenter__
    self._resp = yield from self._coro
  File "/myenv/lib/python3.6/site-packages/aiohttp/client.py", line 226, in _request
    session=self)
  File "/myenv/lib/python3.6/site-packages/aiohttp/client_reqrep.py", line 96, in __init__
    self.update_host(url)
  File "/myenv/lib/python3.6/site-packages/aiohttp/client_reqrep.py", line 123, in update_host
    if not url.host:
  File "/myenv/lib/python3.6/site-packages/yarl/__init__.py", line 59, in __get__
    val = self.wrapped(inst)
  File "/myenv/lib/python3.6/site-packages/yarl/__init__.py", line 433, in host
    return raw.encode('ascii').decode('idna')
UnicodeError: decoding with 'idna' codec failed (UnicodeError: Violation of BIDI requirement 3)
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7f31f464c5f8>
```

I hope this helps clarifying the problem.
Thank you.


Since this is an issue with the Python interpreter and not Requests' code base, I'm going to close this out.

@steven2308 you appear to be using aiohttp and the stacktrace you've provided doesn't contain any Requests code. It appears you're supplying an invalid URI to IDNA which is causing issues. I suggest you follow up with aiohttp if you feel there's an issue.