Request function never returns

I've a scraper that uses 200 threads to scrape and one thread to post data to an api with POST method.
Here's an excerpt from the log (normal behaviour with DEBUG loglevel):

> 06:25:44 MainThread : Before sending
> 06:25:46 MainThread : Resetting dropped connection: api.autosforsale.com
> 06:25:50 MainThread : http://api.autosforsale.com:80 "POST /acceptbulkrecords/?api_key=xxx HTTP/1.1" 200 None
> 06:25:50 MainThread : After sending: <Response [200]>
> 06:25:51 MainThread : Sent 1005. API STATUS: 200 {

And then when it's about to send another batch:

> 06:27:26 MainThread : Before sending
> 06:27:27 MainThread : Resetting dropped connection: api.autosforsale.com

There's no response in 8 hours.

I'm ok with anything that'll give a way to retry. I don't remember exactly, but I think I get this problem every now and then, although quite rarely.

The piece of code (s=Session()):
```python
    b=BytesIO(dumps(batch).encode())
    lg.info('Before sending')
    r=s.post(API_URL, params={'api_key': API_KEY},
             files=[('files[]', ('.json', b, 'application/octet-stream'))],
             data={'source': 'cargurus', 'time_stamp': import_start},
             timeout=20
    )
    lg.info('After sending: {}'.format(r))
```

## System Information

    $ python -m requests.help

```
{
  "chardet": {
    "version": "3.0.4"
  },
  "cryptography": {
    "version": ""
  },
  "idna": {
    "version": "2.6"
  },
  "implementation": {
    "name": "CPython",
    "version": "3.6.3"
  },
  "platform": {
    "release": "2.6.32-696.10.3.el6.x86_64",
    "system": "Linux"
  },
  "pyOpenSSL": {
    "openssl_version": "",
    "version": null
  },
  "requests": {
    "version": "2.18.4"
  },
  "system_ssl": {
    "version": "1000105f"
  },
  "urllib3": {
    "version": "1.22"
  },
  "using_pyopenssl": false
}
```


Ok, got some more information. I was running it with tcpdump and here's what's in there.

```
> 00:34:34.063309 IP 192.126.255.227.60926 > 104.27.177.155.http: Flags [P.], seq 23053526:23054004, ack 3122, win 173, length 478
> E..../@.@....~..h......P58.#.9."P.......nated Power Window Switches|Power Windows: With Safety Reverse|Windows Front Wipers: Variable Intermittent|Suspension Stabilizer Bar(s): Front|Suspension Stabilizer Bar(s): Rear|Windows Rear Wiper: Dual Speed|Windows Lockout Button|Windows Rear Wiper With Washer|Windows Heated Windshield Wiper Rests|Suspension Front Shock Type: Gas Shock Absorbers|Windows Rear Defogger|Audio - Internet Radio: AHA|Audio - Internet Radio: iHeartRadio"}]
> --51bfccd0807e4620929c0a17484ef1a1--
> 
> --------- skipped ---------
> 
> 00:34:34.083435 IP 104.27.177.155.http > 192.126.255.227.60926: Flags [.], ack 23054004, win 1287, length 0
> E..(..@.8.[.h....~...P...9."58..P...........n.
> 00:34:54.098768 IP 192.126.255.227.60926 > 104.27.177.155.http: Flags [F.], seq 23054004, ack 3122, win 173, length 0
> E..(.0@.@....~..h......P58...9."P....(..
> 00:34:54.119856 IP 104.27.177.155.http > 192.126.255.227.60926: Flags [F.], seq 3122, ack 23054005, win 1287, length 0
> E..(..@.8.[.h....~...P...9."58..P.........0.~~
> 00:34:54.119867 IP 192.126.255.227.60926 > 104.27.177.155.http: Flags [.], ack 3123, win 173, length 0
> E..(.1@.@....~..h......P58...9.#P....'..
```

So it sends the last part of the message, and the packet gets acknowledged, but then requests waits for something and after 20 seconds breaks the connection. But still doesn't return.

Hey @aikipooh, I'm not positive what would be causing this if the connection is being closed. My first hunch is it may have to do with your threading implementation. Are you creating a session for the whole program and sharing it among threads, or is each thread instantiating it's own session?

I *believe* there are still some thread-safety issues with Sessions that may be causing this. There is probably another maintainer who has more background than I do around this though.

Hi @nateprewitt!
In this case I've like 100 threads, each with its own Session, and another one to post to API. So if it's something about locking (and looks like it is), it must be deep within requests bowels:( Can they step on each other's toes underground?

> Can they step on each other's toes underground?

No. Two objects in separate threads cannot actually modify each other. This is why the strategy you're employing is what we recommend for now.

That said, your operating system can absolutely cause this. Another potential cause, since you're scraping, is that you're overloading a website and Requests is waiting for the response to complete but the server is keeping the socket alive.

When you say:

> requests waits for something and after 20 seconds breaks the connection. But still doesn't return.

The 20 seconds makes sense with the `timeout=20` you've specified. That means that after 20 seconds of not receiving the end of the response or more data, Requests stops trying to read. What does "But still doesn't return" mean?

@sigmavirus24 Two objects shouldn't modify, but deadlocks on shared resources in the requests' bowels can happen.
I experience this on three different OS. Today confirmed one such case, with 100 threads talking to one site (different Sessions though, as always).
As far as you can see, FIN comes, I don't know anything more explicit to NOT to keep the socket alive:)))
Requests stops trying to read and? Shouldn't it return or throw an exception?

“But still doesn't return” means exactly this — the control isn't passed to the outer frame. We don't execute anything after s.post().

> Two objects shouldn't modify, but deadlocks on shared resources in the requests' bowels can happen.

What shared resources are you creating between the two? Requests and urllib3 don't share resources by default.

> As far as you can see, FIN comes

Quoting what you gave us:

```
> 00:34:34.063309 IP 192.126.255.227.60926 > 104.27.177.155.http: Flags [P.], seq 23053526:23054004, ack 3122, win 173, length 478
> E..../@.@....~..h......P58.#.9."P.......nated Power Window Switches|Power Windows: With Safety Reverse|Windows Front Wipers: Variable Intermittent|Suspension Stabilizer Bar(s): Front|Suspension Stabilizer Bar(s): Rear|Windows Rear Wiper: Dual Speed|Windows Lockout Button|Windows Rear Wiper With Washer|Windows Heated Windshield Wiper Rests|Suspension Front Shock Type: Gas Shock Absorbers|Windows Rear Defogger|Audio - Internet Radio: AHA|Audio - Internet Radio: iHeartRadio"}]
> --51bfccd0807e4620929c0a17484ef1a1--
> 
> --------- skipped ---------
> 
> 00:34:34.083435 IP 104.27.177.155.http > 192.126.255.227.60926: Flags [.], ack 23054004, win 1287, length 0
> E..(..@.8.[.h....~...P...9."58..P...........n.
> 00:34:54.098768 IP 192.126.255.227.60926 > 104.27.177.155.http: Flags [F.], seq 23054004, ack 3122, win 173, length 0
> E..(.0@.@....~..h......P58...9."P....(..
> 00:34:54.119856 IP 104.27.177.155.http > 192.126.255.227.60926: Flags [F.], seq 3122, ack 23054005, win 1287, length 0
> E..(..@.8.[.h....~...P...9."58..P.........0.~~
> 00:34:54.119867 IP 192.126.255.227.60926 > 104.27.177.155.http: Flags [.], ack 3123, win 173, length 0
> E..(.1@.@....~..h......P58...9.#P....'..
```

I don't see the FIN there. do you have extra information?

> Requests stops trying to read and? Shouldn't it return or throw an exception?

It should, unless it doesn't see the FIN. Can you reproduce this with any other HTTP client in Python?

I don't create any shared resources within requests code. Just suspecting there may be some.

Erm, FIN?
00:34:54.098768 IP 192.126.255.227.60926 > 104.27.177.155.http: Flags [F.],

I can't reproduce, because I was using requests only… I'm switching all my projects to aiohttp now, works ok so far. But again, even with requests it was rare, except for one script which suffers from this every day.

Closing due to inactivity — thank you very much for your patience ✨🍰✨

How sweet. There's a bug, you do nothing, then close it because of inactivity. The bug disappears. Brilliant! Just brilliant!

@aikipooh there's no need for theatrics. I've been unable to reproduce the issue with the code you provided and I have a feeling it's leaving out details. Sigmavirus asked to see if you could reproduce this in other python clients, specifically urllib3 and httplib(py2)/http.client(py3). This will help determine if it's an issue in a layer underneath Requests or Requests itself.

The work here on Requests is volunteer and we have a limited amount of resources to allocate. I'm happy to help investigate further but you're going to need to drive the issue if you want it resolved. Without a general repro, it's very difficult for us to move forward. Thanks for your understanding and let us know if you have answers/questions on the next steps.

That's what I've said, the thing is evanescent, requires many threads and days running at times. I've been using requests for 3 years now and this happened only in 3 projects, but on 3 different platforms, which is very good. And I doubt you've been conducting the same test…

I've no time to rewrite and debug too, limited amount of resources to allocate, you know. But last thing in the message exchange was talking about FIN. Not seeing this FIN in my log and then saying "It should, unless it doesn't see the FIN". This I didn't understand and was waiting to continue considering this version. And then: no activity.

But well, no need for theatrics and no need for requests either. Good luck.