grequest hangs after about 250,000 requests on windows 7

I was running grequests with 390,000 youtube video urls. But it hang there after about 250, 000 urls.

The error message:

Traceback (most recent call last):
  File "D:\Users\189120\AppData\Local\Continuum\Anaconda\lib\site-packages\gevent\greenlet.py", line 327, in run
    result = self._run(_self.args, *_self.kwargs)
  File "D:\Users\189120\AppData\Local\Continuum\Anaconda\lib\site-packages\grequests.py", line 71, in send
    self.url, *_merged_kwargs)
  File "D:\Users\189120\AppData\Local\Continuum\Anaconda\lib\site-packages\requests\sessions.py", line 464, in request
    resp = self.send(prep, *_send_kwargs)
  File "D:\Users\189120\AppData\Local\Continuum\Anaconda\lib\site-packages\requests\sessions.py", line 576, in send
    r = adapter.send(request, **kwargs)
  File "D:\Users\189120\AppData\Local\Continuum\Anaconda\lib\site-packages\requests\adapters.py", line 424, in send
    raise ConnectionError(e, request=request)
ConnectionError: HTTPSConnectionPool(host='www.googleapis.com', port=443): Max retries exceeded with url: /youtube/v3/videos?part=snippet&id=nv85S4L0kTU&key=xxxxxx (Caused by ProxyError('Cannot connect to proxy.', error(10053, 'An established connection was aborted by the software in your host machine')))
<Greenlet at 0x35eecc0L: <bound method AsyncRequest.send of <grequests.AsyncRequest object at 0x0000000007434128>>(stream=False)> failed with ConnectionError


I'm afraid we need significantly more information to diagnose this. You say 'about' 250,000 URLs: is it consistently 250,000, or does the exact number vary? Do you have any logs from the run? Is there anything in stdout?


I just repeat the error. Ran the code last night and got it this morning. The youtube key is masked as xxxxxx. 
I run the code behind the Proxy. It would be the proxy problem. But if we can add a connection rebuild (need discussion) function to grequests, that will be perfect.

BTW: python 3.4 got asyncio. Is it possible to use asyncio to replace gevent in this package?


So in that case the connection is being aborted by one of the other servers, probably the Proxy server.

How exactly are you using grequests here?


step 1:

```
def req_gen(urls):
    proxies = {
        "http": "http://qfwsg-prod.qantas.com.au:8080",
        "https": "https://qfwsg-prod.qantas.com.au:8080"
    }

    url = 'https://www.googleapis.com/youtube/v3/videos'
    for cur in urls:
        # because of yield, we need put payload inside the loop
        vid = os.path.basename(cur)
        payload = {'key': 'xxxxxxx',
               'part': 'snippet',
               'id': vid}
        #yield grequests.get(url, proxies=proxies, params=payload)
        yield grequests.AsyncRequest('GET', url, proxies=proxies, params=payload)
```

step 2:
loop each 1000 ulrs in about several thousand urls

```
reqs <- req_gen([1000  urls each time])
```

step 3:

```
for rs in grequests.map(reqs):
    ....Transfer rs.json() to target data row
```


It's hard for me to provide useful advice here. I don't think this is a requests bug: sometimes proxies will refuse connections, and it should be possible to handle them appropriately.

I'm not sure exactly how grequests works here (I've never used it myself) but I suspect it should be possible to iterate manually over the map and retry anything that fails:

``` python
rqs = grequests.map(reqs)

while True:
    try:
        rs = next(rqs)
    except ConnectionError:
        # Do something to retry here, put it on a list or something
    else:
        # Transfer rs.json() to target row
```

This is less clean, but lets you handle the errors.


Thanks for your response.
This may be a solution.  Close this issue. I will come back to you if I got some useful error message with your method.

PS: It looks like our proxy problem. The code ran smoothly last night. Thanks, @Lukasa