The right way to debug requests with https proxy

Summary.

I have a proxy service that has been used for months. Everything works well until Twitter. The code here is a proxy test script between google and twitter. Of course Https.

For "https://www.google.com" and "https://mobile.twitter.com", the script both run 180 times. Most of google succeeded, however, all of twitter failed.

google:success times is 171, failed is 9, and total is 180

twitter:success times is 0, failed is 180, and total is 180

Half is "ConnectTimeoutError", and left is "ProxyError"

[Stackoverflow version](https://stackoverflow.com/questions/44750605/the-right-way-to-debug-python-requests-with-https-proxy)

## Expected Result

The success rate should keep the same 


## Actual Result

```
HTTPSConnectionPool(host='mobile.twitter.com', port=443): Max retries exceeded with url: / (Caused by ConnectTimeoutError(<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x107446150>, 'Connection to proxyip timed out. (connect timeout=10)'))
HTTPSConnectionPool(host='mobile.twitter.com', port=443): Max retries exceeded with url: / (Caused by ProxyError('Cannot connect to proxy.', error(54, 'Connection reset by peer')))

```
Without "Upgrade-Insecure-Requests", the errors keep the same.

So I'm curious about is there any method to debug these errors?



## Reproduction Steps

```python
from gevent import monkey, spawn, joinall, sleep
monkey.patch_all()  # Magic!

import requests
import sys

s = requests.Session()
proxys = "https://username:password@ip:port"
class test_proxy:
    success = 0
    failed = 0
    total = 0

    def tweet(self):
        #url = "https://www.google.com"
        url = "https://mobile.twitter.com"
        header = {
            "Referer": url,
            "Upgrade-Insecure-Requests": 1,
            "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.86 Safari/537.36",
        }
        try:
            a = s.get(url, proxies={"https": proxys}, timeout=10)
            self.success += 1
        except Exception as e:
            print(e)
            self.failed += 1
        self.total += 1
        print("sucess times is {}, failed is {}, and total is {}".format(
            self.success, self.failed, self.total))


test = test_proxy()

while True:
    joinall([spawn(test.tweet) for _ in range(50)])

```

## System Information



```
python2.7
```

Is the proxy local, or is it provided by a third-party web service? It's quite possible that either the proxy or Twitter are refusing to co-operate.

Third-party. For another group of proxy,  test results looks a little different , but not such significant. So I don't think it's the problem of proxys, it's the usage of requests or twitter. Should I use cert or something else to improve it efficiently?


*Google*: sucess times is 114, failed is 231, and total is 345
*Twitter*:  sucess times is 12, failed is 343, and total is 355

```
another_proxy = ['https://89.40.115.60:1189', 'https://52.14.213.166:8080', 'https://107.170.61.29:8799',
               'https://94.177.240.167:1189', 'https://103.1.94.144:8080', 'https://45.77.63.43:1189',
               'https://89.36.212.10:1189', 'https://45.76.45.250:1189', 'https://35.190.139.130:80',
               'https://45.32.145.227:1189', 'https://89.38.151.144:1189', 'https://89.36.213.235:1189',
               'https://130.211.110.105:80', 'https://94.177.232.98:1189', 'https://45.32.144.76:1189',
               'https://89.40.112.173:1189', 'https://104.131.19.130:8080', 'https://36.66.64.74:8081',
               'https://197.210.143.242:53281', 'https://45.123.3.105:8080', 'https://186.139.20.245:8080',
               'https://45.76.45.178:1189', 'https://45.32.149.173:1189', 'https://107.191.62.169:1189',
               'https://89.40.112.31:1189', 'https://109.87.145.9:53281', 'https://185.35.67.92:1189',
               'https://94.177.237.100:1189', 'https://114.199.118.186:8080', 'https://107.191.46.61:1189',
               'https://188.17.158.45:53281', 'https://46.73.33.253:8080', 'https://43.255.22.186:8080',
               'https://197.210.246.30:8080', 'https://185.35.67.154:1189', 'https://109.200.155.196:8080',
               'https://103.7.131.74:80', 'https://94.177.241.37:1189', 'https://89.38.148.210:1189',
               'https://89.40.112.56:1189', 'https://45.76.47.194:1189', 'https://195.181.210.203:5033',
               'https://104.238.189.2:1189', 'https://181.39.128.178:53281', 'https://104.238.188.146:1189',
               'https://183.88.112.51:8080', 'https://31.25.141.148:8080']
```

Hrm, very odd. However, what's likely to be relevant is that your proxies are all using `https` schemed URLs. Can you try setting your proxy url to have a `http` scheme and see if that improves things? Requests does not really formally support using TLS to connect to a proxy.

 so weird, why would  {"http":"http://ip:port"} work for https url .... ?  I will have more test later....

It would not, but `{'https': 'http://ip:port'}` would, which is what I want you to try. =)

After a few more tests, {'https': 'http://ip:port'} doesn't work too.  I
thought {"http":"http://ip:port"}  would work, but I figure it out now, the
requests were sent to target directly .  so sad ......



On Mon, Jun 26, 2017 at 7:17 PM Cory Benfield <notifications@github.com>
wrote:

> It would not, but {'https': 'http://ip:port'} would, which is what I want
> you to try. =)
>
> —
> You are receiving this because you authored the thread.
>
>
> Reply to this email directly, view it on GitHub
> <https://github.com/requests/requests/issues/4176#issuecomment-311031219>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AC6kD-jfhvyPizp1syOefrVz932lzEZ4ks5sH5NYgaJpZM4OE-bQ>
> .
>
-- 

Tony Wang

Warm Regards


So this sounds enormously like there is a problem between your proxy and Twitter.

Maybe.... I will submit a ticket to the proxy service. Thanks for your kind
of help

On Tue, Jun 27, 2017 at 1:51 AM Cory Benfield <notifications@github.com>
wrote:

> So this sounds enormously like there is a problem between your proxy and
> Twitter.
>
> —
> You are receiving this because you authored the thread.
>
>
> Reply to this email directly, view it on GitHub
> <https://github.com/requests/requests/issues/4176#issuecomment-311132648>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AC6kDybAcWa82n4kv3Hmfv5ImdsadSWuks5sH--6gaJpZM4OE-bQ>
> .
>
-- 

Tony Wang

Warm Regards