Missing header information in Response object

## Summary

The response object generated from a POST request is missing header information. I am using Python (3.6.2) and the requests (2.18.4) library to post a query to the [Primer-BLAST](https://www.ncbi.nlm.nih.gov/tools/primer-blast/) website. I need to retrieve the NCBI-RCGI-RetryURL field from the response's header information. However, I can only see this field when I use the HTTP trace extension in Google Chrome. [I have asked this question on StackOverflow as well.](https://stackoverflow.com/questions/45942219/missing-header-information-using-pythons-requests-library) 

## Expected Result

Below is the full trace of the POST and response using Google Chrome:

```
POST https://www.ncbi.nlm.nih.gov/tools/primer-blast/primertool.cgi
Origin: https://www.ncbi.nlm.nih.gov
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.101 Safari/537.36
Content-Type: multipart/form-data; boundary=----WebKitFormBoundaryBflp51Ny9ReeA5A9
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8
Referer: https://www.ncbi.nlm.nih.gov/tools/primer-blast/index.cgi?LINK_LOC=reset
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.8
Cookie: sv-userdata-key-www.ncbi.nlm.nih.gov=G5KxXzyQ81U_vs1aHK_7XDWciF1B8AjjDUmDunVbhIZhZ4p4t_SVK4ASpbTT8iDSJVcxBH9oUAB3K2xNWjp3G0koYCloBlYuSxdoIGIkYzl2; ncbi_sid=0751457F9A561D01_0000SID; _ga=GA1.2.567134514.1503994317; _gid=GA1.2.654339584.1503994317; _gat=1; starnext=MYGwlsDWB2CmAeAXAXAbgK7RAewIYBM4lkAmAXgAcAnMAW1ioCMRcBnRAMgBYzm3FWsXFWAALDgEZymHAUkBOMgAYA7AFYpXFQDF5AQTUA2ACIBRFRKVXrN2xI4klygMJcSltQA59R0wGY/S1tg63t3Shp6JhZ2AFI/PQA5AHlE03i9PnZBYTEMlLSHcgB3UoA6aGBGMAqQWgqwUTKAc2wANwceajoGLMR81NMHQwie6P4BtIy+nJFxEhVRqL7J9ISZoTnVh08l3pj+lWcC9KON3NFYo5OHRQkuLiUOPyd5K2eJMnvH5/JPDWefjIADNcCBBM8eIgqOhYM81F83GpniMJH4SPJnosSFxnrtAoZ5Li/IoXp5DCpuE50RIpNxPlIAtxyNBcIgwG04Q8yDI8IQEJwuAiSBw1EDvk81DxPEo/KKEfIRUYyCRDIZRYtJbtaT81HcOIYnE9DAycUA=

HTTP/1.1 200 OK
Date: Tue, 29 Aug 2017 13:38:27 GMT
Server: Apache
Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
Referrer-Policy: origin-when-cross-origin
Content-Security-Policy: upgrade-insecure-requests
Cache-Control: no-cache, no-store, max-age=0, private, must-revalidate
Expires: 0
NCBI-PHID: 0C421A7A9A56E5310000000000000001.m_2
NCBI-RCGI-RetryURL: https://www.ncbi.nlm.nih.gov/tools/primer-blast/primertool.cgi?ctg_time=1504013907&job_key=aWO2H68Wor6FhLSBueGQs8P6gYHu6Zqc7w
NCBI-SID: 0751457F9A561D01_0000SID
Pragma: no-cache
Access-Control-Allow-Methods: POST, GET, PUT, OPTIONS, PATCH, DELETE
Access-Control-Allow-Origin: https://www.ncbi.nlm.nih.gov
Access-Control-Allow-Credentials: true
Access-Control-Allow-Headers: Origin,X-Accept-Charset,X-Accept,Content-Type,X-Requested-With,NCBI-SID,NCBI-PHID
Content-Type: text/html
Set-Cookie: ncbi_sid=0751457F9A561D01_0000SID; domain=.nih.gov; path=/; expires=Wed, 29 Aug 2018 13:38:27 GMT
Vary: Accept-Encoding
Content-Encoding: gzip
X-UA-Compatible: IE=Edge
X-XSS-Protection: 1; mode=block
Keep-Alive: timeout=1, max=9
Connection: Keep-Alive
Transfer-Encoding: chunked
```
## Actual Result

And here is all the header information I get from my script:

```
Date : Tue, 29 Aug 2017 14:41:08 GMT
Server : Apache
Strict-Transport-Security : max-age=31536000; includeSubDomains; preload
Referrer-Policy : origin-when-cross-origin
Content-Security-Policy : upgrade-insecure-requests
Accept-Ranges : bytes
Vary : Accept-Encoding
Content-Encoding : gzip
X-UA-Compatible : IE=Edge
X-XSS-Protection : 1; mode=block
Content-Length : 2516
Keep-Alive : timeout=1, max=10
Connection : Keep-Alive
Content-Type : text/html
```

## Reproduction Steps

```python
#!/usr/bin/env python

import requests

# BaseURL being accessed
url = 'https://www.ncbi.nlm.nih.gov/tools/primer-blast/primertool.cgi'

# Dictionary of query parameters
data = {
    'INPUT_SEQUENCE' : 'TCTTCTGAGAAAGTCTGAGGCTCCTTAGTACCTTCTCTAGTATGAACTGTTCAGCCTGCCCGCAAGTTGTAACTACGCAGGCGCCAAGACAGCCAACCAAGGAGGCTGCAGA'
}

# Make a POST request and read the response
with requests.session() as session:
    poster = session.post(url, data=data)
    for key, value in poster.headers.items():
        print(key, ':', value)

```

## System Information

    $ python -m requests.help

```
$ python -m requests.help
{
  "chardet": {
    "version": "3.0.4"
  },
  "cryptography": {
    "version": ""
  },
  "idna": {
    "version": "2.6"
  },
  "implementation": {
    "name": "CPython",
    "version": "3.6.2"
  },
  "platform": {
    "release": "16.7.0",
    "system": "Darwin"
  },
  "pyOpenSSL": {
    "openssl_version": "",
    "version": null
  },
  "requests": {
    "version": "2.18.4"
  },
  "system_ssl": {
    "version": "100020cf"
  },
  "urllib3": {
    "version": "1.22"
  },
  "using_pyopenssl": false
}
```

What's the status code and content when you use Requests? Are you getting the response data you expect?

The status code is 200 and the content is just:

`b'<html><head><title>Empty Result</title></head><body>Empty Result</body></html>'`

Which I believe is correct because the results take a few minutes to be processed.

So there are two possibilities: either the server did not send that data, or it did but the header parsing failed. Do you see any warnings emitted from your code?

I don't see any warnings when I run my code.

Hrm, we may need to try to see the raw bytes of the headers received. Can you try to use mitmproxy to show us what the actual data was?

Sure, let me install on my computer. Is there anything in particular you want me to run / show?

In the first instance, I just want you to see if mitmproxy is showing you headers that Requests is not.

Apologies, I have installed mitmproxy but not sure how to run it? When I run mitmproxy at the terminal it just presents with a black screen?

`mitmproxy` should present a curses interface, with information about the port it's running in at the bottom-right. When it does so you should make the request again, telling Requests to use it as a proxy (e.g. `proxies={'https': 'http://localhost:8080'}`).

I have mitmproxy running in one window, and in another I have re-run the script with the proxies parameter inlcuded in the POST function call. It now encounters an error:

```
Traceback (most recent call last):
  File "/Users/james/anaconda/envs/requests/lib/python3.6/site-packages/urllib3/connectionpool.py", line 595, in urlopen
    self._prepare_proxy(conn)
  File "/Users/james/anaconda/envs/requests/lib/python3.6/site-packages/urllib3/connectionpool.py", line 816, in _prepare_proxy
    conn.connect()
  File "/Users/james/anaconda/envs/requests/lib/python3.6/site-packages/urllib3/connection.py", line 326, in connect
    ssl_context=context)
  File "/Users/james/anaconda/envs/requests/lib/python3.6/site-packages/urllib3/util/ssl_.py", line 329, in ssl_wrap_socket
    return context.wrap_socket(sock, server_hostname=server_hostname)
  File "/Users/james/anaconda/envs/requests/lib/python3.6/ssl.py", line 401, in wrap_socket
    _context=self, _session=session)
  File "/Users/james/anaconda/envs/requests/lib/python3.6/ssl.py", line 808, in __init__
    self.do_handshake()
  File "/Users/james/anaconda/envs/requests/lib/python3.6/ssl.py", line 1061, in do_handshake
    self._sslobj.do_handshake()
  File "/Users/james/anaconda/envs/requests/lib/python3.6/ssl.py", line 683, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:748)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/james/anaconda/envs/requests/lib/python3.6/site-packages/requests/adapters.py", line 440, in send
    timeout=timeout
  File "/Users/james/anaconda/envs/requests/lib/python3.6/site-packages/urllib3/connectionpool.py", line 639, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/Users/james/anaconda/envs/requests/lib/python3.6/site-packages/urllib3/util/retry.py", line 388, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='www.ncbi.nlm.nih.gov', port=443): Max retries exceeded with url: /tools/primer-blast/primertool.cgi (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:748)'),))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "primer.py", line 35, in <module>
    poster = session.post(url, data=data, proxies={'https': 'http://localhost:8080'})
  File "/Users/james/anaconda/envs/requests/lib/python3.6/site-packages/requests/sessions.py", line 555, in post
    return self.request('POST', url, data=data, json=json, **kwargs)
  File "/Users/james/anaconda/envs/requests/lib/python3.6/site-packages/requests/sessions.py", line 508, in request
    resp = self.send(prep, **send_kwargs)
  File "/Users/james/anaconda/envs/requests/lib/python3.6/site-packages/requests/sessions.py", line 618, in send
    r = adapter.send(request, **kwargs)
  File "/Users/james/anaconda/envs/requests/lib/python3.6/site-packages/requests/adapters.py", line 506, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='www.ncbi.nlm.nih.gov', port=443): Max retries exceeded with url: /tools/primer-blast/primertool.cgi (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:748)'),))
```

Sorry, please also set `verify=False`

Okay, this time I get the following message from the script:

```
/Users/james/anaconda/envs/requests/lib/python3.6/site-packages/urllib3/connectionpool.py:858: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecureRequestWarning)
```
This request in the mtmproxy window:

https://ibb.co/i2X2K5

and this response in the mtmproxy window:

https://ibb.co/jfjSmk

So it looks as though the retryURL field is not being sent from the server?


That's true. Is Requests seeing all the header fields mitmproxy is? Because there are a number there that were not in your initial report.

It looks like Requests and mitmproxy are seeing the same fields. But HTTP trace (which is where I got the initial response using Google Chrome and the HTTP trace extension) is seeing a few more?

If that's coming from Chrome I assume it's because the requests are different. Can you try adding header fields and body data as needed to get them two requests to be the same?

Okay, I changed my script to contain the same headers:

```
#!/usr/bin/env python

import requests
import time

# Base URL being accessed
url = 'https://www.ncbi.nlm.nih.gov/tools/primer-blast/primertool.cgi'

# Dictionary of query parameters
data = {
    'INPUT_SEQUENCE'              : 'TCTTCTGAGAAAGTCTGAGGCTCCTTAGTACCTTCTCTAGTATGAACTGTTCAGCCTGCCCGCAAGTTGTAACTACGCAGGCGCCAAGACAGCCAACCAAGGAGGCTGCAGA',
#    'ORGANISM'                    : 'Mus musculus',
#    'PRIMER_SPECIFICITY_DATABASE' : 'PRIMERDB/genome_selected_species'
}

# Extra headers
headers = {
    'Origin' : 'https://www.ncbi.nlm.nih.gov',
    'Upgrade-Insecure-Requests' : '1',
    'User-Agent' : 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.101 Safari/537.36',
    'Content-Type' : 'multipart/form-data; boundary=----WebKitFormBoundaryBflp51Ny9ReeA5A9',
    'Accept' : 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8',
    'Referer' : 'https://www.ncbi.nlm.nih.gov/tools/primer-blast/index.cgi?LINK_LOC=reset',
    'Accept-Encoding' : 'gzip, deflate, br',
    'Accept-Language' : 'en-US,en;q=0.8',
    'Cookie' : 'sv-userdata-key-www.ncbi.nlm.nih.gov=G5KxXzyQ81U_vs1aHK_7XDWciF1B8AjjDUmDunVbhIZhZ4p4t_SVK4ASpbTT8iDSJVcxBH9oUAB3K2xNWjp3G0koYCloBlYuSxdoIGIkYzl2; ncbi_sid=0751457F9A561D01_0000SID; _ga=GA1.2.567134514.1503994317; _gid=GA1.2.654339584.1503994317; _gat=1; starnext=MYGwlsDWB2CmAeAXAXAbgK7RAewIYBM4lkAmAXgAcAnMAW1ioCMRcBnRAMgBYzm3FWsXFWAALDgEZymHAUkBOMgAYA7AFYpXFQDF5AQTUA2ACIBRFRKVXrN2xI4klygMJcSltQA59R0wGY/S1tg63t3Shp6JhZ2AFI/PQA5AHlE03i9PnZBYTEMlLSHcgB3UoA6aGBGMAqQWgqwUTKAc2wANwceajoGLMR81NMHQwie6P4BtIy+nJFxEhVRqL7J9ISZoTnVh08l3pj+lWcC9KON3NFYo5OHRQkuLiUOPyd5K2eJMnvH5/JPDWefjIADNcCBBM8eIgqOhYM81F83GpniMJH4SPJnosSFxnrtAoZ5Li/IoXp5DCpuE50RIpNxPlIAtxyNBcIgwG04Q8yDI8IQEJwuAiSBw1EDvk81DxPEo/KKEfIRUYyCRDIZRYtJbtaT81HcOIYnE9DAycUA='
}

auth = ('s1437643@sms.ed.ac.uk', 'password')

# Setup a session
with requests.session() as session:

    # Make a POST request and read the response
    poster = session.post(url, data=data, headers=headers, proxies={'https': 'http://localhost:8080'}, verify=False)

    # Requests header
    print('-' * 50, 'Requests header', '-' * 50)
    for key, value in poster.request.headers.items():
        print(key, ':', value)

    # Response header
    print('-' * 50, 'Response header', '-' * 50)
    for key, value in poster.headers.items():
        print(key, ':', value)

    # Print status code
    print('-' * 50, 'Status code', '-' * 50)
    print(poster.status_code)

    # Print content
    print('-' * 50, 'Response content', '-' * 50)
    print(poster.content)
```
Here is the output from the script:

```
/Users/james/anaconda/envs/requests/lib/python3.6/site-packages/urllib3/connectionpool.py:858: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecureRequestWarning)
-------------------------------------------------- Requests header --------------------------------------------------
User-Agent : Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.101 Safari/537.36
Accept-Encoding : gzip, deflate, br
Accept : text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8
Connection : keep-alive
Origin : https://www.ncbi.nlm.nih.gov
Upgrade-Insecure-Requests : 1
Content-Type : multipart/form-data; boundary=----WebKitFormBoundaryBflp51Ny9ReeA5A9
Referer : https://www.ncbi.nlm.nih.gov/tools/primer-blast/index.cgi?LINK_LOC=reset
Accept-Language : en-US,en;q=0.8
Cookie : sv-userdata-key-www.ncbi.nlm.nih.gov=G5KxXzyQ81U_vs1aHK_7XDWciF1B8AjjDUmDunVbhIZhZ4p4t_SVK4ASpbTT8iDSJVcxBH9oUAB3K2xNWjp3G0koYCloBlYuSxdoIGIkYzl2; ncbi_sid=0751457F9A561D01_0000SID; _ga=GA1.2.567134514.1503994317; _gid=GA1.2.654339584.1503994317; _gat=1; starnext=MYGwlsDWB2CmAeAXAXAbgK7RAewIYBM4lkAmAXgAcAnMAW1ioCMRcBnRAMgBYzm3FWsXFWAALDgEZymHAUkBOMgAYA7AFYpXFQDF5AQTUA2ACIBRFRKVXrN2xI4klygMJcSltQA59R0wGY/S1tg63t3Shp6JhZ2AFI/PQA5AHlE03i9PnZBYTEMlLSHcgB3UoA6aGBGMAqQWgqwUTKAc2wANwceajoGLMR81NMHQwie6P4BtIy+nJFxEhVRqL7J9ISZoTnVh08l3pj+lWcC9KON3NFYo5OHRQkuLiUOPyd5K2eJMnvH5/JPDWefjIADNcCBBM8eIgqOhYM81F83GpniMJH4SPJnosSFxnrtAoZ5Li/IoXp5DCpuE50RIpNxPlIAtxyNBcIgwG04Q8yDI8IQEJwuAiSBw1EDvk81DxPEo/KKEfIRUYyCRDIZRYtJbtaT81HcOIYnE9DAycUA=
Content-Length : 127
-------------------------------------------------- Response header --------------------------------------------------
Date : Tue, 29 Aug 2017 18:17:53 GMT
Server : Apache
Strict-Transport-Security : max-age=31536000; includeSubDomains; preload
Referrer-Policy : origin-when-cross-origin
Content-Security-Policy : upgrade-insecure-requests
Accept-Ranges : bytes
Vary : Accept-Encoding
Content-Encoding : gzip
X-UA-Compatible : IE=Edge
X-XSS-Protection : 1; mode=block
Content-Length : 2517
Keep-Alive : timeout=1, max=10
Connection : Keep-Alive
Content-Type : text/html
-------------------------------------------------- Status code --------------------------------------------------
403
-------------------------------------------------- Response content --------------------------------------------------
b'<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN" "http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd"> \n\n<html>\n\n<head>\n<title>NCBI - WWW Error 403 Diagnostic</title>\n<meta http-equiv="Content-Type" content="application/xhtml+xml; charset=utf-8" />\n<meta http-equiv="pragma" content="no-cache" />\n<meta http-equiv="cache-control" content="no-cache" />\n<meta name="robots" content="noarchive,none" />\n\n<style type="text/css">\nbody {\nmin-width: 950px;\n_width: 950px;\n}\nh1.error {color: red; font-size: 40pt}\n\n#mainContent {\npadding: 1em 2em;\n}\n\ndl#diags {\npadding: 0.5em 0.5em 1.5em 0.5em;\npadding-left: 2em;\nborder: solid 1px #888;\nborder-left: none;\nborder-right: none;\nmargin-bottom:0;\nbackground-color:#eeeeee;\ncolor:#666;\nfont-size: 80%;\n_font-size: 70%;\nfont-family: Verdana, sans-serif;\n}\n\ndl#diags dt {\nfloat: left;\nfont-weight: bold;\nwidth: auto;\n}\n\ndl#diags dd {\nmargin-left:1em;\nfloat: left;\nmargin-right: 2em;\n}\n\n#footer span {\nfloat: left;\ncolor: #888;\nfont-size: 80%;\n}\n\n#footer {\ntext-align: right;\npadding: 0 0.5em;\nclear: left;\n}\n\n#footer img {\nborder: none;\n}\n\n.ncbi {\nmargin: 0;\npadding:0;\nfont-size:240%;\nfont-weight: bold;\nfont-family: Times, serif;\ncolor: #336699;\nfloat: left;\ndisplay: inline;\n}\n\n.ncbi a {\ntext-decoration: none;\ncolor: #336699;\n}\n\n.ncbi a:visited {\ntext-decoration: none;\ncolor: #336699;\n}\n\n.ncbi a:hover {\ntext-decoration: underline;\n}\n\n.message {\nfont-family: Verdana, sans-serif;\nbackground-color: #336699;\ncolor: white;\npadding: .35em;\nmargin-left: 7em;\nmargin-top: .67em;\n_margin-top: 0.5em;\nfont-weight: bold;\nfont-size: 100%;\nmargin-bottom: 0;\n}\n\nh1 {\nclear: left;\nfont-size: 110%;\nfont-family: Verdana, sans-serif;\n}\n\n\nbody.denied {\nbackground-color: black;\ncolor: white;\n}\n\nbody.denied h1 {\ncolor: red;\n}\n\nbody.denied a {\ncolor: green;\n}\n\nbody.denied #footer, body.denied #diags {\ncolor: black;\n}\n\n#searchme:focus {\nbackground-color: #ffa;\n}\n\n.errurl {\nletter-spacing: 0;\nmargin: 0 1em;\npadding: 0.25em;\nbackground-color: #fff0f0;\ncolor: #c00;\nfont-family: "Courier New", Courier, monospace;\nfont-size: 90%;\n_font-size: 80%;\n}\n\nbody.denied .errurl {\nbackground-color: black;\ncolor: yellow;\n}\n\nspan.x {\ndisplay: none;\n}\n\n</style>\n\n<!--[if IE]>\n<script type="text/javascript">\n/* <![CDATA[ */\n_isIE=1;\n/* ]]> */\n</script>\n<![endif]-->\n\n<script type="text/javascript">\n/* <![CDATA[ */\naddEvent = function(obj, type, fn, b) {\n    if (obj.attachEvent) {\n        var name = "" + type + fn; \n        name = name.substring(0, name.indexOf("\\n"));\n        obj["e" + name] = fn;\n        obj[name] = function(){ obj["e" + name](window.event);}\n        obj.attachEvent("on" + type, obj[name]);\n    } else {\n        obj.addEventListener(type, fn, b);\n        return true;\n    }\n}\n\nfixlink = function() {\n   var ie = typeof(_isIE) != \'undefined\';\n   var v = document.getElementById("mlink");\n\n   if (v) {\n      var ieHACK = v.innerHTML;\n      var newText;\n\n      newText = "%3E%20REQUEST_URI%3D" + document.REQUEST_URI + "%0D" +\n                "%3E%20HTTP_HOST%3D" + document.HTTP_HOST + "%0D" +\n                "%3E%20REMOTE_ADDR%3D" + document.REMOTE_ADDR + "%0D" +\n                "%3E%20REQUEST_METHOD%3D" + document.REQUEST_METHOD + "%0D";\n\n      var href = v.getAttribute("href");\n      newText = newText.replace(/\'/g, "%27");\n      newText = newText.replace(/\\?/g, "%3F");\n\n      // IE bugfixes\n      if (ie) {\n          var m = /(.*)\\?(.*)/.exec(href);\n          if (m) {\n             href = m[1] + "?" + escape(m[2]);\n          }\n          href = href.replace(/(.*)(%0D-----%0D.*)/m, "$1" + newText + "$2");\n          href = href.replace(/subject%3D/, "subject=");\n          href = href.replace(/%26body%3D/, "&body=");\n          href=href.replace(/%0D/mg, "%0A");\n      } else {\n          href = href.replace(/(.*)(%0D-----%0D.*)/m, "$1" + newText + "$2");\n      }\n      v.setAttribute("href", href);\n      v.innerHTML = ieHACK;  // I can\'t believe this...\n   }  \n   v = document.getElementById("searchme");\n   if (v) { v.focus(); }\n}\n\n\naddEvent(window, "load", fixlink, false);\n/* ]]> */\n</script>\n\n\n</head>\n\n<body>\n\n<div id=\'header\'>\n<a href="#mainContent" title="Skip to main content" />\n<p class="ncbi"><a href="http://www.ncbi.nlm.nih.gov">NCBI</a></p>\n<p class="message">Error</p>\n</div>\n\n<div id=\'mainContent\'>\n<h1>The requested page has restricted access</h1>\n\n<p class=\'errurl\'>\n<script type="text/javascript">\n/* <![CDATA[ */\n  document.REQUEST_URI = "/tools/primer-blast/primertool.cgi";\n  document.HTTP_HOST = "www.ncbi.nlm.nih.gov";\n  document.REMOTE_ADDR = "192.41.130.86";\n  document.REQUEST_METHOD = "GET";\n\n  var p = "http://" + document.HTTP_HOST + document.REQUEST_URI;\n  p = p.replace(/(.)/g, "<span class=\\"x\\"> </span>$1");\n  p = p.replace(/\'/g, "&apos;");\n  document.write(p);\n//  document.write(p.replace(/[^A-Za-z0-9\\#\\;\\:\\.\\/\\?&\\=\\_\\- ]*/g, "<span class=\'x\'> </span>"));\n/* ]]> */\n</script>\n</p>\n\n<p>\nYou have requested a page which is not open to the public.  Your\nrequest did not meet the criteria required to grant access to this\npage.\n</p>\n\n<p>\nIf you believe that you are being denied access to the page in\nerror, send an e-mail problem report to <a id=\'mlink\'\nhref="mailto:info@ncbi.nlm.nih.gov?subject=NCBI%20Web%20site%20error%20403&amp;body=%3E%20Error%3D403%0D%3E%20Server%3D130.14.29.110%0D%3E%20Client%3D192.41.130.86%0D%3E%20Time%3DTuesday, 29-Aug-2017 14:17:53 EDT%0D%0D-----%0DPlease%20enter%20comments%20below:%0D%0D">info@ncbi.nlm.nih.gov</a>. \n</p>\n\n\n<form method="post" action="http://www.ncbi.nlm.nih.gov/gquery/gquery.fcgi"\n     enctype="application/x-www-form-urlencoded">\n<p>\n<strong>Search NCBI</strong>\n<input name="term" size="40" style="margin-left: 1em; border: solid 1px #336699;" id="searchme"/>\n<input type="submit" value="Go" />\n</p>\n</form>\n\n\n</div>\n\n<dl id="diags">\n   <dt>Error</dt><dd>403</dd>\n   <dt>Server</dt><dd>130.14.29.110</dd>\n   <dt>Client</dt><dd>192.41.130.86</dd>\n   <dt>Time</dt><dd>Tuesday, 29-Aug-2017 14:17:53 EDT</dd>\n</dl>\n\n<p id=\'footer\'>\n\n<span id=\'rev\'>Rev. 18 Jul 2017 16:09:24 EDT</span>\n<!-- <a href="http://validator.w3.org/check?uri=referer"><img src="http://www.w3.org/Icons/valid-xhtml11" alt="Valid XHTML 1.1!" /></a> -->\n</p>\n\n</body>\n</html>\n\n'
```
Here is the request and response from mitmproxy:

https://ibb.co/ngTjz5
https://ibb.co/cfmtsQ

So this time I'm getting a 403 error. Maybe the server maintainers have blocked script requests somehow?

That cookie may be a problem. Does this service require logging in?

Nope, it's free to use for anyone and doesn't require you to login with any user ID or password.

Hrm, well we need to fix the 403 error. Try removing the Cookie header and see what happens.

Without the Cookie header I get the following content from the script:

```
/Users/james/anaconda/envs/requests/lib/python3.6/site-packages/urllib3/connectionpool.py:858: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecureRequestWarning)
-------------------------------------------------- Requests header --------------------------------------------------
User-Agent : Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.101 Safari/537.36
Accept-Encoding : gzip, deflate, br
Accept : text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8
Connection : keep-alive
Origin : https://www.ncbi.nlm.nih.gov
Upgrade-Insecure-Requests : 1
Content-Type : multipart/form-data; boundary=----WebKitFormBoundaryBflp51Ny9ReeA5A9
Referer : https://www.ncbi.nlm.nih.gov/tools/primer-blast/index.cgi?LINK_LOC=reset
Accept-Language : en-US,en;q=0.8
Content-Length : 127
-------------------------------------------------- Response header --------------------------------------------------
Date : Tue, 29 Aug 2017 18:25:39 GMT
Server : Apache
Strict-Transport-Security : max-age=31536000; includeSubDomains; preload
Referrer-Policy : origin-when-cross-origin
Content-Security-Policy : upgrade-insecure-requests
Accept-Ranges : bytes
Vary : Accept-Encoding
Content-Encoding : gzip
X-UA-Compatible : IE=Edge
X-XSS-Protection : 1; mode=block
Content-Length : 2517
Keep-Alive : timeout=1, max=10
Connection : Keep-Alive
Content-Type : text/html
-------------------------------------------------- Status code --------------------------------------------------
403
-------------------------------------------------- Response content --------------------------------------------------
b'<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN" "http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd"> \n\n<html>\n\n<head>\n<title>NCBI - WWW Error 403 Diagnostic</title>\n<meta http-equiv="Content-Type" content="application/xhtml+xml; charset=utf-8" />\n<meta http-equiv="pragma" content="no-cache" />\n<meta http-equiv="cache-control" content="no-cache" />\n<meta name="robots" content="noarchive,none" />\n\n<style type="text/css">\nbody {\nmin-width: 950px;\n_width: 950px;\n}\nh1.error {color: red; font-size: 40pt}\n\n#mainContent {\npadding: 1em 2em;\n}\n\ndl#diags {\npadding: 0.5em 0.5em 1.5em 0.5em;\npadding-left: 2em;\nborder: solid 1px #888;\nborder-left: none;\nborder-right: none;\nmargin-bottom:0;\nbackground-color:#eeeeee;\ncolor:#666;\nfont-size: 80%;\n_font-size: 70%;\nfont-family: Verdana, sans-serif;\n}\n\ndl#diags dt {\nfloat: left;\nfont-weight: bold;\nwidth: auto;\n}\n\ndl#diags dd {\nmargin-left:1em;\nfloat: left;\nmargin-right: 2em;\n}\n\n#footer span {\nfloat: left;\ncolor: #888;\nfont-size: 80%;\n}\n\n#footer {\ntext-align: right;\npadding: 0 0.5em;\nclear: left;\n}\n\n#footer img {\nborder: none;\n}\n\n.ncbi {\nmargin: 0;\npadding:0;\nfont-size:240%;\nfont-weight: bold;\nfont-family: Times, serif;\ncolor: #336699;\nfloat: left;\ndisplay: inline;\n}\n\n.ncbi a {\ntext-decoration: none;\ncolor: #336699;\n}\n\n.ncbi a:visited {\ntext-decoration: none;\ncolor: #336699;\n}\n\n.ncbi a:hover {\ntext-decoration: underline;\n}\n\n.message {\nfont-family: Verdana, sans-serif;\nbackground-color: #336699;\ncolor: white;\npadding: .35em;\nmargin-left: 7em;\nmargin-top: .67em;\n_margin-top: 0.5em;\nfont-weight: bold;\nfont-size: 100%;\nmargin-bottom: 0;\n}\n\nh1 {\nclear: left;\nfont-size: 110%;\nfont-family: Verdana, sans-serif;\n}\n\n\nbody.denied {\nbackground-color: black;\ncolor: white;\n}\n\nbody.denied h1 {\ncolor: red;\n}\n\nbody.denied a {\ncolor: green;\n}\n\nbody.denied #footer, body.denied #diags {\ncolor: black;\n}\n\n#searchme:focus {\nbackground-color: #ffa;\n}\n\n.errurl {\nletter-spacing: 0;\nmargin: 0 1em;\npadding: 0.25em;\nbackground-color: #fff0f0;\ncolor: #c00;\nfont-family: "Courier New", Courier, monospace;\nfont-size: 90%;\n_font-size: 80%;\n}\n\nbody.denied .errurl {\nbackground-color: black;\ncolor: yellow;\n}\n\nspan.x {\ndisplay: none;\n}\n\n</style>\n\n<!--[if IE]>\n<script type="text/javascript">\n/* <![CDATA[ */\n_isIE=1;\n/* ]]> */\n</script>\n<![endif]-->\n\n<script type="text/javascript">\n/* <![CDATA[ */\naddEvent = function(obj, type, fn, b) {\n    if (obj.attachEvent) {\n        var name = "" + type + fn; \n        name = name.substring(0, name.indexOf("\\n"));\n        obj["e" + name] = fn;\n        obj[name] = function(){ obj["e" + name](window.event);}\n        obj.attachEvent("on" + type, obj[name]);\n    } else {\n        obj.addEventListener(type, fn, b);\n        return true;\n    }\n}\n\nfixlink = function() {\n   var ie = typeof(_isIE) != \'undefined\';\n   var v = document.getElementById("mlink");\n\n   if (v) {\n      var ieHACK = v.innerHTML;\n      var newText;\n\n      newText = "%3E%20REQUEST_URI%3D" + document.REQUEST_URI + "%0D" +\n                "%3E%20HTTP_HOST%3D" + document.HTTP_HOST + "%0D" +\n                "%3E%20REMOTE_ADDR%3D" + document.REMOTE_ADDR + "%0D" +\n                "%3E%20REQUEST_METHOD%3D" + document.REQUEST_METHOD + "%0D";\n\n      var href = v.getAttribute("href");\n      newText = newText.replace(/\'/g, "%27");\n      newText = newText.replace(/\\?/g, "%3F");\n\n      // IE bugfixes\n      if (ie) {\n          var m = /(.*)\\?(.*)/.exec(href);\n          if (m) {\n             href = m[1] + "?" + escape(m[2]);\n          }\n          href = href.replace(/(.*)(%0D-----%0D.*)/m, "$1" + newText + "$2");\n          href = href.replace(/subject%3D/, "subject=");\n          href = href.replace(/%26body%3D/, "&body=");\n          href=href.replace(/%0D/mg, "%0A");\n      } else {\n          href = href.replace(/(.*)(%0D-----%0D.*)/m, "$1" + newText + "$2");\n      }\n      v.setAttribute("href", href);\n      v.innerHTML = ieHACK;  // I can\'t believe this...\n   }  \n   v = document.getElementById("searchme");\n   if (v) { v.focus(); }\n}\n\n\naddEvent(window, "load", fixlink, false);\n/* ]]> */\n</script>\n\n\n</head>\n\n<body>\n\n<div id=\'header\'>\n<a href="#mainContent" title="Skip to main content" />\n<p class="ncbi"><a href="http://www.ncbi.nlm.nih.gov">NCBI</a></p>\n<p class="message">Error</p>\n</div>\n\n<div id=\'mainContent\'>\n<h1>The requested page has restricted access</h1>\n\n<p class=\'errurl\'>\n<script type="text/javascript">\n/* <![CDATA[ */\n  document.REQUEST_URI = "/tools/primer-blast/primertool.cgi";\n  document.HTTP_HOST = "www.ncbi.nlm.nih.gov";\n  document.REMOTE_ADDR = "192.41.130.86";\n  document.REQUEST_METHOD = "GET";\n\n  var p = "http://" + document.HTTP_HOST + document.REQUEST_URI;\n  p = p.replace(/(.)/g, "<span class=\\"x\\"> </span>$1");\n  p = p.replace(/\'/g, "&apos;");\n  document.write(p);\n//  document.write(p.replace(/[^A-Za-z0-9\\#\\;\\:\\.\\/\\?&\\=\\_\\- ]*/g, "<span class=\'x\'> </span>"));\n/* ]]> */\n</script>\n</p>\n\n<p>\nYou have requested a page which is not open to the public.  Your\nrequest did not meet the criteria required to grant access to this\npage.\n</p>\n\n<p>\nIf you believe that you are being denied access to the page in\nerror, send an e-mail problem report to <a id=\'mlink\'\nhref="mailto:info@ncbi.nlm.nih.gov?subject=NCBI%20Web%20site%20error%20403&amp;body=%3E%20Error%3D403%0D%3E%20Server%3D130.14.29.110%0D%3E%20Client%3D192.41.130.86%0D%3E%20Time%3DTuesday, 29-Aug-2017 14:25:39 EDT%0D%0D-----%0DPlease%20enter%20comments%20below:%0D%0D">info@ncbi.nlm.nih.gov</a>. \n</p>\n\n\n<form method="post" action="http://www.ncbi.nlm.nih.gov/gquery/gquery.fcgi"\n     enctype="application/x-www-form-urlencoded">\n<p>\n<strong>Search NCBI</strong>\n<input name="term" size="40" style="margin-left: 1em; border: solid 1px #336699;" id="searchme"/>\n<input type="submit" value="Go" />\n</p>\n</form>\n\n\n</div>\n\n<dl id="diags">\n   <dt>Error</dt><dd>403</dd>\n   <dt>Server</dt><dd>130.14.29.110</dd>\n   <dt>Client</dt><dd>192.41.130.86</dd>\n   <dt>Time</dt><dd>Tuesday, 29-Aug-2017 14:25:39 EDT</dd>\n</dl>\n\n<p id=\'footer\'>\n\n<span id=\'rev\'>Rev. 18 Jul 2017 16:09:24 EDT</span>\n<!-- <a href="http://validator.w3.org/check?uri=referer"><img src="http://www.w3.org/Icons/valid-xhtml11" alt="Valid XHTML 1.1!" /></a> -->\n</p>\n\n</body>\n</html>\n\n'
```
and the following from mitmproxy:

https://ibb.co/iUGF6k
https://ibb.co/nFamCQ

This time it's another 403 error

Try removing `Referer` as well.

Sadly it looks to be the same error with Referer header removed...

Script output:
```
/Users/james/anaconda/envs/requests/lib/python3.6/site-packages/urllib3/connectionpool.py:858: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecureRequestWarning)
-------------------------------------------------- Requests header --------------------------------------------------
User-Agent : Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.101 Safari/537.36
Accept-Encoding : gzip, deflate, br
Accept : text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8
Connection : keep-alive
Origin : https://www.ncbi.nlm.nih.gov
Upgrade-Insecure-Requests : 1
Content-Type : multipart/form-data; boundary=----WebKitFormBoundaryBflp51Ny9ReeA5A9
Accept-Language : en-US,en;q=0.8
Content-Length : 127
-------------------------------------------------- Response header --------------------------------------------------
Date : Tue, 29 Aug 2017 18:30:42 GMT
Server : Apache
Strict-Transport-Security : max-age=31536000; includeSubDomains; preload
Referrer-Policy : origin-when-cross-origin
Content-Security-Policy : upgrade-insecure-requests
Accept-Ranges : bytes
Vary : Accept-Encoding
Content-Encoding : gzip
X-UA-Compatible : IE=Edge
X-XSS-Protection : 1; mode=block
Content-Length : 2516
Keep-Alive : timeout=1, max=10
Connection : Keep-Alive
Content-Type : text/html
-------------------------------------------------- Status code --------------------------------------------------
403
-------------------------------------------------- Response content --------------------------------------------------
b'<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN" "http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd"> \n\n<html>\n\n<head>\n<title>NCBI - WWW Error 403 Diagnostic</title>\n<meta http-equiv="Content-Type" content="application/xhtml+xml; charset=utf-8" />\n<meta http-equiv="pragma" content="no-cache" />\n<meta http-equiv="cache-control" content="no-cache" />\n<meta name="robots" content="noarchive,none" />\n\n<style type="text/css">\nbody {\nmin-width: 950px;\n_width: 950px;\n}\nh1.error {color: red; font-size: 40pt}\n\n#mainContent {\npadding: 1em 2em;\n}\n\ndl#diags {\npadding: 0.5em 0.5em 1.5em 0.5em;\npadding-left: 2em;\nborder: solid 1px #888;\nborder-left: none;\nborder-right: none;\nmargin-bottom:0;\nbackground-color:#eeeeee;\ncolor:#666;\nfont-size: 80%;\n_font-size: 70%;\nfont-family: Verdana, sans-serif;\n}\n\ndl#diags dt {\nfloat: left;\nfont-weight: bold;\nwidth: auto;\n}\n\ndl#diags dd {\nmargin-left:1em;\nfloat: left;\nmargin-right: 2em;\n}\n\n#footer span {\nfloat: left;\ncolor: #888;\nfont-size: 80%;\n}\n\n#footer {\ntext-align: right;\npadding: 0 0.5em;\nclear: left;\n}\n\n#footer img {\nborder: none;\n}\n\n.ncbi {\nmargin: 0;\npadding:0;\nfont-size:240%;\nfont-weight: bold;\nfont-family: Times, serif;\ncolor: #336699;\nfloat: left;\ndisplay: inline;\n}\n\n.ncbi a {\ntext-decoration: none;\ncolor: #336699;\n}\n\n.ncbi a:visited {\ntext-decoration: none;\ncolor: #336699;\n}\n\n.ncbi a:hover {\ntext-decoration: underline;\n}\n\n.message {\nfont-family: Verdana, sans-serif;\nbackground-color: #336699;\ncolor: white;\npadding: .35em;\nmargin-left: 7em;\nmargin-top: .67em;\n_margin-top: 0.5em;\nfont-weight: bold;\nfont-size: 100%;\nmargin-bottom: 0;\n}\n\nh1 {\nclear: left;\nfont-size: 110%;\nfont-family: Verdana, sans-serif;\n}\n\n\nbody.denied {\nbackground-color: black;\ncolor: white;\n}\n\nbody.denied h1 {\ncolor: red;\n}\n\nbody.denied a {\ncolor: green;\n}\n\nbody.denied #footer, body.denied #diags {\ncolor: black;\n}\n\n#searchme:focus {\nbackground-color: #ffa;\n}\n\n.errurl {\nletter-spacing: 0;\nmargin: 0 1em;\npadding: 0.25em;\nbackground-color: #fff0f0;\ncolor: #c00;\nfont-family: "Courier New", Courier, monospace;\nfont-size: 90%;\n_font-size: 80%;\n}\n\nbody.denied .errurl {\nbackground-color: black;\ncolor: yellow;\n}\n\nspan.x {\ndisplay: none;\n}\n\n</style>\n\n<!--[if IE]>\n<script type="text/javascript">\n/* <![CDATA[ */\n_isIE=1;\n/* ]]> */\n</script>\n<![endif]-->\n\n<script type="text/javascript">\n/* <![CDATA[ */\naddEvent = function(obj, type, fn, b) {\n    if (obj.attachEvent) {\n        var name = "" + type + fn; \n        name = name.substring(0, name.indexOf("\\n"));\n        obj["e" + name] = fn;\n        obj[name] = function(){ obj["e" + name](window.event);}\n        obj.attachEvent("on" + type, obj[name]);\n    } else {\n        obj.addEventListener(type, fn, b);\n        return true;\n    }\n}\n\nfixlink = function() {\n   var ie = typeof(_isIE) != \'undefined\';\n   var v = document.getElementById("mlink");\n\n   if (v) {\n      var ieHACK = v.innerHTML;\n      var newText;\n\n      newText = "%3E%20REQUEST_URI%3D" + document.REQUEST_URI + "%0D" +\n                "%3E%20HTTP_HOST%3D" + document.HTTP_HOST + "%0D" +\n                "%3E%20REMOTE_ADDR%3D" + document.REMOTE_ADDR + "%0D" +\n                "%3E%20REQUEST_METHOD%3D" + document.REQUEST_METHOD + "%0D";\n\n      var href = v.getAttribute("href");\n      newText = newText.replace(/\'/g, "%27");\n      newText = newText.replace(/\\?/g, "%3F");\n\n      // IE bugfixes\n      if (ie) {\n          var m = /(.*)\\?(.*)/.exec(href);\n          if (m) {\n             href = m[1] + "?" + escape(m[2]);\n          }\n          href = href.replace(/(.*)(%0D-----%0D.*)/m, "$1" + newText + "$2");\n          href = href.replace(/subject%3D/, "subject=");\n          href = href.replace(/%26body%3D/, "&body=");\n          href=href.replace(/%0D/mg, "%0A");\n      } else {\n          href = href.replace(/(.*)(%0D-----%0D.*)/m, "$1" + newText + "$2");\n      }\n      v.setAttribute("href", href);\n      v.innerHTML = ieHACK;  // I can\'t believe this...\n   }  \n   v = document.getElementById("searchme");\n   if (v) { v.focus(); }\n}\n\n\naddEvent(window, "load", fixlink, false);\n/* ]]> */\n</script>\n\n\n</head>\n\n<body>\n\n<div id=\'header\'>\n<a href="#mainContent" title="Skip to main content" />\n<p class="ncbi"><a href="http://www.ncbi.nlm.nih.gov">NCBI</a></p>\n<p class="message">Error</p>\n</div>\n\n<div id=\'mainContent\'>\n<h1>The requested page has restricted access</h1>\n\n<p class=\'errurl\'>\n<script type="text/javascript">\n/* <![CDATA[ */\n  document.REQUEST_URI = "/tools/primer-blast/primertool.cgi";\n  document.HTTP_HOST = "www.ncbi.nlm.nih.gov";\n  document.REMOTE_ADDR = "192.41.130.86";\n  document.REQUEST_METHOD = "GET";\n\n  var p = "http://" + document.HTTP_HOST + document.REQUEST_URI;\n  p = p.replace(/(.)/g, "<span class=\\"x\\"> </span>$1");\n  p = p.replace(/\'/g, "&apos;");\n  document.write(p);\n//  document.write(p.replace(/[^A-Za-z0-9\\#\\;\\:\\.\\/\\?&\\=\\_\\- ]*/g, "<span class=\'x\'> </span>"));\n/* ]]> */\n</script>\n</p>\n\n<p>\nYou have requested a page which is not open to the public.  Your\nrequest did not meet the criteria required to grant access to this\npage.\n</p>\n\n<p>\nIf you believe that you are being denied access to the page in\nerror, send an e-mail problem report to <a id=\'mlink\'\nhref="mailto:info@ncbi.nlm.nih.gov?subject=NCBI%20Web%20site%20error%20403&amp;body=%3E%20Error%3D403%0D%3E%20Server%3D130.14.29.110%0D%3E%20Client%3D192.41.130.86%0D%3E%20Time%3DTuesday, 29-Aug-2017 14:30:42 EDT%0D%0D-----%0DPlease%20enter%20comments%20below:%0D%0D">info@ncbi.nlm.nih.gov</a>. \n</p>\n\n\n<form method="post" action="http://www.ncbi.nlm.nih.gov/gquery/gquery.fcgi"\n     enctype="application/x-www-form-urlencoded">\n<p>\n<strong>Search NCBI</strong>\n<input name="term" size="40" style="margin-left: 1em; border: solid 1px #336699;" id="searchme"/>\n<input type="submit" value="Go" />\n</p>\n</form>\n\n\n</div>\n\n<dl id="diags">\n   <dt>Error</dt><dd>403</dd>\n   <dt>Server</dt><dd>130.14.29.110</dd>\n   <dt>Client</dt><dd>192.41.130.86</dd>\n   <dt>Time</dt><dd>Tuesday, 29-Aug-2017 14:30:42 EDT</dd>\n</dl>\n\n<p id=\'footer\'>\n\n<span id=\'rev\'>Rev. 18 Jul 2017 16:09:24 EDT</span>\n<!-- <a href="http://validator.w3.org/check?uri=referer"><img src="http://www.w3.org/Icons/valid-xhtml11" alt="Valid XHTML 1.1!" /></a> -->\n</p>\n\n</body>\n</html>\n\n'
```
mitmproxy output:

https://ibb.co/gfhisQ
https://ibb.co/g59UXQ

Is it weird that when setting the header variable in my POST request, I end up with less fields in my response header (than not sending a header variable)?

Ok, new plan. Make one change at a time until the 200 changes to a 403: that'll identify the bad header.

Okay, the bad header seems to be:

'Content-Type' : 'multipart/form-data; boundary=----WebKitFormBoundaryBflp51Ny9ReeA5A9'

When I use all other combinations of headers apart from that I get 200, as soon as I introduce that header I get a 403.

Script output (without Content-Type header):

```
/Users/james/anaconda/envs/requests/lib/python3.6/site-packages/urllib3/connectionpool.py:858: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecureRequestWarning)
-------------------------------------------------- Requests header --------------------------------------------------
User-Agent : Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.101 Safari/537.36
Accept-Encoding : gzip, deflate, br
Accept : text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8
Connection : keep-alive
Origin : https://www.ncbi.nlm.nih.gov
Upgrade-Insecure-Requests : 1
Referer : https://www.ncbi.nlm.nih.gov/tools/primer-blast/index.cgi?LINK_LOC=reset
Accept-Language : en-US,en;q=0.8
Cookie : sv-userdata-key-www.ncbi.nlm.nih.gov=G5KxXzyQ81U_vs1aHK_7XDWciF1B8AjjDUmDunVbhIZhZ4p4t_SVK4ASpbTT8iDSJVcxBH9oUAB3K2xNWjp3G0koYCloBlYuSxdoIGIkYzl2; ncbi_sid=0751457F9A561D01_0000SID; _ga=GA1.2.567134514.1503994317; _gid=GA1.2.654339584.1503994317; _gat=1; starnext=MYGwlsDWB2CmAeAXAXAbgK7RAewIYBM4lkAmAXgAcAnMAW1ioCMRcBnRAMgBYzm3FWsXFWAALDgEZymHAUkBOMgAYA7AFYpXFQDF5AQTUA2ACIBRFRKVXrN2xI4klygMJcSltQA59R0wGY/S1tg63t3Shp6JhZ2AFI/PQA5AHlE03i9PnZBYTEMlLSHcgB3UoA6aGBGMAqQWgqwUTKAc2wANwceajoGLMR81NMHQwie6P4BtIy+nJFxEhVRqL7J9ISZoTnVh08l3pj+lWcC9KON3NFYo5OHRQkuLiUOPyd5K2eJMnvH5/JPDWefjIADNcCBBM8eIgqOhYM81F83GpniMJH4SPJnosSFxnrtAoZ5Li/IoXp5DCpuE50RIpNxPlIAtxyNBcIgwG04Q8yDI8IQEJwuAiSBw1EDvk81DxPEo/KKEfIRUYyCRDIZRYtJbtaT81HcOIYnE9DAycUA=
Content-Length : 127
Content-Type : application/x-www-form-urlencoded
-------------------------------------------------- Response header --------------------------------------------------
Date : Tue, 29 Aug 2017 18:43:10 GMT
Server : Apache
Strict-Transport-Security : max-age=31536000; includeSubDomains; preload
Referrer-Policy : origin-when-cross-origin
Content-Security-Policy : upgrade-insecure-requests
Cache-Control : private
NCBI-PHID : 0C422DEB9A5B5BE10000000000000001.m_3
NCBI-RCGI-JobStatus : Done
NCBI-SID : 0751457F9A561D01_0000SID
Access-Control-Allow-Methods : POST, GET, PUT, OPTIONS, PATCH, DELETE
Access-Control-Allow-Origin : https://www.ncbi.nlm.nih.gov
Access-Control-Allow-Credentials : true
Access-Control-Allow-Headers : Origin,X-Accept-Charset,X-Accept,Content-Type,X-Requested-With,NCBI-SID,NCBI-PHID
Content-Type : text/html
Set-Cookie : ncbi_sid=0751457F9A561D01_0000SID; domain=.nih.gov; path=/; expires=Wed, 29 Aug 2018 18:43:10 GMT
Vary : Accept-Encoding
Content-Encoding : gzip
X-UA-Compatible : IE=Edge
X-XSS-Protection : 1; mode=block
Keep-Alive : timeout=1, max=10
Connection : Keep-Alive
Transfer-Encoding : chunked
-------------------------------------------------- Status code --------------------------------------------------
200
-------------------------------------------------- Response content --------------------------------------------------
b'<html><head><title>Empty Result</title></head><body>Empty Result</body></html>'
```
mitmproxy output (without Content-Type header):

https://ibb.co/ekPMCQ
https://ibb.co/keZMCQ


Oh, yeah, ok, let's skip that header. :grin: Want to add in all the others?

That output is from a run where all the others have been added, except the Content-Type header

Oh, I see you did. So, the URL isn't being sent. This seems like a problem with the interaction with the service, but it does not appear to be a Requests problem, and it's a bit beyond my ability to fix. 😞 

If I were interacting with the service I could probably do it, but I think that's going above-and-beyond for OSS maintenance. :wink:

Yes you're right, thank you for helping and being patient with my issue. I'll contact the maintainers of the service and see if they can offer up some more help. Many thanks!

No problem at all! Good luck! :sparkles: