Can't use a way to wait until HTML content response is completely saved

I am using requests to create a get request and wanting to display the whole html content:
import requests
```
url = 'http://website.net/thelink'
headers = {"User-agent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.80 Safari/537.36"}
req = requests.get(url,headers=headers)

```
However when using print, does not output the html correctly in a section:

```
print req.content
<div id='error' class='tabdata'>
Please wait while we validate your browser.
</div>
```

**I have seen this div/tag shown already on the browser and eventually the content displays. What I need is to wait on the get request for a little while and then save the content response. Any ideas how to do that ?** 

What is the *real* URL you try to access? Please note that Requests does not run JavaScript code that may be included in the page. It is not a web browser!

This one **http://bgp.he.net/AS36375**

Well, that is some unfriendly service. Using `curl` or `wget` I get a `403 Forbidden`. Only a 'real' browser (or what this site thinks is one) is getting any HTML result at all:

    $ curl -v -H 'User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.80 Safari/537.36' http://bgp.he.net/AS36375
    *   Trying 72.52.94.234...
    * Connected to bgp.he.net (72.52.94.234) port 80 (#0)
    > GET /AS36375 HTTP/1.1
    > Host: bgp.he.net
    > Accept: */*
    > User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.80 Safari/537.36
    >
    < HTTP/1.1 302 Found
    < Date: Wed, 01 Mar 2017 21:05:02 GMT
    < Server: Apache/2.4.7 (Ubuntu)
    < Cache-Control: no-cache
    < X-Runtime: 1
    < X-Powered-By: Phusion Passenger 5.0.30
    < Set-Cookie: c=; path=/; expires=Thu, 01-Jan-1970 00:00:00 GMT
    < Set-Cookie: path=BAgiDS9BUzM2Mzc1--8b1b2f0af47aa09b29bfa2d05c37c7456a16a074; path=/
    < Location: http://bgp.he.net/cc
    < Content-Length: 86
    < Status: 302 Found
    < Content-Type: text/html; charset=utf-8
    <
    * Connection #0 to host bgp.he.net left intact
    <html><body>You are being <a href="http://bgp.he.net/cc">redirected</a>.</body></html>

Request will get the same result unless it follows the redirect to http://bgp.he.net/cc. You don't have a problem, everything is working fine (except the server being stupid).

This is what I get using requests

```
>>> req = requests.get(url,headers=headers, allow_redirects=True)
>>> print req.content
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
 "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<!-- rmosher 2010 - 2016 -->
<meta http-equiv="Content-type" content="text/html;charset=UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<script src="/javascripts/jquery/jquery-1.3.2.js?1235084520" type="text/javascript"></script>
<script src="/javascripts/jquery/jquery.history.js?1364589087" type="text/javascript"></script>
<script src="/javascripts/jquery/jquery-ui.js?1269850573" type="text/javascript"></script>
<script src="/javascripts/jquery/jrails.js?1269850578" type="text/javascript"></script>
<script src="/javascripts/bgp.js?1260526324" type="text/javascript"></script>
<link href="/stylesheets/bgp.css?1455825849" media="all" rel="stylesheet" type="text/css" />


<script src="/javascripts/jstest.js?1442294748" type="text/javascript"></script>
<meta http-equiv="refresh" content="15; url=/jf">


</head>

<body>
	<div id='header'>
		<a href="http://www.he.net/"><img alt='Hurricane Electric' src='/helogo.gif' /></a>
		<form action="/search" method="get">
			<div class='search'>
			<input id="search_search" name="search[search]" size="15" type="text" />
			<input name="commit" type="submit" value="Search" />
		</div>
		</form>
		
		<div class='clear'></div>
		<div class='floatleft'>
			<div class='leftsidemenu'>
				<div class='menuheader'>Quick Links</div>
				<ul class='leftsidemenuitems'>
					<li><a href='http://bgp.he.net/'>BGP Toolkit Home</a></li>
					<li><a href="/report/prefixes">BGP Prefix Report</a></li>
					<li><a href="/report/peers">BGP Peer Report</a></li>
					<li><a href="/report/exchanges">Exchange Report</a></li>
					<li><a href="/report/bogons">Bogon Routes</a></li>
					<li><a href="/report/world">World Report</a></li>
					<li><a href="/report/multi-origin-routes">Multi Origin Routes</a></li>
					<li><a href="/report/dns">DNS Report</a></li>
					<li><a href="/report/tophosts">Top Host Report</a></li>
					<li><a href="/report/netstats">Internet Statistics</a></li>
					<li><a href='http://lg.he.net/'>Looking Glass</a></li>
					<li><a href='http://networktools.he.net/'>Network Tools App</a></li>
					<li><a href='http://tunnelbroker.net/'>Free IPv6 Tunnel</a></li>
					<li><a href='http://ipv6.he.net/certification/'>IPv6 Certification</a></li>
					<li><a href='http://bgp.he.net/ipv6-progress-report.cgi'>IPv6 Progress</a></li>
					<li><a href='http://bgp.he.net/going-native.pdf'>Going Native</a></li>
					<li><a href='http://bgp.he.net/contact/'>Contact Us</a></li>
				</ul>
	
			</div>
			<div class='clear'></div>
			<div class='social'>
			
				<a href="/r/Twitter" title="Hurricane Electric on Twitter"><img alt="Hurricane Electric on Twitter" src="/images/twitter.png?1215539178" /></a>
			
				<a href="/r/Facebook" title="Hurricane Electric on Facebook"><img alt="Hurricane Electric on Facebook" src="/images/facebook.png?1215539178" /></a>
			
			</div>
		</div>
	</div>

	<div id='content'>
			
		

		



<div class='clear'></div>

<div id='error' class='tabdata'>
Please wait while we validate your browser.
</div>
<script type='text/javascript'>
var _0xb539=["\x62\x67\x70\x2E\x68\x65\x2E\x6E\x65\x74\x20\x72\x65\x71\x75\x69\x72\x65\x73\x20\x6A\x61\x76\x61\x73\x63\x72\x69\x70\x74\x20\x61\x6E\x64\x20\x63\x6F\x6F\x6B\x69\x65\x73\x20\x74\x6F\x20\x66\x75\x6E\x63\x74\x69\x6F\x6E\x2E\x20\x20\x50\x6C\x65\x61\x73\x65\x20\x65\x6E\x61\x62\x6C\x65\x20\x74\x68\x65\x73\x65\x20\x69\x6E\x20\x79\x6F\x75\x72\x20\x62\x72\x6F\x77\x73\x65\x72\x2E","\x74\x65\x78\x74","\x23\x65\x72\x72\x6F\x72","\x68\x61\x73\x68","\x6C\x6F\x63\x61\x74\x69\x6F\x6E","\x3F\x68\x3D","\x72\x65\x73\x70\x6F\x6E\x73\x65","\x70\x61\x74\x68","\x63\x6F\x6F\x6B\x69\x65","\x6A\x73\x74\x65\x73\x74","\x70\x6F\x73\x74","\x61\x6A\x61\x78"];function printerror(){$(_0xb539[2])[_0xb539[1]](_0xb539[0])}function doredirect(_0x1cc4x3){url='/cr';if(window[_0xb539[4]][_0xb539[3]]){url+=_0xb539[5]+encodeURIComponent(window[_0xb539[4]][_0xb539[3]])};window[_0xb539[4]]=url;}$(function(){$[_0xb539[11]]({url:'/i',dataType:_0xb539[1],complete:function(_0x1cc4x3){ip=_0x1cc4x3[_0xb539[6]];$[_0xb539[11]]({url:'/jc',data:{p:$[_0xb539[9]]($[_0xb539[8]](_0xb539[7])),i:$[_0xb539[9]](ip)},type:_0xb539[10],error:printerror,complete:doredirect});},error:printerror})});
</script>



	</div>
	
	<div id='footer'>
	Updated 28 Feb 2017 13:45 PST &copy; 2017 Hurricane Electric
	</div>
	
	<script type="text/javascript">
		var gaJsHost = (("https:" == document.location.protocol) ? "https://ssl." : "http://www.");
		document.write(unescape("%3Cscript src='" + gaJsHost + "google-analytics.com/ga.js' type='text/javascript'%3E%3C/script%3E"));
	</script>
	<script type="text/javascript">
		try {
			var pageTracker = _gat._getTracker("UA-12276073-1");
			pageTracker._trackPageview();
		} catch(err) {}
	</script>
</body>
</html>
```

I need what it's inside this div

```
<div id='error' class='tabdata'>
Please wait while we validate your browser.
</div>
<script type='text/javascript'>
var _0xb539=["\x62\x67\x70\x2E\x68\x65\x2E\x6E\x65\x74\x20\x72\x65\x71\x75\x69\x72\x65\x73\x20\x6A\x61\x76\x61\x73\x63\x72\x69\x70\x74\x20\x61\x6E\x64\x20\x63\x6F\x6F\x6B\x69\x65\x73\x20\x74\x6F\x20\x66\x75\x6E\x63\x74\x69\x6F\x6E\x2E\x20\x20\x50\x6C\x65\x61\x73\x65\x20\x65\x6E\x61\x62\x6C\x65\x20\x74\x68\x65\x73\x65\x20\x69\x6E\x20\x79\x6F\x75\x72\x20\x62\x72\x6F\x77\x73\x65\x72\x2E","\x74\x65\x78\x74","\x23\x65\x72\x72\x6F\x72","\x68\x61\x73\x68","\x6C\x6F\x63\x61\x74\x69\x6F\x6E","\x3F\x68\x3D","\x72\x65\x73\x70\x6F\x6E\x73\x65","\x70\x61\x74\x68","\x63\x6F\x6F\x6B\x69\x65","\x6A\x73\x74\x65\x73\x74","\x70\x6F\x73\x74","\x61\x6A\x61\x78"];function printerror(){$(_0xb539[2])[_0xb539[1]](_0xb539[0])}function doredirect(_0x1cc4x3){url='/cr';if(window[_0xb539[4]][_0xb539[3]]){url+=_0xb539[5]+encodeURIComponent(window[_0xb539[4]][_0xb539[3]])};window[_0xb539[4]]=url;}$(function(){$[_0xb539[11]]({url:'/i',dataType:_0xb539[1],complete:function(_0x1cc4x3){ip=_0x1cc4x3[_0xb539[6]];$[_0xb539[11]]({url:'/jc',data:{p:$[_0xb539[9]]($[_0xb539[8]](_0xb539[7])),i:$[_0xb539[9]](ip)},type:_0xb539[10],error:printerror,complete:doredirect});},error:printerror})});
</script>
```

That's not a problem request can help you with. Use a HTML parser like https://www.crummy.com/software/BeautifulSoup/ or http://lxml.de/lxmlhtml.html.

BeautifulSoup will parse only after opening and reading the content, so it doesn't work.

Requests cannot run the JavaScript for you. If you need to run JavaScript (and it looks like you do), then you'll need to use something like selenium that can automate a web browser for you. 

@Lukasa I was just going to that direction. Thanks.

I'm going to close this, as this isn't an issue with Requests. Cheers @brangi @lutzhorn!

> @Lukasa I was just going to that direction. Thanks.

@brangi did you solved this?