POST Request in Heroku - 403 Forbidden

I'm new to using Requests, and I'm trying to scrape a schedule of classes. My POST request has resulted in "403 Forbidden" on Heroku, but works fine on my local machine. Here's a code snippet I'm having trouble with in my application, using Python 2.7.4, Bottle, BeautifulSoup4, and of course, Requests.

``` python
import requests
from bs4 import BeautifulSoup as Soup

url = "https://telebears.berkeley.edu/enrollment-osoc/osc"
code = "26187"
values = dict(_InField1 = "RESTRIC", _InField2 = code, _InField3 = "13D2")
html = requests.post(url, params=values)
soup = Soup(html.content, from_encoding="utf-8")

sp = soup.find_all("div", {"class" : "layout-div"})[2]
print sp.text
```

It gives me back the string "Computer Science 61A P 001 LEC:" in development. However, when I tried to run it on Heroku (using `heroku run bash` and then run `python`), I got back "403 Forbidden".

My attempts so far include manually setting User Agent headers, changing to `verify=False`, but neither worked. At first I thought it's the school settings, but then I was wondering why it works locally without any problem... Any explanation/suggestion would be really appreciated! Thanks in advance.


Are you aware that `params` here refers to query string parameters? (I wouldn't ask if it wasn't something that seems to frequently confuse new users.) In other words, your request looks a post without a body to the following url:

```
https://telebears.berkeley.edu/enrollment-osoc/osc?_InField1=RESTRIC&_InField2=26187&_InField3=13D2
```

If instead you wanted a post to `https://telebears.berkeley.edu/enrollment-osoc/osc` with the body `_InField1=RESTRIC&_InField2=code&_InField3=13D2` then you need to pass your `values` dictionary to the `data` parameter like so:

```
requests.post(url, data=values)
```


I was not aware of that at all - thanks for the suggestion! Unfortunately I still got the same error after changing `params` to `data`.


@kqdtran I opened that URL in the browser and received the same 403 error. Do you authenticate before getting to that page? If so you need to login with python/requests first.


I don't think you need to authenticate before getting there, but I believe the page can only be accessed via a post request, like in the following code:

``` python
<form action="https://telebears.berkeley.edu/enrollment-osoc/osc" method="post" target="_blank"/>
   <input type="hidden" name="_InField1" value="RESTRIC"/>
   <input type="hidden" name="_InField2" value="8461"/>
   <input type="hidden" name="_InField3" value="13D2"/>
   <input type="submit" value="Enrollment Information" />
</form>
```

Thanks!


If it helps, here are my attempts at reproducing the error: 

![1mycomp](https://f.cloud.github.com/assets/2239263/990736/3dd1297a-093c-11e3-82b2-04c53830a444.png)

![2heroku](https://f.cloud.github.com/assets/2239263/990738/413812fe-093c-11e3-93c5-911f7917b58f.png)


So if I recall correctly there is a serious bug fix in python 2.7.5 with SSL. I could be wrong though. Try upgrading your other version of python to 2.7.5.


Ohhhhh... If that's true, it makes a lot of sense. Unfortunately, I can't upgrade the other version, since it belongs to heroku... I could try to switch to Python 3 though. Will keep you posted, and thanks for all the help!


@kennethreitz any idea when Heroku will upgrade to python 2.7.5? I thought it was already using that.


Sorry I'll take it back! By default, Heroku is still using Python 2.7.4. But if you provide a runtime.txt with the string "python-2.7.5" inside, it will be using 2.7.5. Unfortunately, I still have the same error under this version :(

![3herokustill](https://f.cloud.github.com/assets/2239263/990766/e14333b2-093e-11e3-98b4-8be6c802d3c6.png)


@kqdtran I just looked at this again with python 2.7.3 (so still not exactly what you're using but close):

``` pycon
>>> import requests
>>> r = requests.post('https://telebears.berkeley.edu/enrollment-osoc/osc', data={'_InField1': 'RESTRIC', '_InField2': 'code', '_InField3': '13d2'})
>>> r
<Response [200]>
```

Are you still having this issue? It works with curl too:

``` bash
~$ curl -i -X POST -d '_InField1=RESTRIC&_InField2=code&_InField3=13D2' https://telebears.berkeley.edu/enrollment-osoc/osc
HTTP/1.1 200 OK
Date: Tue, 24 Sep 2013 02:57:17 GMT
Content-Type: text/html;charset=UTF-8
Content-Length: 1801
Connection: close
```


Ah right, forgot this was a heroku only issue.


Humorously enough that curl command gets a 403 on Heroku's servers as well

```
~ $ curl -i -X POST -d '_InField1=RESTRIC&_InField2=code&_InField3=13D2' https://telebears.berkeley.edu/enrollment-osoc/osc
HTTP/1.1 403 Forbidden
Date: Tue, 24 Sep 2013 03:07:17 GMT
Content-Length: 310
Connection: close
Content-Type: text/html; charset=iso-8859-1

<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>403 Forbidden</title>
</head><body>
<h1>Forbidden</h1>
<p>You don't have permission to access /enrollment-osoc/osc
on this server.</p>
<hr>
<address>Apache/2.2.3 (Red Hat) Server at telebears.berkeley.edu Port 443</address>
</body></html>
```

So this is a heroku issue and not a python-requests issue.


Thanks for helping me confirming the problem, @sigmavirus24!


@kqdtran did you ever figure out this issue? I'm facing a similar Heroku-specific problem with a GET request. 


@cvolawless can you try making the same request as if it were a curl request? If you receive the same response as requests, you should contact Heroku support. If you get a different response feel free to open a new issue so @Lukasa and I can be of use. :)


@cvolawless also, if you can't discuss the issue publicly, @Lukasa and I both have our emails on our profile pages, so you can send us an email as an alternative for private (confidential) support.


@sigmavirus24 yup, curl fails as well. Have a ticket open to Heroku--will update if I find out anything helpful. 


Let us know if we can be of any assistance with Heroku's support. :)


been playing around with the javascript Todoapp and using my own server url. I find that login/logout doesn't work on versions 1.1.12-min.js. I upgrated to 1.6.14 or even parse-latest.js. Both don't have any reference ParseCollections. It's been very hard figuring out how to fix it.

2016-04-25T09:53:19.298790+00:00 heroku[router]: at=info method=POST path="/parse/1/users" host=intense-hollows-60991.herokuapp.com request_id=9cb5eaf9-603e-4b98-9f78-35058ca0dfd6 fwd="5.148.6.116" dyno=web.1 connect=0ms service=51ms status=403 bytes=466/not well-formed

2016-04-25T09:54:06.692075+00:00 heroku[router]: at=info method=POST path="/parse/1/login" host=intense-hollows-60991.herokuapp.com request_id=d934f856-85be-4315-baf4-7bb64ac501c8 fwd="5.148.6.116" dyno=web.1 connect=1ms service=14ms status=403 bytes=466/not well-formed

http://www.mobileparadigm.co.uk/urls/cft_access/services/cft/


@arkangelx I'm sorry, what specifically is your issue and what does it have to do with the requests library?


i have pointed my todoapp to my own Parse.serverUrl as instructed, however I can’t perform a login/sign up now. I can save objects easily to my account with mLabs. So in essence

 Parse.$ = jQuery;

  // Initialize Parse with your Parse application javascript keys
     Parse.initialize("5N5FqrkAmn1C8g6hVHvb","aFZ7RUqpgrUVZrDKfM6f”); // this works using the standard Parse sdk WITHOUT hosting myself (login/signup save data)
adding the line below to point to the heroic setup after importing all my data from my dashboard now doesn’t allow me to sign up

Logs

2016-04-25T09:54:06.692075+00:00 heroku[router]: at=info method=POST path="/parse/1/login" host=intense-hollows-60991.herokuapp.com request_id=d934f856-85be-4315-baf4-7bb64ac501c8 fwd="5.148.6.116" dyno=web.1 connect=1ms service=14ms status=403 bytes=466

So I tried to change the version of the parse sdk I’m using from 1.1.12-min.js to 1.6.14-min.js or parse-latest.js, this throws an error saying that Parse.Collections isn’t referenced.

   Parse.serverURL = 'https://intense-hollows-60991.herokuapp.com/parse/' <https://intense-hollows-60991.herokuapp.com/parse/'> (login/signup save data)

> On 25 Apr 2016, at 11:27, Cory Benfield notifications@github.com wrote:
> 
> @arkangelx https://github.com/arkangelx I'm sorry, what specifically is your issue and what does it have to do with the requests library?
> 
> —
> You are receiving this because you were mentioned.
> Reply to this email directly or view it on GitHub https://github.com/kennethreitz/requests/issues/1542#issuecomment-214251913


@arkangelx That really seems to have nothing to do with the Requests library. The sample code you're providing appears to be Javascript code, which of course is not applicable to this Python library. So once again: where does the Python Requests library come into your problem?


My apologies, I posted to the wrong session

> On 25 Apr 2016, at 11:35, Cory Benfield notifications@github.com wrote:
> 
> @arkangelx https://github.com/arkangelx That really seems to have nothing to do with the Requests library. The sample code you're providing appears to be Javascript code, which of course is not applicable to this Python library. So once again: where does the Python Requests library come into your problem?
> 
> —
> You are receiving this because you were mentioned.
> Reply to this email directly or view it on GitHub https://github.com/kennethreitz/requests/issues/1542#issuecomment-214257644


Just seeing this — my best guess is that the server in question is blocking requests from AWS (which Heroku is hosted on). Many edge CDNs (and I assume also standard security appliances) offer that as a feature, since it's ultra easy to spin up a botnet (or just socially unacceptable campaigns) there.

e.g. there's nothing un-vanilla about Heroku's Python, so I highly doubt it's not something network related. 


any way to get around this? An Uber API is giving my node app hosted on AWS EC2 403s...

There's a heroku addon called "priximo" which allows you to proxy your way out

There's another one too

do you know how to set it up properly? my app still gets a 403 when it sends a request to the Uber API

I'm shutting down this conversation so that well over 1000 folks don't get email notifications for an issue unrelated to Requests.