Python Requests Tutorial - GeeksforGeeks

12 Mar, 2020
Requests library is one of the integral part of Python for making HTTP requests to a specified URL. Whether it be REST APIs or Web Scrapping, requests is must to be learned for proceeding further with these technologies. When one makes a request to a URI, it returns a response. Python requests provides inbuilt functionalities for managing both the request and response.
Why learn Python requests module?
Installing Requests
Making a Request
Http Request Methods
Response object
Response Methods
Authentication using Python Requests
SSL Certificate Verification
Session Objects
Requests is an Apache2 Licensed HTTP library, that allows to send HTTP/1.1 requests using Python.
To play with web, Python Requests is must. Whether it be hitting APIs, downloading entire facebook pages, and much more cool stuff, one will have to make a request to the URL.
Requests play a major role is dealing with REST APIs, and Web Scrapping.
Checkout an Example Python Script using Requests and Web Scrapping – Implementing Web Scraping in Python with BeautifulSoupRecent Articles on Requests !!Installing RequestsRequests installation depends on type of operating system on eis using, the basic command anywhere would be to open a command terminal and run,pip install requestsThe basic method for installation of requests on any operating system is to grab the base files and install requests manually and Requests is actively developed on GitHub, where the code is always available. For code – visit here.You can either clone the public repository:git clone git://github.com/psf/requests.gitOnce you have a copy of the source, you can embed it in your own Python package, or install it into your site-packages easily:cd requests
pip install .For more checkout – How to install requests in Python – For windows, linux, macMaking a RequestPython requests module has several built-in methods to make Http requests to specified URI using GET, POST, PUT, PATCH or HEAD requests. A Http request is meant to either retrieve data from a specified URI or to push data to a server. It works as a request-response protocol between a client and a server. Let’s demonstrate how to make a GET request to an endpoint.GET method is used to retrieve information from the given server using a given URI. The GET method sends the encoded user information appended to the page request. The page and the encoded information are separated by the ‘?’ character.For example:https://www.google.com/search?q=helloHow to make GET request through Python RequestsPython’s requests module provides in-built method called get() for making a GET request to a specified URI.Syntax –requests.get(url, params={key: value}, args)
Example –Let’s try making a request to github’s APIs for example purposes.import requests   # Making a GET requestr = requests.get('https://api.github.com/users/naveenkrnl')  # check status code for response received# success code - 200print(r)  # print content of requestprint(r.content)save this file as request.py and through terminal run,python request.pyOutput –For more, visit GET method – Python requestsHttp Request Methods –MethodDescriptionGETGET method is used to retrieve information from the given server using a given URI.POSTPOST request method requests that a web server accepts the data enclosed in the body of the request message, most likely for storing itPUTThe PUT method requests that the enclosed entity be stored under the supplied URI. If the URI refers to an already existing resource, it is modified and if the URI does not point to an existing resource, then the server can create the resource with that URI.DELETEThe DELETE method deletes the specified resourceHEADThe HEAD method asks for a response identical to that of a GET request, but without the response body.PATCHIt is used for modify capabilities. The PATCH request only needs to contain the changes to the resource, not the complete resourceResponse objectWhen one makes a request to a URI, it returns a response. This Response object in terms of python is returned by requests.method(), method being – get, post, put, etc. Response is a powerful object with lots of functions and attributes that assist in normalizing data or creating ideal portions of code. For example, response.status_code returns the status code from the headers itself, and one can check if the request was processed successfully or not.Response object can be used to imply lots of features, methods, and functionalities.Example :# import requests moduleimport requests  # Making a get requestresponse = requests.get('https://api.github.com/')  # print request objectprint(response.url)  # print status codeprint(response.status_code)Save this file as request.py, and run using below commandPython request.pyStatus code 200 indicates that request was made successfully.Response MethodsMethodDescriptionresponse.headersresponse.headers returns a dictionary of response headers.response.encodingresponse.encoding returns the encoding used to decode response.content.response.elapsedresponse.elapsed returns a timedelta object with the time elapsed from sending the request to the arrival of the response.response.close()response.close() closes the connection to the server.response.contentresponse.content returns the content of the response, in bytes.response.cookiesresponse.cookies returns a CookieJar object with the cookies sent back from the server.response.historyresponse.history returns a list of response objects holding the history of request (url).response.is_permanent_redirectresponse.is_permanent_redirect returns True if the response is the permanent redirected url, otherwise False.response.is_redirectresponse.is_redirect returns True if the response was redirected, otherwise False.response.iter_content()response.iter_content() iterates over the response.content.response.json()response.json() returns a JSON object of the result (if the result was written in JSON format, if not it raises an error).response.urlresponse.url returns the URL of the response.response.textresponse.text returns the content of the response, in unicode.response.status_coderesponse.status_code returns a number that indicates the status (200 is OK, 404 is Not Found).response.requestresponse.request returns the request object that requested this response.response.reasonresponse.reason returns a text corresponding to the status code.response.raise_for_status()response.raise_for_status() returns an HTTPError object if an error has occurred during the process.response.okresponse.ok returns True if status_code is less than 200, otherwise False.response.linksresponse.links returns the header links.Authentication using Python RequestsAuthentication refers to giving a user permissions to access a particular resource. Since, everyone can’t be allowed to access data from every URL, one would require authentication primarily. To achieve this authentication, typically one provides authentication data through Authorization header or a custom header defined by server.Example –# import requests moduleimport requestsfrom requests.auth import HTTPBasicAuth  # Making a get requestresponse = requests.get('https://api.github.com / user, ',            auth = HTTPBasicAuth('user', 'pass'))  # print request objectprint(response)Replace “user” and “pass” with your username and password. It will authenticate the request and return a response 200 or else it will return error 403.For more visit – Authentication using Python requestsSSL Certificate VerificationRequests verifies SSL certificates for HTTPS requests, just like a web browser. SSL Certificates are small data files that digitally bind a cryptographic key to an organization’s details. Often, an website with a SSL certificate is termed as secure website. By default, SSL verification is enabled, and Requests will throw a SSLError if it’s unable to verify the certificate.Disable SSL certificate verificationLet us try to access a website with an invalid SSL certificate, using Python requests# import requests moduleimport requests  # Making a get requestresponse = requests.get('https://expired.badssl.com/')  # print request objectprint(response)Output :-This website doesn’t have SSL setup so it raises this error.one can also pass the link to the certificate for validation via python requests only.# import requests moduleimport requests  # Making a get requestresponse = requests.get('https://github.com', verify ='/path/to/certfile')  # print request objectprint(response)This would work in case the path provided is correct for SSL certificate for github.com.For more visit- SSL Certificate Verification – Python requestsSession ObjectsSession object allows one to persist certain parameters across requests. It also persists cookies across all requests made from the Session instance and will use urllib3’s connection pooling. So if several requests are being made to the same host, the underlying TCP connection will be reused, which can result in a significant performance increase. A session object all the methods as of requests.Using Session ObjectsLet us illustrate use of session objects by setting a cookie to a url and then making a request again to check if cookie is set.# import requests moduleimport requests  # create a session objects = requests.Session()  # make a get requests.get('https://httpbin.org/cookies/set/sessioncookie/123456789')  # again make a get requestr = s.get('https://httpbin.org/cookies')  # check if cookie is still setprint(r.text)Output:For more, visit – Session Objects – Python requestsMy Personal Notes
arrow_drop_upSave
Requests installation depends on type of operating system on eis using, the basic command anywhere would be to open a command terminal and run,
pip install requests
The basic method for installation of requests on any operating system is to grab the base files and install requests manually and Requests is actively developed on GitHub, where the code is always available. For code – visit here.You can either clone the public repository:
git clone git://github.com/psf/requests.git
Once you have a copy of the source, you can embed it in your own Python package, or install it into your site-packages easily:
cd requests
pip install .
For more checkout – How to install requests in Python – For windows, linux, mac
Python requests module has several built-in methods to make Http requests to specified URI using GET, POST, PUT, PATCH or HEAD requests. A Http request is meant to either retrieve data from a specified URI or to push data to a server. It works as a request-response protocol between a client and a server. Let’s demonstrate how to make a GET request to an endpoint.GET method is used to retrieve information from the given server using a given URI. The GET method sends the encoded user information appended to the page request. The page and the encoded information are separated by the ‘?’ character.For example:
https://www.google.com/search?q=hello
Python’s requests module provides in-built method called get() for making a GET request to a specified URI.
Syntax –
requests.get(url, params={key: value}, args)

Example –
Let’s try making a request to github’s APIs for example purposes.
import requests   # Making a GET requestr = requests.get('https://api.github.com/users/naveenkrnl')  # check status code for response received# success code - 200print(r)  # print content of requestprint(r.content)
save this file as request.py and through terminal run,
python request.py
Output –
For more, visit GET method – Python requests
When one makes a request to a URI, it returns a response. This Response object in terms of python is returned by requests.method(), method being – get, post, put, etc. Response is a powerful object with lots of functions and attributes that assist in normalizing data or creating ideal portions of code. For example, response.status_code returns the status code from the headers itself, and one can check if the request was processed successfully or not.Response object can be used to imply lots of features, methods, and functionalities.
# import requests moduleimport requests  # Making a get requestresponse = requests.get('https://api.github.com/')  # print request objectprint(response.url)  # print status codeprint(response.status_code)
Save this file as request.py, and run using below command
Python request.py
Status code 200 indicates that request was made successfully.
Authentication refers to giving a user permissions to access a particular resource. Since, everyone can’t be allowed to access data from every URL, one would require authentication primarily. To achieve this authentication, typically one provides authentication data through Authorization header or a custom header defined by server.
Example –
# import requests moduleimport requestsfrom requests.auth import HTTPBasicAuth  # Making a get requestresponse = requests.get('https://api.github.com / user, ',            auth = HTTPBasicAuth('user', 'pass'))  # print request objectprint(response)
Replace “user” and “pass” with your username and password. It will authenticate the request and return a response 200 or else it will return error 403.For more visit – Authentication using Python requests
Requests verifies SSL certificates for HTTPS requests, just like a web browser. SSL Certificates are small data files that digitally bind a cryptographic key to an organization’s details. Often, an website with a SSL certificate is termed as secure website. By default, SSL verification is enabled, and Requests will throw a SSLError if it’s unable to verify the certificate.
Let us try to access a website with an invalid SSL certificate, using Python requests
# import requests moduleimport requests  # Making a get requestresponse = requests.get('https://expired.badssl.com/')  # print request objectprint(response)
Output :-This website doesn’t have SSL setup so it raises this error.one can also pass the link to the certificate for validation via python requests only.
# import requests moduleimport requests  # Making a get requestresponse = requests.get('https://github.com', verify ='/path/to/certfile')  # print request objectprint(response)
This would work in case the path provided is correct for SSL certificate for github.com.For more visit- SSL Certificate Verification – Python requests
Session object allows one to persist certain parameters across requests. It also persists cookies across all requests made from the Session instance and will use urllib3’s connection pooling. So if several requests are being made to the same host, the underlying TCP connection will be reused, which can result in a significant performance increase. A session object all the methods as of requests.
Let us illustrate use of session objects by setting a cookie to a url and then making a request again to check if cookie is set.
# import requests moduleimport requests  # create a session objects = requests.Session()  # make a get requests.get('https://httpbin.org/cookies/set/sessioncookie/123456789')  # again make a get requestr = s.get('https://httpbin.org/cookies')  # check if cookie is still setprint(r.text)
Output:For more, visit – Session Objects – Python requests
Python-requests
Python
Writing code in comment?
Please use ide.geeksforgeeks.org,
generate link and share the link here.
Python Dictionary
How to Install PIP on Windows ?
Enumerate() in Python
Different ways to create Pandas Dataframe
Python String | replace()
*args and **kwargs in Python
Create a Pandas DataFrame from Lists
Check if element exists in list in Python
How To Convert Python Dictionary To JSON?
Convert integer to string in Python