How to get the next page on BeautifulSoup?

16 May, 2021
In this article, we are going to see how to Get the next page on beautifulsoup.
BeautifulSoup: Beautiful Soup(bs4) is a Python library for pulling data out of HTML and XML files. To install this module type the below command in the terminal.
pip install bs4
requests: This library allows you to send HTTP/1.1 requests extremely easily. To install this module type the below command in the terminal.
pip install requests
Approach:
Get the next page on beautifulsoup means first we will scrap one-page content and if many links are given on the page, and we want to scrap them also. We can get the next page first we will scrap the sample website after that any other links find, and we will call again requests. Get method for that page and will create a soup of that also. So this way we can get to the next page on beautifulsoup.
Let’s execute the script step-by-step :
Step 1: Import all dependence
from bs4 import BeautifulSoup
import requests
Step 2: We need to request the page URL with requests.
page=requests.get(sample_website)
Step 3: With the help of beautifulsoup method and HTML parser we will create a soup of the page.
soup = BeautifulSoup(page, 'html.parser')
Step 4:
We will search in the parse tree and find the link. If we want that URL, then with the help of the requests module and beautiful module we will again create the soup of the next page hence we can get the next page on beautifulsoup.
Python3
for i in soup.find_all('a', href = True):      # check all link which is contain  # "www.geeksforgeeks.org" string   if("www.geeksforgeeks.org" in i['href']):          # call get method to request next url    nextpage = requests.get(i['href'])          # create soup for next url    nextsoup = BeautifulSoup(nextpage.content, 'html.parser')          # we can scrap any thing of the    # next page here we are scraping title of     # nexturl page string    print("next url title : ",nextsoup.find('title').string)
Below is the full Implementation:
Python3
from bs4 import BeautifulSoupimport requests  # sample websitesample_website='https://www.geeksforgeeks.org/different-ways-to-remove-all-the-digits-from-string-in-java/'  # call get method to request the pagepage=requests.get(sample_website)  # with the help of BeautifulSoup# method and html parser created soupsoup = BeautifulSoup(page.content, 'html.parser')  # With the help of find_all# method perform searching in parser treefor i in soup.find_all('a', href = True):      # check all link which is contain  # "www.geeksforgeeks.org" string   if("www.geeksforgeeks.org" in i['href']):          # call get method to request next url    nextpage = requests.get(i['href'])          # create soup for next url    nextsoup = BeautifulSoup(nextpage.content, 'html.parser')          # we can scrap any thing of the    # next page here we are scraping title of     # nexturl page string    print("next url title : ",nextsoup.find('title').string)
Output:
next url title :  GeeksforGeeks | A computer science portal for geeks
next url title :  Analysis of Algorithms | Set 1 (Asymptotic Analysis) - GeeksforGeeks
next url title :  Analysis of Algorithms | Set 2 (Worst, Average and Best Cases) - GeeksforGeeks
next url title :  Analysis of Algorithms | Set 3 (Asymptotic Notations) - GeeksforGeeks
next url title :  Analysis of algorithms | little o and little omega notations - GeeksforGeeks
next url title :  Lower and Upper Bound Theory - GeeksforGeeks
next url title :  Analysis of Algorithms | Set 4 (Analysis of Loops) - GeeksforGeeks
next url title :  Analysis of Algorithm | Set 4 (Solving Recurrences) - GeeksforGeeks
next url title :  Analysis of Algorithm | Set 5 (Amortized Analysis Introduction) - GeeksforGeeks
next url title :  What does 'Space Complexity' mean? - GeeksforGeeks
next url title :  Pseudo-polynomial Algorithms - GeeksforGeeks
next url title :  Polynomial Time Approximation Scheme - GeeksforGeeks
next url title :  A Time Complexity Question - GeeksforGeeks
................................................................. 
Picked
Python BeautifulSoup
Python
Writing code in comment?
Please use ide.geeksforgeeks.org,
generate link and share the link here.
How to Install PIP on Windows ?
Python Classes and Objects
Python OOPs Concepts
Introduction To PYTHON
How to drop one or multiple columns in Pandas Dataframe
Python | os.path.join() method
Check if element exists in list in Python
How To Convert Python Dictionary To JSON?
Python | Get unique values from a list
Python | datetime.timedelta() function