Web scraping 101. How to make a simple bot for winning in... | by German Sharabok | Towards Data Science

Web scraping is a process of retrieving data and conducting certain actions online. It is usually done by a bot in order to automate certain repetitive tasks. This is a rather large field of study and there are a lot of fun things you can do with it. Getting started does not have to be slow and difficult. In this article, I will bring you through a simple process of creating a bot and how you can expand on it in the future. No prior experience is required, just Python and Internet connection.
This tutorial is intended for beginners with any programming level and everything will be explained as clear and straightforward as possible. I will use a random folklore competition to demonstrate the voting. You can use any website you like and adapt based on that.
Before diving straight into coding there are two things we have to have set up.
First, we need a place where the code will be written if you do not have one already. I prefer using Visual Studio Code as it is simple and provides anything you can need. Simply search “Visual Studio Code” and download the latest version.
Once it is installed click the Extension Button and install Python.
We need a module to be able to work with Google Chrome. This step is rather short. Search for “ChromeDriver Chromium” and download the latest version. Once it is downloaded, unzip the file.
For Windows, copy the file, create a folder such as “C:\webdrivers” and place it there. Now, go to the PATH settings and add the folder path there(if you do not know where this is, there are plenty of explanations in Google).
For Mac and Linux, open terminal and type
mv ~/Downloads/chromedriver /usr/local/bin
Great, now we are set up and ready to start! Let’s create a new folder and call it “web-scraping”. Now, open Visual Studio Code and create a new file called “vote_bot.py”.
In the terminal window type:
python -m venv venv
This will create a virtual environment, which basically allows us to install packages without modifying anything in your system packages. To activate it:
source venv/Scripts/activate
Next, we need one package, that will help us in working with a web browser. Type:
pip install selenium
Open our vote_bot file in VS Code. The first line we are going to have is:
from selenium import webdriverimport time
This will add the web driver support to our project and help us in later steps. Additionally, we are importing time just to be able to keep the page open for longer.
Next, we are going to define a bot. This bot will be doing the work for us. The class definition is basically describing an object we want to have. For example, a dog has certain characteristics, such as legs, height, speed, and age. Same with a bot, we need to input its characteristics. In this tutorial, I am trying to make it as simple as possible so the only definition we need is the driver we will use for working with a browser.
class VoteBot():  def __init__(self):      self.driver = webdriver.Chrome("C:\webdrivers\chromedriver.exe")
Now, when we create a bot the first thing that happens is the driver is defined. We can actually try it out and see how it works. By typing python vote_bot.py we can run the program. If we add “-i”, we can directly interact with the code.
After repeating the steps above a new browser window should open. If you do not see a new Chrome window make sure you have properly installed ChromeDriver and selenium, which is explained in the previous steps.
We need a function that will do the voting. Here the voting process is simply opening a link, so say we are picking the second contestant and we will just copy the link for voting. The following code will open the page for voting, keep it open for 1 second, and then close it.
class VoteBot():  def __init__(self):      self.driver = webdriver.Chrome("C:\webdrivers\chromedriver.exe")def vote(self):    self.driver.get('https://eaff.eu/bg/online-festival-vote-choice/30')    time.sleep(1)    self.driver.close()
Now, we have finished making our bot. All we need is to add a loop so we can generate an infinite amount of votes and test it. The full code is attached below.
class VoteBot():  def __init__(self):      self.driver = webdriver.Chrome("C:\webdrivers\chromedriver.exe")def vote(self):    self.driver.get('https://eaff.eu/bg/online-festival-vote-choice/30')    time.sleep(1)    self.driver.close()while True:  bot = VoteBot()  bot.vote()
Simply type “python vote_bot.py” in your terminal and watch the bot repeat the task many many times. As you can see the bot will vote, close the windows, and vote again.
This tutorial is made for people who are just getting started with programming and web scraping. For any questions please feel free to leave your concerns in this post and I will help to my best knowledge. To further experiment, try using different types of buttons and login sequences, more on that I will be posing soon. Keep Calm and Keep Coding!