Setting up a scraping server on Digital Ocean — Part 2
By: Abhishek Chaudhary
Tags: Selenium, Development, Digitalocean, Scraping, Python

Once you have created the droplet successfully all that remains is interacting with it and configuring it for our purpose.

Step One: Logging in

Take note of the IP of the droplet, you’ll be needing that for login purposes.

Another thing which you’ll need are the keys you generated earlier.

Windows users can make use of putty which we installed in the previous part of the series and rest should be self-explanatory or you can follow this guide On OS X, you can use the following guide to connect to the server. It uses your key to connect to the server’s IP address using the username root . Make sure the path to your key is correct, and make sure you put in your IP address.

Step Two: Welcome to your server!

Let’s begin by updating the software already present on the server.

apt update

apt -y upgrade

Step Three: Install Python, update pip and install some packages

We install Python 3 using apt

apt install -y python3

2. Now we want to get PIP and the packages needed with the following command

sudo apt-get -y install python3-pip python3-dev build-essential libssl-dev libffi-dev xvfb

3. Upgrade PIP with the command:

pip3 install --upgrade pip

4. Installing virtualenv will allow us to create a virtual environment and install any Python packages in it without affecting our system’s Python.

sudo pip3 install virtualenv

virtualenv /var/venv

source /var/venv/bin/activate

Step Four: Installing Chrome

First, we’ll install some packages Chrome depends on.

apt install -y libxss1 libappindicator1 libindicator7

2. Then we’ll download the Chrome package.

3. We’ll use dpkg to try to install it from the package file.

dpkg -i google-chrome*.deb

4. For some reasons it fails and then we’ll need to run the following command which fixes everything.

apt install -y -f

Step Five: Installing Chrome Driver

First, we download the chrome driver and unzip to open it.

2. Now we can use the unzip command to… unzip it.

unzip chromedriver_linux64.zip

3. Change it to executable. Then we move it into somewhere findable, in our PATH.

chmod +x chromedriver

mv -f chromedriver /usr/local/bin/chromedriver

Step Six: Installing a fake display

As we know selenium really really want a display in order to scrape the information off of the websites, so we’ll need to provide it with one here.

First, let’s install the supporting packages.

apt install -y xvfb

Then we install the main thing.

pip3 install pyvirtualdisplay

Step Seven: Test it out

Open an ipython screen by typing ipython and execute the following code

from pyvirtualdisplay import Display

from selenium import webdriver display = Display(visible=0, size=(800, 600))

display.start() options = webdriver.ChromeOptions()

options.add_argument('--no-sandbox')

options = webdriver.ChromeOptions()

options.add_argument('--disable-extensions')

options.add_argument('--headless')

options.add_argument('--disable-gpu')

options.add_argument('--no-sandbox')

options.add_argument('--disable-dev-shm-usage') driver = webdriver.Chrome(chrome_options=options)

driver.get('http://google.com')

print(driver.title)

Step Eight: Let the gremlins in your head take over.