Create Your Own Web Scraper in C# in Just a Few Minutes!
By: Dan Suciu
Tags: C Sharp Programming, C Sharp Web Scraper, Web Scrapers, C Programming Language, Web Scraping

Creating your own web scraper in C#

For this tutorial, I will show you how a web scraper can be written in C#. I know that using a different programming language such as Python can be more advantageous for this task, but that doesn’t mean it is impossible to do it in C#.

Coding in C# has its advantages, such as:

It is object-oriented;

Has better integrity and interoperability;

It is a cross-platform;

1. Choose the page you want to scrape

First things first, you need to decide what web page to scrape. In this example, I will be scraping Greece on Wikipedia and see what subjects are presented in its Table of Contents. This is an easy example, but you can scale it to other web pages as well.

2. Inspect the code of the website

Using the developer tools, you can inspect each element to check and see under which tag the information you need is. Simply right-click on the web page and select “inspect”, and a “Browser Inspector Box” will pop up.

You can search for the class directly in the elements section or using the inspect tool on the web page as shown below.

Thus, you found out that the data you need is located within the span tag which has the class toctext . What you’ll do next is extract the whole HTML of the page, parse it, and select only the data within that specific class. Let’s make some quick preparations first!

3. Prepare the workspace

You can use whatever IDE is comfortable for you. In this example, I will use Visual Studio Code. You will also need to install .NET SDK.

Now you need to create your project. To do so, you obviously have to open up Visual Studio Code. Then, you will go to the extensions menu and install C# for Visual Studio Code.

You need a place to write and run our code. In the menu bar, you will select File > Open File (File > Open… on macOS) and in the dialog, you will create a folder that will serve as our workspace.

After you created the workplace, you can create a simple “Hello World” application template by entering the following command in our projects’ terminal:

dotnet new console

Your new project should look like this:

Next, you need to install these two packages:

HtmlAgilityPack is an HTML parser written in C# to read/write DOM.

CsvHelper is a package that’s used to read and write CSV files.

You can install them using these command lines inside your projects’ terminal:

dotnet add package csvhelper

dotnet add package htmlagilitypack

4. Write the code

Let’s import the packages we installed a few minutes ago, and some other helpful packages for later use:

using CsvHelper; using HtmlAgilityPack; using System.IO; using System.Collections.Generic; using System.Globalization;

Outside our Main function , you will create a public class for your table of contents titles.

public class Row { public string Title {get; set;} }

Now, coming back to the Main function , you need to load the page you wish to scrape. As I mentioned before, we will look at what Wikipedia is writing about Greece!

HtmlWeb web = new HtmlWeb(); HtmlDocument doc = web.Load("https://en.wikipedia.org/wiki/Greece");

Our next step is to parse and select the nodes containing the information you are looking for, which is located in the span tags with the class toctext .

var HeaderNames = doc.DocumentNode.SelectNodes("//span[@class='toctext']");

What should you do with this information now? Let’s store it in a .csv file for later use. To do that, you first need to iterate over each node we extracted earlier and store its text into a list.

CsvHelper will do the rest of the job, creating and writing the extracted information into a file.

var titles = new List<Row>(); foreach (var item in HeaderNames) { titles.Add(new Row { Title = item.InnerText}); } using (var writer = new StreamWriter("your_path_here/example.csv")) using (var csv = new CsvWriter(writer, CultureInfo.InvariantCulture)) { csv.WriteRecords(titles); }

5. Run the code

The code is done and now we just have to run it! Use this command line in the terminal. Be sure you saved your file first!

dotnet run

You’re done!