High-performance CSV parser with System.IO.Pipelines
By: Matias Paulo
Tags: Csharp, Csv File, Net Core, Performance, Pipelines

Hello! Today, I want to share with you, my experience creating a high performance CSV file parser.

Let me explain you the scenario, I downloaded the a csv file from http://eforexcel.com/wp/downloads-18-sample-csv-files-data-sets-for-testing-sales/, for this particullary scenario, I downloaded a sales file with 100000 records.

High-performance CSV parser with System.IO.PipelinesThe CSV has 14 fields of different types, and the idea is to read the file, parse the records and return a collection with all the records. We are going to achieve that, following 3 different approaches:

CsvHelper : one of the most popular libraries for write/read CSV files.

: one of the most popular libraries for write/read CSV files. StreamReader : we are going to make use of the IAsyncEnumerable interface, which was introduced with C# 8.0, which will allow us to work with a single data stream at the time instead of whole file.

: we are going to make use of the IAsyncEnumerable interface, which was introduced with C# 8.0, which will allow us to work with a single data stream at the time instead of whole file. Pipelines: introduced in Net Core 2.1, is a library designed for a high IO performance.

Today, I’m going to start with the benchmarks results and then, we can explore how I implemented the different approaches.

Results

As we can notice the performance between using CsvHelper library and IAsyncEnumerable is pretty much the same, the difference between both is with the memory allocation. The real winner here is System.IO.Pipelines, which takes half of the time to process the file and the memory allocation is insignificant against the other approaches.

Let’s start coding!

First to all we need to define the data model.

Sale model

CsvHelper

To make use of this library we need to include the following package: <PackageReference Include=”CsvHelper” Version=”27.1.1" />.

Once we added the package, the library is pretty straightforward to use.

We just open the file, create a new instance of the reader and call the “GetRecords” method.

IAsyncEnumerable

In this case, we are going to open the file and then process a line at the time.

As you can see, we are returning one line at a time and making the transformation with Linq. We are skipping the first element because is the file header.

System.IO.Pipelines

In order to make use of the library, we need to add the following line to the “csproj” file:

<PackageReference Include=”System.IO.Pipelines” Version=”5.0.1" />

I’ll try to explain to you the different sections of the code, if you want to know better how pipelines work, I strongly suggest you check this blog: https://devblogs.microsoft.com/dotnet/system-io-pipelines-high-performance-io-in-net/

The main entry point is the “Parse” method. The first thing that we are doing is creating an Array pool, which is designed for high-performance array management. Once we finish using the array, we can return it to the pool.

Inmediatly, we are opening the file and creating a new “PipeReader”, which is making use of the “FileStream”.

Something that could be weird, is the “while” loop, in this case, what we are doing is getting the results from the “PipeReader”, parsing the data with the “ParseLine” method and moving the reader’s cursor once we consumed the data. We call the “CompleteAsync” method when no more data will be read from it.

“ParseLine” just read a line and call the “GetSale” method, which transforms the byte sequence to a “Sale” instance, if is not null, we are adding it to the sales collection.

And finally, the last piece here is the parsing of the byte sequence into a “Sale” model.

We start checking if the sequence is the file header, if it is, we are returning null otherwise creating a new “Sale” instance.

Then, we are iterating until 14 (the number of fields per record) and per each iteration, we are parsing the field for that position and assigning it.

In this case, we are not handling null values or possible failures, this is just for demo purposes.