Understand CSV Files from your Terminal with XSV | by Khuyen Tran | Towards Data Science

Have you ever tried to have a general understanding of your CSV file by staring at it but ended up not understanding your file? You can open a Jupyter Notebook to analyze your CSV file, but it is time-consuming to open a notebook just to understand a CSV file, especially when you are working primarily with Python scripts and terminal.
Is there a way that you can quickly analyze your CSV files from your terminal in 1 line of code such as this?
$ xsv stats bestsellers.csv | xsv table
This is when xsv comes in handy.
xsv is a command-line program for indexing, slicing, analyzing, splitting, and joining CSV files. I like xsv because makes it extremely quick and easy to work with CSV files.
Find the instruction to install xsv here. In this tutorial, I will show you how to manipulate and analyze your data using Amazon Top 50 Bestselling Books 2009–2019 from Kaggle as an example.
Download and unzip the CSV file. Change the file’s name to bestsellers.csv so it is easier for us to work with.
Now, our task is to try to understand our CSV file
Type xsv -h to get the commands. You should see something like this
Commands:    cat         Concatenate by row or column    count       Count records    fixlengths  Makes all records have same length    flatten     Show one field per line    fmt         Format CSV output (change field delimiter)    frequency   Show frequency tables    headers     Show header names    help        Show this usage message.    index       Create CSV index for faster access    input       Read CSV data with special quoting rules    join        Join CSV files    sample      Randomly sample CSV data    search      Search CSV data with regexes    select      Select columns from CSV    slice       Slice records from CSV    sort        Sort CSV data    split       Split CSV data into many files    stats       Compute basic statistics    table       Align CSV data into columns
To get a random sample of 10 records of our data, use
$ xsv sample 10 bestsellers.csv
Output:
The output is difficult to read. Let’s prettify the output by adding xsv table
$ xsv sample 10 bestsellers.csv | xsv table
The output looks much easier to read now!
We might just want to look at the name of the book, author, user rating, and price to decide which books we should buy. To select particular columns, use xsv select col1,col2
$ xsv sample 10 bestsellers.csv | xsv select Name,Author,'User Rating',Price | xsv table
If you prefer to view one field per line, use xsv flatten instead of xsv table
$ xsv sample 2 bestsellers.csv | xsv flatten
How many records do we have in total? Let’s figure it out using xsv count
$ xsv count bestsellers.csv550
We have a total of 550 rows of data. We also want to get the stats of the data (.i.e, what is the average user rating). How can we find that?
We can use xsv stats to view the stats such as sum, min, max, mean, and standard deviation
$ xsv stats bestsellers.csv | xsv table
Cool! We created a summary table in 0.010 s! But what if we just care about the stats of User Rating? We can select the relevant rows by using xsv search
$ xsv stats bestsellers.csv | xsv search "User Rating" | xsv table
Then if we want to extract only the mean of user ratings, we can use xsv select
$ xsv stats bestsellers.csv | xsv search "User Rating" | xsv select mean | xsv table
Note that xsv search is used to select rows while xsv select is used to select columns.
Look like some books were mentioned more than one time. Let’s find out the frequency of each book. Use --limit number to specify the number of outputs.
$ xsv frequency bestsellers.csv --limit 20| xsv search Name | xsv table
There are 10 Publication Manual of the American Psychological Association, 6th Edition in our data! How come?
The beauty of xsv is that we can reuse the same command to perform different functions. We will use xsv search -s column string_to_search to search for rows within a column that match the search string ‘Publication Manual of the American Psychological Association, 6th Edition’.
$ xsv search -s Name "Publication Manual of the American Psychological Association, 6th Edition" bestsellers.csv | xsv table
Cool! It turns out that Publication Manual of the American Psychological Association, 6th Edition was published in 10 different years. The author and ratings are still the same.
You may not want to get a random sample of your data but to get the first 10 rows. To get the rows in a specific range, use xsv slice . Below is the usage
$ xsv slice --help
If the start of the range isn’t specified, then the slice starts from the firstrecord in the CSV data. To get the first 10 rows, use
$ xsv slice -e 10 bestsellers.csv | xsv table
To get the data starting from the 2nd row to the 10th row, use
$ xsv slice -s 2 -e 10 bestsellers.csv | xsv table
Note that -s stands for start while -e stands for end.
You can also sort data based on one or more columns. To sort data based on the columnUser Rating , in reversed order, use
$ xsv sort -s 'User Rating' -R bestsellers.csv  | xsv slice -e 10 | xsv table
Note that -s stands for select and -R stands for reverse .
As promised in the title, we get what we want in one line of code!
Congratulations! You have just learned how to manipulate and analyze your CSV files using xsv. In case you don’t remember the commands shown above, just use xsv --help or xsv <command> --help , such as xsv search --help .
Even though xsv cannot replace other sophisticated feature engineering methods, if you just want to quickly analyze your CSV file from your terminal, xsv is the way to go!
I like to write about basic data science concepts and play with different algorithms and data science tools. You could connect with me on LinkedIn and Twitter.
Star this repo if you want to check out the codes for all of the articles I have written. Follow me on Medium to stay informed with my latest data science articles like these: