Package: andrews-xsv
xsv is a command line program for indexing, slicing, analyzing, splitting
and joining CSV files. Commands should be simple, fast and composable:Simple tasks should be easy.Performance trade offs should be exposed in the CLI interface.Composition should not come at the expense of performance.This README contains information on how toinstallxsv, in addition to
a quick tour of several commands.Dual-licensed under MIT or theUNLICENSE.Available commandscat- Concatenate CSV files by row or by column.count- Count the rows in a CSV file. (Instantaneous with an index.)fixlengths- Force a CSV file to have same-length records by either
padding or truncating them.flatten- A flattened view of CSV records. Useful for viewing one record
at a time. e.g.,xsv slice -i 5 data.csv | xsv flatten.fmt- Reformat CSV data with different delimiters, record terminators
or quoting rules. (Supports ASCII delimited data.)frequency- Build frequency tables of each column in CSV data. (Uses
parallelism to go faster if an index is present.)headers- Show the headers of CSV data. Or show the intersection of all
headers between many CSV files.index- Create an index for a CSV file. This is very quick and provides
constant time indexing into the CSV file.input- Read CSV data with exotic quoting/escaping rules.join- Inner, outer and cross joins. Uses a simple hash index to make it
fast.partition- Partition CSV data based on a column value.sample- Randomly draw rows from CSV data using reservoir sampling (i.e.,
use memory proportional to the size of the sample).reverse- Reverse order of rows in CSV data.search- Run a regex over CSV data. Applies the regex to each field
individually and shows only matching rows.select- Select or re-order columns from CSV data.slice- Slice rows from any part of a CSV file. When an index is present,
this only has to parse the rows in the slice (instead of all rows leading up
to the start of the slice).sort- Sort CSV data.split- Split one CSV file into many CSV files of N chunks.stats- Show basic types and statistics of each column in the CSV file.
(i.e., mean, standard deviation, median, range, etc.)table- Show aligned output of any CSV data usingelastic tabstops.A whirlwind tourLet's say you're playing with some of the data from theData Science Toolkit, which contains
several CSV files. Maybe you're interested in the population counts of each
city in the world. So grab the data and start examining it:$curl-LOhttps://burntsushi.net/stuff/worldcitiespop.csv
$xsvheadersworldcitiespop.csv1Country2City3AccentCity4Region5Population6Latitude7LongitudeThe next thing you might want to do is get an overview of the kind of data that
appears in each column. Thestatscommand will do this for you:$xsvstatsworldcitiespop.csv--everything|xsvtable
fieldtypeminmaxmin_lengthmax_lengthmeanstddevmedianmodecardinality
CountryUnicodeadzw22cn234CityUnicodebabelahmarÞykkvibaer191sanjose2351892AccentCityUnicodeBâbelAhmarïnBouChella191SanAntonio2375760RegionUnicode00Z9021304397PopulationInteger7314804980847719.570634302885.5592041077928754LatitudeFloat-54.93333382.48333311227.18816621.95261432.49722251.151038349LongitudeFloat-179.98333318011437.0888663.2230135.2823.81167162Thexsv tablecommand takes any CSV data and formats it into aligned columns
usingelastic tabstops. You'll
notice that it even gets alignment right with respect to Unicode characters.So, this command takes about 12 seconds to run on my machine, but we can speed
it up by creating an index and re-running the command:$xsvindexworldcitiespop.csv
$xsvstatsworldcitiespop.csv--everything|xsvtable
...Which cuts it down to about 8 seconds on my machine. (And creating the index
takes less than 2 seconds.)Notably, the same type of "statistics" command in anotherCSV command line toolkittakes about 2 minutes to produce similar statistics on the same data set.Creating an index gives us more than just faster statistics gathering. It also
makes slice operations extremely fast becauseonly the sliced portionhas to
be parsed. For example, let's say you wanted to grab the last 10 records:$xsvcountworldcitiespop.csv3173958$xsvsliceworldcitiespop.csv-s3173948|xsvtable
CountryCityAccentCityRegionPopulationLatitudeLongitude
zwzibalonkweZibalonkwe06-19.833333327.4666667
zwzibunkululuZibunkululu06-19.666666727.6166667
zwzigaZiga06-19.216666727.4833333
zwzikamanasvillageZikamanasVillage00-18.216666727.95
zwzimbabweZimbabwe07-20.266666730.9166667
zwzimreparkZimrePark04-17.866111131.2136111
zwziyakamanasZiyakamanas00-18.216666727.95
zwzizalisariZizalisari04-17.758888931.0105556
zwzuzumbaZuzumba06-20.033333327.9333333
zwzvishavaneZvishavane0779876-20.333333330.0333333These commands areinstantaneousbecause they run in time and memory
proportional to the size of the slice (which means they will scale to
arbitrarily large CSV data).Switching gears a little bit, you might not always want to see every column in
the CSV data. In this case, maybe we only care about the country, city and
population. So let's take a look at 10 random rows:$xsvselectCountry,AccentCity,Populationworldcitiespop.csv\|xsvsample10\|xsvtable
CountryAccentCityPopulation
cnGuankoushang
zaKlipdrift
maOuledHammou
frLesGravues
laBanPhadèng
deLüdenscheid80045qaUmmashShubrum
bdPanditgoan
usAppleton
uaLukashenkivskeWhoops! It seems some cities don't have population counts. How pervasive is
that?$xsvfrequencyworldcitiespop.csv--limit5field,value,count
Country,cn,238985
Country,ru,215938
Country,id,176546
Country,us,141989
Country,ir,123872
City,sanjose,328
City,sanantonio,320
City,santarosa,296
City,santacruz,282
City,sanjuan,255
AccentCity,SanAntonio,317
AccentCity,SantaRosa,296
AccentCity,SantaCruz,281
AccentCity,SanJuan,254
AccentCity,SanMiguel,254
Region,04,159916
Region,02,142158
Region,07,126867
Region,03,122161
Region,05,118441
Population,(NULL),3125978
Population,2310,12
Population,3097,11
Population,983,11
Population,2684,11
Latitude,51.15,777
Latitude,51.083333,772
Latitude,50.933333,769
Latitude,51.116667,769
Latitude,51.133333,767
Longitude,23.8,484
Longitude,23.2,477
Longitude,23.05,476
Longitude,25.3,474
Longitude,23.1,459(Thexsv frequencycommand builds a frequency table for each column in the
CSV data. This one only took 5 seconds.)So it seems that most cities do not have a population count associated with
them at all. No matter—we can adjust our previous command so that it only
shows rows with a population count:$xsvsearch-sPopulation'[0-9]'worldcitiespop.csv\|xsvselectCountry,AccentCity,Population\|xsvsample10\|xsvtable
CountryAccentCityPopulation
esBarañáin22264esPuertoReal36946atMoosburg4602huHejobaba1949ruPolyarnyyeZori15092grKandíla1245isÓlafsvík992huDecs4210bgSliven94252gbLeatherhead43544Erk. Which country isat? No clue, but the Data Science Toolkit has a CSV
file calledcountrynames.csv. Let's grab it and do a join so we can see which
countries these are:curl-LOhttps://gist.githubusercontent.com/anonymous/063cb470e56e64e98cf1/raw/98e2589b801f6ca3ff900b01a87fbb7452eb35c7/countrynames.csv
$xsvheaderscountrynames.csv1Abbrev2Country
$xsvjoin--no-caseCountrysample.csvAbbrevcountrynames.csv|xsvtable
CountryAccentCityPopulationAbbrevCountry
esBarañáin22264ESSpain
esPuertoReal36946ESSpain
atMoosburg4602ATAustria
huHejobaba1949HUHungary
ruPolyarnyyeZori15092RURussianFederation|Russia
grKandíla1245GRGreece
isÓlafsvík992ISIceland
huDecs4210HUHungarybgSliven94252BGBulgaria
gbLeatherhead43544GBGreatBritain|UK|England|Scotland|Wales|NorthernIreland|UnitedKingdomWhoops, now we have two columns calledCountryand anAbbrevcolumn that we
no longer need. This is easy to fix by re-ordering columns with thexsv selectcommand:$xsvjoin--no-caseCountrysample.csvAbbrevcountrynames.csv\|xsvselect'Country[1],AccentCity,Population'\|xsvtable
CountryAccentCityPopulation
SpainBarañáin22264SpainPuertoReal36946AustriaMoosburg4602HungaryHejobaba1949RussianFederation|RussiaPolyarnyyeZori15092GreeceKandíla1245IcelandÓlafsvík992HungaryDecs4210BulgariaSliven94252GreatBritain|UK|England|Scotland|Wales|NorthernIreland|UnitedKingdomLeatherhead43544Perhaps we can do this with the original CSV data? Indeed we can—because
joins inxsvare fast.$xsvjoin--no-caseAbbrevcountrynames.csvCountryworldcitiespop.csv\|xsvselect'!Abbrev,Country[1]'\>worldcitiespop_countrynames.csv
$xsvsample10worldcitiespop_countrynames.csv|xsvtable
CountryCityAccentCityRegionPopulationLatitudeLongitude
SriLankamiriswatteMiriswatte367.233333379.9
RomanialivezileLivezile26198544.51222222.863333
IndonesiatawainaluTawainalu22-4.0225121.9273
RussianFederation|RussiaotarOtar4556.97527848.305278
Francelebreuil-boisrobertleBreuil-BoisRobertA848.9455671.717026
FrancelissacLissacB145.1030941.464927
AlbanialumalasiLumalasi4640.658611120.7363889
ChinamotzushihMotzushih1127.65111.966667
RussianFederation|RussiasvakinoSvakino6955.6021134.559785
RomaniatirgupancestiTirguPancesti3846.21666727.1The!Abbrev,Country[1]syntax means, "remove theAbbrevcolumn and remove
the second occurrence of theCountrycolumn." Since we joined withcountrynames.csvfirst, the firstCountryname (fully expanded) is now
included in the CSV data.Thisxsv joincommand takes about 7 seconds on my machine. The performance
comes from constructing a very simple hash index of one of the CSV data files
given. Thejoincommand does an inner join by default, but it also has left,
right and full outer join support too.InstallationBinaries for Windows, Linux and macOS are availablefrom Github.If you're amacOS Homebrewuser, then you can install xsv
from homebrew-core:$ brew install xsvIf you're amacOS MacPortsuser, then you can install xsv
from theofficial ports:$ sudo port install xsvIf you're aNix/NixOSuser, you can install xsv from nixpkgs:$ nix-env -i xsvAlternatively, you can compile from source byinstalling Cargo(Rust'spackage manager)
and installingxsvusing Cargo:cargoinstallxsvCompiling from this repository also works similarly:gitclonegit://github.com/BurntSushi/xsvcdxsv
cargobuild--releaseCompilation will probably take a few minutes depending on your machine. The
binary will end up in./target/release/xsv.BenchmarksI've compiled somevery rough
benchmarksof
variousxsvcommands.MotivationHere are several valid criticisms of this project:You shouldn't be working with CSV data because CSV is a terrible format.If your data is gigabytes in size, then CSV is the wrong storage type.Various SQL databases provide all of the operations available inxsvwith
more sophisticated indexing support. And the performance is a zillion times
better.I'm sure there are more criticisms, but the impetus for this project was a 40GB
CSV file that was handed to me. I was tasked with figuring out the shape of the
data inside of it and coming up with a way to integrate it into our existing
system. It was then that I realized that every single CSV tool I knew about was
woefully inadequate. They were just too slow or didn't provide enough
flexibility. (Another project I had comprised of a few dozen CSV files. They
were smaller than 40GB, but they were each supposed to represent the same kind
of data. But they all had different column and unintuitive column names. Useful
CSV inspection tools were critical here—and they had to be reasonably fast.)The key ingredients for helping me with my task were indexing, random sampling,
searching, slicing and selecting columns. All of these things made dealing with
40GB of CSV data a bit more manageable (or dozens of CSV files).Getting handed a large CSV fileoncewas enough to launch me on this quest.
From conversations I've had with others, CSV data files this large don't seem
to be a rare event. Therefore, I believe there is room for a tool that has a
hope of dealing with data that large.Naming collisionThis project is unrelated to another similar project with the same name:https://mj.ucw.cz/sw/xsv/