Package: ph
ph (pronouncedφ) - the tabular data shell toolSpoiler: Working with tabular data (csv) in the command line is difficult.phmakes it easy:$pipinstallph
$catiris.csv|phcolumns1504setosa
versicolor
virginica
$catiris.csv|phcolumnssetosaversicolor|phhead15|phtail5|phshowsetosaversicolor
----------------------01.50.211.60.221.40.131.10.141.20.2$catiris.csv|phdescribe1504setosaversicolorvirginica
count150.000000150.000000150.000000150.000000150.000000
mean5.8433333.0573333.7580001.1993331.000000
std0.8280660.4358661.7652980.7622380.819232
min4.3000002.0000001.0000000.1000000.00000025%5.1000002.8000001.6000000.3000000.00000050%5.8000003.0000004.3500001.3000001.00000075%6.4000003.3000005.1000001.8000002.000000
max7.9000004.4000006.9000002.5000002.000000Occasionally you would like to plot a CSV file real quick, in which case you can
simply pipe it toph plot:Suppose you have a datasetcovid.csvSK,Italy,Iran,France,Spain,US
51,79,95,57,84,85
104,150,139,100,125,111
204,227,245,130,169,176
433,320,388,191,228,252
602,445,593,212,282,352
833,650,978,285,365,495
977,888,1501,423,430,640
1261,1128,2336,613,674,926
1766,1694,2922,949,1231,NaN
2337,2036,3513,1126,1696,NaN
3150,2502,4747,1412,NaN,NaN
4212,3089,5823,1748,NaN,NaN
4812,3858,6566,NaN,NaN,NaN
5328,4638,7161,NaN,NaN,NaN
5766,5883,8042,NaN,NaN,NaN
6284,7375,NaN,NaN,NaN,NaN
6767,9172,NaN,NaN,NaN,NaN
7134,10149,NaN,NaN,NaN,NaN
7382,NaN,NaN,NaN,NaN,NaN
7513,NaN,NaN,NaN,NaN,NaNWith this simple command, you get a certified"So fancy" plot.$catcovid.csv|phplot(Notice that this needsmatplotlib:pip install ph[plot])Raison d'êtreUsing thepipelinein Linux is nothing short of a dream in the life of the
computer super user.However the pipe is clearly most suited for a stream of lines of textual data,
and not when the stream is actually tabular data.Tabular data is much more complex to work with due to its dual indexing and the
fact that we often read horizontally and often read vertically.The defacto format for tabular data iscsv(comma-separated values,
which is not perfect in any sense
of the word), and the defacto tool for working with tabular data in Python is
Pandas.This is a shell utilityph(pronouncedphi)
that reads tabular data fromstandard inand allows
you to perform a pandas function on the data, before writing it to standard out
incsvformat.The goal is to create a tool which makes it nicer to work with tabular data in a
pipeline.To achieve the goal,phthen reads csv data, does some manipulation,
and prints out csv data.  With csv as the invariant,phcan be used in
a pipeline.A very quick introduction to whatphcan do for you,
run this in your shell:phopencsvhttps://raw.githubusercontent.com/owid/covid-19-data/master/public/data/archived/ecdc/total_cases.csv\|phslugify\|phcolumnsdatenorwayswedendenmark\|phdiffnorwayswedendenmark\|phspencernorwayswedendenmark\|phrolling7norwayswedendenmark--how=mean\|phdropna\|phslice50:\|phplot--linewidth=3--savefig=cases.svg--index=dateTable of contentsGetting startedExample usageThe toolsConcatenating, merging, filteringcat,open,fromdropnaandfillnaheadandtaildatemergeEditing the csvcolumns, listing, selecting and re-ordering ofrenamereplacesliceeval; Mathematipulating and creating new columnsnormalizequerygrepstripremoveprefixandremovesuffixAnalyzing the csv filedescribeshowtabulatesortvalues by columnplotgroupbyrolling,ewm,expandingindexpolyfitWorking with different formatsopentoandfrom; Exporting and importingSupported formatsGetting startedIf you have installedph[data], you can experiment usingph datasetif you
don't have an appropriate csv file available.phdatasetboston|phdescribeAvailable datasets are fromscikit-learn.datasetsToy datasets:bostonirisdiabetesdigitslinnerudwinebreast_cancerReal world:olivetti_faceslfw_peoplelfw_pairsrcv1kddcup99california_housingExample usageSuppose you have a csv filea.csvthat looks like this:x,y
3,8
4,9
5,10
6,11
7,12
8,13Transpose:$cata.csv|phtranspose0,1,2,3,4,53,4,5,6,7,88,9,10,11,12,13median(as well as many others, e.g.abs,corr,count,cov,cummax,cumsum,diff,max,product,quantile,rank,round,sum,std,varetc.):$cata.csv|phmedian
x,y5.5,10.5Useph helpto list all commandsThe toolsConcatenating, merging, filteringcat,open,fromcatIt is possible toconcatenate(cat) multiple csv-files withph cat:$phcata.csvb.csv--axis=index$phcata.csvb.csv--axis=columnsThe functionality is described inpandas.concat.openYou can open a csv, json, excel, gpx (etc., seesupported
formats) usingph open type file:$phopenexcela.xlsx$phopenexcela.xlsx--sheet_name=0--skiprows=3$phopentsva.tsv$phopencsva.csvIn the event that the csv data starts on the first line (i.e. no
header is present), use--header=None:$phopencsva.csv--header=NonefromTheph fromcommand works similarly toph openbut reads from stdin
instead of opening a file.  It therefore does not take a filename
argument:$cat/etc/passwd|phfromcsv--sep=':'--header=NonedropnaandfillnaConsider again thecovid.csvfile from above.$catcovid.csv|phdropnawill remove all rows that contain N/A (nan) values.  If we want to keep all
rows with at least 5 non-N/A values, we can use$catcovid.csv|phdropna--thresh=5If we want to drop allcolumnswith N/A values instead of allrows, we use--axis=1.If we want to drop only columns (resp. rows) withall n/avalues, we use--how=all.ToreplaceN/A values with other values, we can simply runcatcovid.csv|phfillna999.75If we instead want topadthe N/A values, we use--method=padcatcovid.csv|phfillna--method=padWe can limit the number of consecutive N/A values that are filled by using
(e.g.)--limit=7.headandtailUsingheadandtailworks approximately as the normal shell equivalents,
however they will preserve the header if there is one, e.g.$cata.csv|phhead7|phtail3x,y6,117,128,13dateIf thecsvfile contains a column, e.g. namedxcontaining
timestamps, it can be parsed as such withph date x:$cata.csv|phdatex
x,y1970-01-04,81970-01-05,91970-01-06,101970-01-07,111970-01-08,121970-01-09,13If your column is formatted withfreedom units,mm/dd/yyyy, you can
use the flag--dayfirst=True:dateRep,geoId
01/04/2020,US
31/03/2020,US
30/03/2020,US
29/03/2020,US
28/03/2020,US$cat~/cov.csv|phdatedateRep--dayfirst=True
dateRep,geoId2020-04-01,US2020-03-31,US2020-03-30,US2020-03-29,US2020-03-28,USTo get a column with integers (e.g. 3-8) parsed as, e.g. 2003 - 2008, some
amount of hacking is necessary.  We will go into details later on theevalandappendstr.$cata.csv|pheval"x = 2000 + x"|phappendstrx-|phdatex
x,y2003-01-01,82004-01-01,92005-01-01,102006-01-01,112007-01-01,122008-01-01,13However, it is possible to provide a--formatinstruction todate:$cata.csv|pheval"x = 2000 + x"|phdatex--format="%Y"x,y2003-01-01,82004-01-01,92005-01-01,102006-01-01,112007-01-01,122008-01-01,13Under some very special circumstances, we may have aunix timestampin
a column, in which the--utc=Truehandle becomes useful:Considerutc.csv:date,x,y
1580601600,3,8
1580688000,4,9
1580774400,5,10
1580860800,6,11
1580947200,7,12
1581033600,8,13where you get the correct dates:$catutc.csv|phdatedate--utc=True
date,x,y2020-02-02,3,82020-02-03,4,92020-02-04,5,102020-02-05,6,112020-02-06,7,122020-02-07,8,13mergeMerging two csv files is made available throughph merge f1 f2.Considerleft.csvkey1,key2,A,B
K0,K0,A0,B0
K0,K1,A1,B1
K1,K0,A2,B2
K2,K1,A3,B3andright.csvkey1,key2,C,D
K0,K0,C0,D0
K1,K0,C1,D1
K1,K0,C2,D2
K2,K0,C3,D3We can merge them using (default to--how=inner)$phmergeleft.csvright.csv
key1,key2,A,B,C,D
K0,K0,A0,B0,C0,D0
K1,K0,A2,B2,C1,D1
K1,K0,A2,B2,C2,D2or using anouterjoin:$phmergeleft.csvright.csv--how=outer
key1,key2,A,B,C,D
K0,K0,A0,B0,C0,D0
K0,K1,A1,B1,,
K1,K0,A2,B2,C1,D1
K1,K0,A2,B2,C2,D2
K2,K1,A3,B3,,
K2,K0,,,C3,D3and we can specify on which column to join:$phmergeleft.csvright.csv--on=key1--how=outer
key1,key2_x,A,B,key2_y,C,D
K0,K0,A0,B0,K0,C0,D0
K0,K1,A1,B1,K0,C0,D0
K1,K0,A2,B2,K0,C1,D1
K1,K0,A2,B2,K0,C2,D2
K2,K1,A3,B3,K0,C3,D3In the case when the two files do not share a common column key, we can
join them on key1 from the left file and key2 from the right file by specifying$phmergemergel.csvmerger.csv--left=key1--right=key2Editing the csvcolumns, listing, selecting and re-ordering ofConsiderc.csv:it,fr,de
79,57,79
157,100,130
229,130,165
323,191,203
470,212,262
655,285,545
889,423,670
1128,653,800
1701,949,1040
2036,1209,1224
2502,1412,1565
3089,1784,1966
3858,2281,2745
4636,2876,3675
5883,3661,4181Print the column names:$catc.csv|phcolumns
it
fr
deSelecting only certain columns, e.g.deandit$catc.csv|phcolumnsdeit|phtail3de,it2745,38583675,46364181,5883rename$catc.csv|phrenamedeGermany|phrenameitItaly|phcolumnsItalyGermany
Italy,Germany79,79157,130229,165323,203470,262655,545889,6701128,8001701,10402036,12242502,15653089,19663858,27454636,36755883,4181In addition torenamethere is an auxiliary functionslugifythat
lets youslugifythe column names.  Considerslugit.csvStupid column 1,  Jerky-column No. 2
3,8
4,9
5,10
6,11
7,12
8,13$catslugit.csv|phslugify
stupid_column_1,jerky_column_no_23,84,95,106,117,128,13Then you can do$catslugit.csv|phslugify|phrenamestupid_column_1first|phrenamejerky_column_no_2second
first,second3,84,95,106,117,128,13replaceWe can replace values in the data (or in a single column) usingph replace.  The syntax isph replace old new [--column=x [--newcolumn=xp]]:$cata.csv|phreplace8100x,y3,1004,95,106,117,12100,13$cata.csv|phreplace8100--column=x
x,y3,84,95,106,117,12100,13$cata.csv|phreplace8100--column=x--newcolumn=xp
x,y,xp3,8,34,9,45,10,56,11,67,12,78,13,100sliceSlicing in Python is essential, and occasionally, we want to slice
tabular data, e.g. look at only the 100 first, or 100 last rows, or
perhaps we want to look at only every 10th row.  All of this is achieved
usingph slice start:end:stepwith standard Python slice syntax.$cata.csv|phslice1:9:2
x,y4,96,118,13Reversing:$ cat a.csv|ph slice ::-1
x,y
8,13
7,12
6,11
5,10
4,9
3,8See alsoph headandph tail.$cata.csv|phslice:3
x,y3,84,95,10equivalent to$cata.csv|phhead3x,y3,84,95,10eval; Mathematipulating and creating new columnsYou can sum columns and place the result in a new column usingeval(frompandas.DataFrame.eval).$catc.csv|pheval"total = it + fr + de"|phtail3it,fr,de,total3858,2281,2745,88844636,2876,3675,111875883,3661,4181,13725$cata.csv|pheval"z = x**2 + y"x,y,z3,8,174,9,255,10,356,11,477,12,618,13,77If you only want the result, you leave theevalexpression without assignment$cata.csv|pheval"x**2"x91625364964normalizeYou can normalize a column usingph normalize col.$cata.csv|pheval"z = x * y"|phnormalizez
x,y,z3,8,0.04,9,0.155,10,0.3256,11,0.5257,12,0.758,13,1.0queryWe canquerydata usingph query expr.$cata.csv|phquery"x > 5"x,y6,117,128,13$phopencsv'http://bit.ly/2cLzoxH'|phquery"country == 'Norway'"|phtabulate--headerscountryyearpopcontinentlifeExpgdpPercap
-----------------------------------------------------------0Norway19523.32773e+06Europe72.6710095.41Norway19573.49194e+06Europe73.44116542Norway19623.63892e+06Europe73.4713450.43Norway19673.78602e+06Europe74.0816361.94Norway19723.933e+06Europe74.3418965.15Norway19774.04320e+06Europe75.3723311.36Norway19824.11479e+06Europe75.9726298.67Norway19874.18615e+06Europe75.89315418Norway19924.28636e+06Europe77.3233965.79Norway19974.40567e+06Europe78.3241283.210Norway20024.53559e+06Europe79.054468411Norway20074.62793e+06Europe80.19649357.2grepThe powerfulgrepis one of the most used command line tools, and it
would be silly to not ship a version of it ourselves.  Usingph grepis rarely necessary, but helps when you want to ensure the header is
kept.$cattxtfile.csv|phgrep"a|b"--case=False--column=Text_Column--regex=FalseThe arguments denote--caseshould be case sensitive?--columngrep only in given column--regexuse regex for pattern?stripOccasionally csv files come with additional spaces which can lead to
difficulties in parsing the cells' contents.  A csv file should be
formatted without spaces after the comma42,17over42, 17.  But
since we are human, we sometimes make mistakes.If we want tostrip, ortrim, the contents of a column, we useph strip:$cattxtfile.csv|phstripcol1col2removeprefixandremovesuffixIfstripis not sufficiently powerful, it is possible toremoveprefixorremovesuffixusing$cattxtfile.csv|phremoveprefixcol1patternand similarly forremovesuffix.Analyzing the csv filedescribeThe normal Pandasdescribeis of course available:$cata.csv|phdescribexy
count6.0000006.000000
mean5.50000010.500000
std1.8708291.870829
min3.0000008.00000025%4.2500009.25000050%5.50000010.50000075%6.75000011.750000
max8.00000013.000000showThe shorthandph showsimply calls the belowph tabulate --headers.$cata.csv|phshowxy
--------0381492510361147125813tabulateThe amazingtabulatetool comes from the Python packagetabulate on PyPI.Thetabulatecommand takes arguments--headersto toggle printing of header
row,--format=[grid,...]to modify the table style and--noindexto remove
the running index (leftmost column in the example above).Among the supported format styles areplain,simple,grid,fancy_grid,pretty,github,rst,mediawiki,html,latex,... (See full list at the project homepage atpython-tabulate.)sortvalues by columnYou can the columns in the csv data by a certain column:$catiris.csv|phsortsetosa|phtail5150,4,setosa,versicolor,virginica7.9,3.8,6.4,2.0,27.6,3.0,6.6,2.1,27.7,3.8,6.7,2.2,27.7,2.8,6.7,2.0,27.7,2.6,6.9,2.3,2plotYou can plot data usingph plot [--index=col].$phopenparquet1A_2019.parquet|phcolumnsTimeValue|phplot--index=TimeThis will take the columnsTimeandValuefrom the timeseries provided by
the givenparquetfile and plot theValueseries usingTimeasindex.The following example plots the life expectancy in Norway usingyearasindex:$phopencsvhttp://bit.ly/2cLzoxH|phquery"country == 'Norway'"|phappendstryear-01-01|phcolumnsyearlifeExp|phplot--index=yearNote:The strangeph appendstr year -01-01turns the items1956into"1956-01-01"and2005into"2005-01-01".  These are necessary to make
pandas to interpret1956as ayearand not as amillisecond.The commandph appendstr col str [newcol]takes a string and appends it to a
column, overwriting the original column, or writing it tonewcolif provided.Advanced plottingYou can choose thekindof plotting ( ‘line’, ‘bar’, ‘barh’, ‘hist’, ‘box’,
‘kde’, ‘density’, ‘area’, ‘pie’, ‘scatter’, ‘hexbin’), thestyleof plotting
(e.g.--style=o), and in case of scatter plot, you need to specify--x=col1and--y=col2, e.g.:$phopencsvhttp://bit.ly/2cLzoxH|phquery"continent == 'Europe'"|phplot--kind=scatter--x=lifeExp--y=gdpPercapTo specify the stylingk--gives a black dashed line:$phopencsvhttp://bit.ly/2cLzoxH|phquery"country == 'Norway'"|phappendstryear-01-01|phcolumnsyearlifeExp|phplot--index=year--style=k--UsingplotheadlessOccasionally we would like to generate a plot to an image(-like) file on
the command line or in a script, without necessarily launching any
graphic user interface.Callingph plotwith the argument--savefig=myfile.pngwill create a
PNG file calledmyfile.pnginstead of opening the matplotlib window.
It is also possible to get other formats by using different extensions,
likeeps,pdf,pgf,png,ps,raw,rgba,svg,svgz.iplotwithplotlyandcufflinksInstead of using thematplotlibbackend, there is an option for usingplotlyandcufflinksto generate
interactive plots.
This depends oncufflinks, and can be installed withpip install ph[iplot].$cata.csv|phiplot--kind=bar--barmode=stack$cata.csv|phiplot--kind=scatter--mode=markersgroupbySuppose you have a csv fileAnimal,Max Speed
Falcon,380.0
Falcon,370.0
Parrot,24.0
Parrot,26.0You can use Pandas'groupbyfunctionality to get the aggregatedsum,mean, orfirstvalue:$catgroup.csv|phgroupbyAnimal--how=mean
MaxSpeed375.025.0If you want to retain the index column,$catgroup.csv|phgroupbyAnimal--how=mean--as_index=False
Animal,MaxSpeed
Falcon,375.0
Parrot,25.0rolling,ewm,expandingrollingCompute rolling averages/sums usingph rolling 3 --how=meanConsider againa.csv:x,y
3,8
4,9
5,10
6,11
7,12
8,13Moving average with window size 3:$cata.csv|phrolling3--how=mean|phdropna
x,y4.0,9.05.0,10.06.0,11.07.0,12.0Rolling sum with window size 2:$cata.csv|phrolling2--how=sum|phdropna
x,y7.0,17.09.0,19.011.0,21.013.0,23.015.0,25.0ewm — exponentially weighted methods$cata.csv|phewm--com=0.5--how=mean|phshowxy
-----------------03813.758.7524.615389.6153835.5510.5546.5206611.520757.5082412.5082Use eithercom(center of mass),span,halflife, oralpha,
together with--how=mean,--how=std,--how=var, etc.expanding — expanding windowA common alternative to rolling statistics is to use an expanding
window, which yields the value of the statistic with all the data
available up to that point in time.$cata.csv|phexpanding3x,y
,
,12.0,27.018.0,38.025.0,50.033.0,63.0Spencer's 15-weight averageWe also support an experimental and slow version of Spencer's 15-weight
average.  This method takes a window of size 15, and pointwise multiply
with the following vector (normalized)(-3, -6, -5, 3, 21, 46, 67, 74, 67, 46, 21, 3, -5, -6, -3)and then takes the sum of the resulting vector.Spencer's 15-weight average is an interesting (impulse response) filter
that preserves all up to cubic polynomial functions.indexOccasionally you need to have an index, in which caseph indexis your tool:$cata.csv|phindex
index,x,y0,3,81,4,92,5,103,6,114,7,125,8,13polyfitYou can performlinear regressionandpolynomial regressionon a certain
index columnxand ay = f(x)column usingph polyfit.  It takes two
arguments, thexcolumn name, theycolumn name and an optional--deg=<degree>, the degree of the polynomial.  The default option is--deg=1which corresponds to a linear regression.Suppose you have a csv filelr.csvwith contentx,y
4,12
5,19
6,17
7,24
8,28
9,34With linear (polynomial) regression, you get an extra column,polyfit_{deg}:$catlr.csv|phpolyfitxy|phastypeint
x,y,polyfit_14,12,125,19,166,17,207,24,248,28,289,34,32Usingph plot --index=xresults in this plot:Working with different formatsopenPandas supports reading a multitude ofreaders.To read an Excel file and pipe the stream, you can useph open.The syntax ofph openisph open ftype fname, wherefnameis the
file you want to stream andftypeis the type of the file.A list of all available formats is given below.$phopenxlsa.xlsx
x,y3,84,95,106,117,128,13You can open asemicolon separated valuesfile using--sep=";"$phopencsv--sep=";"fname.csvtoandfrom; Exporting and importingObserve the following:{"x":{"0":3,"1":4,"2":5,"3":6,"4":7,"5":8},"y":{"0":8,"1":9,"2":10,"3":11,"4":12,"5":13}}Of course, then,$cata.csv|phtojson|phfromjson
x,y3,84,95,106,117,128,13This also means that$cata.csv|phtojson>a.json
$cata.json{"x":{"0":3,"1":4,"2":5,"3":6,"4":7,"5":8},"y":{"0":8,"1":9,"2":10,"3":11,"4":12,"5":13}}$cata.json|phfromjson
x,y3,84,95,106,117,128,13You can open Excel-like formats usingph open excel fname.xls[x],parquetfiles withph open parquet data.parquet.  Note that these two examples requirexlrdandpyarrow, respectively, or simplypip install ph[complete]Supported formatscsv/tsv(the latter for tab-separated values)fwf(fixed-width file format)jsonhtmlclipboard(pastes tab-separated content from clipboard)xlsodfhdf5featherparquetorcstatasasspsspicklesqlgbq/google/bigqueryWe also support reading GPX files withph open gpx.
This uses the GPX Python librarygpxpy.