Package: meza
meza: A Python toolkit for processing tabular dataIndexIntroduction|Requirements|Motivation|Hello World|Usage|Interoperability|Installation|Project Structure|Design Principles|Scripts|Contributing|Credits|More Info|LicenseIntroductionmezais a Pythonlibraryfor reading and processing tabular data.
It has a functional programming style API, excels at reading/writing large files,
and can process 10+ file types.With meza, you canRead csv/xls/xlsx/mdb/dbf files, and more!Type cast records (date, float, text…)Process Uñicôdë textLazily stream files by defaultand much more…Requirementsmeza has been tested and is known to work on Python 3.7, 3.8, and 3.9; and PyPy3.7.Optional DependenciesFunctionDependencyInstallationFile type / extensionmeza.io.read_mdbmdbtoolssudo port install mdbtoolsMicrosoft Access / mdbmeza.io.read_htmllxml[1]pip install lxmlHTML / htmlmeza.convert.records2arrayNumPy[2]pip install numpyn/ameza.convert.records2dfpandaspip install pandasn/aNotes[1]Iflxmlisn’t present,read_htmlwill default to the builtin Python html reader[2]records2arraycan be used withoutnumpyby passingnative=Truein the function call. This will convertrecordsinto a list of nativearray.arrayobjects.MotivationWhy I built mezapandas is great, but installing it isn’t exactly awalk in the park, and it
doesn’t play nice withPyPy. I designedmezato be a lightweight, easy to install, less featureful alternative to
pandas. I also optimizedmezafor low memory usage, PyPy compatibility, and functional programming best practices.Why you should use mezamezaprovides a number of benefits / differences from similar libraries such
aspandas. Namely:a functional programming (instead of object oriented) APIiterators by default(reading/writing)PyPy compatibilitygeojson support(reading/writing)seamless integrationwith sqlachemy (and other libs that work with iterators of dicts)For more detailed information, please check-out theFAQ.Hello WorldA simple data processing example is shown below:First create a simple csv file (in bash)echo'col1,col2,col3\nhello,5/4/82,1\none,1/1/15,2\nhappy,7/1/92,3\n'>data.csvNow we can read the file, manipulate the data a bit, and write the manipulated
data back to a new file.>>>frommezaimportio,processaspr,convertascv>>>fromioimportopen>>># Load the csv file>>>records=io.read_csv('data.csv')>>># `records` are iterators over the rows>>>row=next(records)>>>row{'col1':'hello','col2':'5/4/82','col3':'1'}>>># Let's replace the first row so as not to lose any data>>>records=pr.prepend(records,row)# Guess column types. Note: `detect_types` returns a new `records`# generator since it consumes rows during type detection>>>records,result=pr.detect_types(records)>>>{t['id']:t['type']fortinresult['types']}{'col1':'text','col2':'date','col3':'int'}# Now type cast the records. Note: most `meza.process` functions return# generators, so lets wrap the result in a list to view the data>>>casted=list(pr.type_cast(records,result['types']))>>>casted[0]{'col1':'hello','col2':datetime.date(1982,5,4),'col3':1}# Cut out the first column of data and merge the rows to get the max value# of the remaining columns. Note: since `merge` (by definition) will always# contain just one row, it is returned as is (not wrapped in a generator)>>>cut_recs=pr.cut(casted,['col1'],exclude=True)>>>merged=pr.merge(cut_recs,pred=bool,op=max)>>>merged{'col2':datetime.date(2015,1,1),'col3':3}# Now write merged data back to a new csv file.>>>io.write('out.csv',cv.records2csv(merged))# View the result>>>withopen('out.csv','utf-8')asf:...f.read()'col2,col3\n2015-01-01,3\n'Usagemeza is intended to be used directly as a Python library.Usage IndexReading dataProcessing dataNumerical analysis (à la pandas)Text processing (à la csvkit)Geo processing (à la mapbox)Writing dataCookbookReading datameza can read both filepaths and file-like objects. Additionally, all readers
return equivalentrecordsiterators, i.e., a generator of dictionaries with
keys corresponding to the column names.>>>fromioimportopen,StringIO>>>frommezaimportio"""Read a filepath""">>>records=io.read_json('path/to/file.json')"""Read a file like object and de-duplicate the header""">>>f=StringIO('col,col\nhello,world\n')>>>records=io.read_csv(f,dedupe=True)"""View the first row""">>>next(records){'col':'hello','col_2':'world'}"""Read the 1st sheet of an xls file object opened in text mode."""# Also, santize the header names by converting them to lowercase and# replacing whitespace and invalid characters with `_`.>>>withopen('path/to/file.xls','utf-8')asf:...forrowinio.read_xls(f,sanitize=True):...# do something with the `row`...pass"""Read the 2nd sheet of an xlsx file object opened in binary mode"""# Note: sheets are zero indexed>>>withopen('path/to/file.xlsx')asf:...records=io.read_xls(f,encoding='utf-8',sheet=1)...first_row=next(records)...# do something with the `first_row`"""Read any recognized file""">>>records=io.read('path/to/file.geojson')>>>f.seek(0)>>>records=io.read(f,ext='csv',dedupe=True)Please seereadersfor a complete list of available readers and recognized
file types.Processing dataNumerical analysis (à la pandas)[3]In the following example,pandasequivalent methods are preceded by-->.>>>importitertoolsasit>>>importrandom>>>fromioimportStringIO>>>frommezaimportio,processaspr,convertascv,stats# Create some data in the same structure as what the various `read...`# functions output>>>header=['A','B','C','D']>>>data=[(random.random()for_inrange(4))forxinrange(7)]>>>df=[dict(zip(header,d))fordindata]>>>df[0]{'A':0.53908...,'B':0.28919...,'C':0.03003...,'D':0.65363...}"""Sort records by the value of column `B` --> df.sort_values(by='B')""">>>next(pr.sort(df,'B')){'A':0.53520...,'B':0.06763...,'C':0.02351...,'D':0.80529...}"""Select column `A` --> df['A']""">>>next(pr.cut(df,['A'])){'A':0.53908170489952006}"""Select the first three rows of data --> df[0:3]""">>>len(list(it.islice(df,3)))3"""Select all data whose value for column `A` is less than 0.5
--> df[df.A < 0.5]
""">>>next(pr.tfilter(df,'A',lambdax:x<0.5)){'A':0.21000...,'B':0.25727...,'C':0.39719...,'D':0.64157...}# Note: since `aggregate` and `merge` (by definition) return just one row,# they return them as is (not wrapped in a generator)."""Calculate the mean of column `A` across all data --> df.mean()['A']""">>>pr.aggregate(df,'A',stats.mean)['A']0.5410437473067938"""Calculate the sum of each column across all data --> df.sum()""">>>pr.merge(df,pred=bool,op=sum){'A':3.78730...,'C':2.82875...,'B':3.14195...,'D':5.26330...}Text processing (à la csvkit)[4]In the following example,csvkitequivalent commands are preceded by-->.First create a few simple csv files (in bash)echo'col_1,col_2,col_3\n1,dill,male\n2,bob,male\n3,jane,female'>file1.csvecho'col_1,col_2,col_3\n4,tom,male\n5,dick,male\n6,jill,female'>file2.csvNow we can read the files, manipulate the data, convert the manipulated data to
json, and write the json back to a new file. Also, note that since all readers
return equivalentrecordsiterators, you can use them interchangeably (in
place ofread_csv) to open any supported file. E.g.,read_xls,read_sqlite, etc.>>>importitertoolsasit>>>frommezaimportio,processaspr,convertascv"""Combine the files into one iterator
--> csvstack file1.csv file2.csv
""">>>records=io.join('file1.csv','file2.csv')>>>next(records){'col_1':'1','col_2':'dill','col_3':'male'}>>>next(it.islice(records,4,None)){'col_1':'6','col_2':'jill','col_3':'female'}# Now let's create a persistent records list>>>records=list(io.read_csv('file1.csv'))"""Sort records by the value of column `col_2`
--> csvsort -c col_2 file1.csv
""">>>next(pr.sort(records,'col_2')){'col_1':'2','col_2':'bob','col_3':'male'"""Select column `col_2` --> csvcut -c col_2 file1.csv""">>>next(pr.cut(records,['col_2'])){'col_2':'dill'}"""Select all data whose value for column `col_2` contains `jan`
--> csvgrep -c col_2 -m jan file1.csv
""">>>next(pr.grep(records,[{'pattern':'jan'}],['col_2'])){'col_1':'3','col_2':'jane','col_3':'female'}"""Convert a csv file to json --> csvjson -i 4 file1.csv""">>>io.write('file.json',cv.records2json(records))# View the result>>>withopen('file.json','utf-8')asf:...f.read()'[{"col_1": "1", "col_2": "dill", "col_3": "male"}, {"col_1": "2","col_2":"bob","col_3":"male"},{"col_1":"3","col_2":"jane","col_3":"female"}]'Geo processing (à la mapbox)[5]In the following example,mapboxequivalent commands are preceded by-->.First create a geojson file (in bash)echo'{"type": "FeatureCollection","features": ['>file.geojsonecho'{"type": "Feature", "id": 11, "geometry": {"type": "Point", "coordinates": [10, 20]}},'>>file.geojsonecho'{"type": "Feature", "id": 12, "geometry": {"type": "Point", "coordinates": [5, 15]}}]}'>>file.geojsonNow we can open the file, split the data by id, and finally convert the split data
to a new geojson file-like object.>>>frommezaimportio,processaspr,convertascv# Load the geojson file and peek at the results>>>records,peek=pr.peek(io.read_geojson('file.geojson'))>>>peek[0]{'lat':20,'type':'Point','lon':10,'id':11}"""Split the records by feature ``id`` and select the first feature
--> geojsplit -k id file.geojson
""">>>splits=pr.split(records,'id')>>>feature_records,name=next(splits)>>>name11"""Convert the feature records into a GeoJSON file-like object""">>>geojson=cv.records2geojson(feature_records)>>>geojson.readline()'{"type": "FeatureCollection", "bbox": [10, 20, 10, 20], "features": ''[{"type": "Feature", "id": 11, "geometry": {"type": "Point", ''"coordinates": [10, 20]}, "properties": {"id": 11}}], "crs": {"type": ''"name", "properties": {"name": "urn:ogc:def:crs:OGC:1.3:CRS84"}}}'# Note: you can also write back to a file as shown previously# io.write('file.geojson', geojson)Writing datameza can persistrecordsto disk via the following functions:meza.convert.records2csvmeza.convert.records2jsonmeza.convert.records2geojsonEach function returns a file-like object that you can write to disk viameza.io.write('/path/to/file',result).>>>frommezaimportio,convertascv>>>fromioimportStringIO,open# First let's create a simple tsv file like object>>>f=StringIO('col1\tcol2\nhello\tworld\n')>>>f.seek(0)# Next create a records list so we can reuse it>>>records=list(io.read_tsv(f))>>>records[0]{'col1':'hello','col2':'world'}# Now we're ready to write the records data to file"""Create a csv file like object""">>>cv.records2csv(records).readline()'col1,col2\n'"""Create a json file like object""">>>cv.records2json(records).readline()'[{"col1": "hello", "col2": "world"}]'"""Write back csv to a filepath""">>>io.write('file.csv',cv.records2csv(records))>>>withopen('file.csv','utf-8')asf_in:...f_in.read()'col1,col2\nhello,world\n'"""Write back json to a filepath""">>>io.write('file.json',cv.records2json(records))>>>withopen('file.json','utf-8')asf_in:...f_in.readline()'[{"col1": "hello", "col2": "world"}]'CookbookPlease see thecookbookor ipythonnotebookfor more examples.Notes[3]http://pandas.pydata.org/pandas-docs/stable/10min.html#min[4]https://csvkit.readthedocs.org/en/0.9.1/cli.html#processing[5]https://github.com/mapbox?utf8=%E2%9C%93&query=geojsonInteroperabilitymeza plays nicely with NumPy and friends out of the boxsetupfrommezaimportprocessaspr# First create some records and types. Also, convert the records to a list# so we can reuse them.>>>records=[{'a':'one','b':2},{'a':'five','b':10,'c':20.1}]>>>records,result=pr.detect_types(records)>>>records,types=list(records),result['types']>>>types[{'type':'text','id':'a'},{'type':'int','id':'b'},{'type':'float','id':'c'}]from records to pandas.DataFrame to records>>>importpandasaspd>>>frommezaimportconvertascv"""Convert the records to a DataFrame""">>>df=cv.records2df(records,types)>>>dfabc0one2NaN1five1020.1# Alternatively, you can do `pd.DataFrame(records)`"""Convert the DataFrame back to records""">>>next(cv.df2records(df)){'a':'one','b':2,'c':nan}from records to arrays to records>>>importnumpyasnp>>>fromarrayimportarray>>>frommezaimportconvertascv"""Convert records to a structured array""">>>recarray=cv.records2array(records,types)>>>recarrayrec.array([('one',2,nan),('five',10,20.100000381469727)],dtype=[('a','O'),('b','<i4'),('c','<f4')])>>>recarray.barray([2,10],dtype=int32)"""Convert records to a native array""">>>narray=cv.records2array(records,types,native=True)>>>narray[[array('u','a'),array('u','b'),array('u','c')],[array('u','one'),array('u','five')],array('i',[2,10]),array('f',[0.0,20.100000381469727])]"""Convert a 2-D NumPy array to a records generator""">>>data=np.array([[1,2,3],[4,5,6]],np.int32)>>>dataarray([[1,2,3],[4,5,6]],dtype=int32)>>>next(cv.array2records(data)){'column_1':1,'column_2':2,'column_3':3}"""Convert the structured array back to a records generator""">>>next(cv.array2records(recarray)){'a':'one','b':2,'c':nan}"""Convert the native array back to records generator""">>>next(cv.array2records(narray,native=True)){'a':'one','b':2,'c':0.0}Installation(You are using avirtualenv, right?)At the command line, install meza using eitherpip(recommended)pipinstallmezaoreasy_installeasy_installmezaPlease see theinstallation docfor more details.Project Structure┌──CONTRIBUTING.rst├──LICENSE├──MANIFEST.in├──Makefile├──README.rst├──data│├──converted/*│└──test/*├──dev-requirements.txt├──docs│├──AUTHORS.rst│├──CHANGES.rst│├──COOKBOOK.rst│├──FAQ.rst│├──INSTALLATION.rst│└──TODO.rst├──examples│├──usage.ipynb│└──usage.py├──helpers/*├──manage.py├──meza│├──__init__.py│├──convert.py│├──dbf.py│├──fntools.py│├──io.py│├──process.py│├──stats.py│├──typetools.py│└──unicsv.py├──optional-requirements.txt├──py2-requirements.txt├──requirements.txt├──setup.cfg├──setup.py├──tests│├──__init__.py│├──standard.rc│├──test_fntools.py│├──test_io.py│└──test_process.py└──tox.iniDesign Principlesprefer functions over objectsprovide enough functionality out of the box to easily implement the most common data analysis use casesmake conversion betweenrecords,arrays, andDataFramesdead simplewhenever possible, lazily read objects and stream the result[6][6]Notable exceptions aremeza.process.group,meza.process.sort,meza.io.read_dbf,meza.io.read_yaml, andmeza.io.read_html. These functions read the entire contents into memory up front.Scriptsmeza comes with a built in task managermanage.pySetuppipinstall-rdev-requirements.txtExamplesRun python linter and nose testsmanagelintmanagetestContributingPlease mimic the coding style/conventions used in this repo.
If you add new classes or functions, please add the appropriate doc blocks with
examples. Also, make sure the python linter and nose tests pass.Please see thecontributing docfor more details.CreditsShoutouts tocsvkit,messytables, andpandasfor heavily inspiring meza.More InfoFAQcookbookipythonnotebookLicensemeza is distributed under theMIT License.ChangelogHere you can find the recent changes to meza..