Package: alphareader
AlphaReaderAfter several attempts to try thecsvpackage orpandasfor reading large files with custom delimiters, I ended up writting a little program that does the job without complaints.AlphaReaderis a high performant, pure python, 15-line of code library, that reads chunks of bytes from your files, and retrieve line by line, the content of it.The inspiration of this library came by having to extract data from a MS-SQL Server database, and having to deal with theCP1252encoding. By default AlphaReader takes this encoding as it was useful in our use case.It works also withHDFSthrough thepyarrowlibrary. But is not a depedency.CSVs# !cat file.csv# 1,John,Doe,2010# 2,Mary,Smith,2011# 3,Peter,Jones,2012>reader=AlphaReader(open('file.csv','rb'),encoding='cp1252',terminator=10,delimiter=44)>next(reader)>['1','John','Doe','2010']TSVs# !cat file.tsv# 1    John    Doe    2010# 2    Mary    Smith  2011# 3    Peter   Jones  2012>reader=AlphaReader(open('file.tsv','rb'),encoding='cp1252',terminator=10,delimiter=9)>next(reader)>['1','John','Doe','2010']XSVs# !cat file.tsv# 1Â¦JohnÂ¦DoeÂ¦2010# 2Â¦MaryÂ¦SmithÂ¦2011# 3Â¦PeterÂ¦JonesÂ¦2012>ord('Â¦')>166>chr(166)>'Â¦'>reader=AlphaReader(open('file.tsv','rb'),encoding='cp1252',terminator=10,delimiter=166)>next(reader)>['1','John','Doe','2010']HDFS# !hdfs dfs -cat /raw/tsv/file.tsv# 1    John    Doe    2010# 2    Mary    Smith  2011# 3    Peter   Jones  2012>importpyarrowaspa>fs=pa.hdfs.connect()>reader=AlphaReader(fs.open('/raw/tsv/file.tsv','rb'),encoding='cp1252',terminator=10,delimiter=9)>next(reader)>['1','John','Doe','2010']Transformations# !cat file.csv# 1,2,3# 10,20,30# 100,200,300>fn=lambdax:int(x)>reader=AlphaReader(open('/raw/tsv/file.tsv','rb'),encoding='cp1252',terminator=10,delimiter=44,fn_transform=fn)>next(reader)>[1,2,3]>next(reader)>[10,20,30]Chain Transformations# !cat file.csv# 1,2,3# 10,20,30# 100,200,300>fn_1=lambdax:x+1>fn_2=lambdax:x*10>reader=AlphaReader(open('/raw/tsv/file.tsv','rb'),encoding='cp1252',terminator=10,delimiter=44,fn_transform=[int,fn_1,fn_2])>next(reader)>[20,30,40]>next(reader)>[110,210,310]Caution>reader=AlphaReader(open('large_file.xsv','rb'),encoding='cp1252',terminator=172,delimiter=173)>records=list(reader)# Avoid this as it will load all file in memoryLimitationsNo support formulti-bytedelimitersRelatively slower performance thancsvlibrary. Usecsvand dialects when your files have\r\nterminatorsTransformations are per row, perhaps vectorization could aid performancePerformance24MB file loaded withlist(AlphaReader(file_handle))tests/test_profile.py::test_alphareader_with_encoding
---------------------------------------------------------------------------------livelogcallINFOroot:test_profile.py:22252343functioncallsin0.386secondsOrderedby:cumulativetimencallstottimepercallcumtimepercallfilename:lineno(function)1196050.0390.0000.3860.000.\alphareader\__init__.py:39(AlphaReader)1222280.2660.0000.2660.000{method'split'of'str'objects}26250.0050.0000.0540.000{method'decode'of'bytes'objects}26240.0010.0000.0490.000.\Python-3.7.4\lib\encodings\cp1252.py:14(decode)26240.0480.0000.0480.000{built-inmethod_codecs.charmap_decode}26250.0270.0000.0270.000{method'read'of'_io.BufferedReader'objects}10.0000.0000.0000.000.\__init__.py:5(_validate)10.0000.0000.0000.000{built-inmethod_codecs.lookup}