Package: csvnav
CSVNAV: a python 3 class for memory-efficient navigation of CSV/Text files.This package can be installed with pip:pip install csvnavor by downloading this repo and using setup tools:pythonsetup.pyinstallrun from within thecsvnavdirectory.The filecsvnav.pyis a python module containing the classNavigator. When instantiated,Navigatorwill open a given path and then store pointers to the location of each row in the opened file. In the simplest case, one can use the instantiation sort of like a list. For instance, if I have a file "inventory.csv" containing the following CSV data:time,product,quantity
5,tire,4
8,sparkplug,20
2,battery,120
10,tire,2
11,tire,3
30,sparkplug,35I can instantiate the class and query rows by index:fromcsvnavimportNavigatornav=Navigator('./inventory.csv',header=True,delimiter=',')print(nav[0])print(nav[2])print(nav.size(force=True))nav.close()where the output would be:{'product': 'tire', 'quantity': '4', 'time': '5'}
{'product': 'battery', 'quantity': '120', 'time': '2'}
6Note that the number of data rows (excluding any skipped lines and the header) can be printed by callingNavigator.size(force=True). In this case,force=Truemeans that the number of data rows in the file will be determined even if the last row in the file has not be accessed yet. If the last row had been accessed,force=Falsewould return the same result. However, if the last row had not yet been accessed,force=Falsewould returnNone. Another thing to note is that the rows are returned as a dictionary. As long asNavigator.headercontains a list of the column names (done automatically from the first row of the CSV file after any skipped lines whenheader=Truein instantiation or when column names are provided with theNavigator.set_header()method), the rows will be returned as a dictionary. Otherwise, the rows are returned as lists. For example, if "inventory.csv" did not have a header then the output would be:['5', 'tire', '4']
['2', 'battery', '120']
6TheNavigatorclass is also iterable and will iterate through rows in order:forrowinnav:print(row)gives the output (assuming we have a header):{'time': '5', 'product': 'tire', 'quantity': '4'}
{'time': '8', 'product': 'sparkplug', 'quantity': '20'}
{'time': '2', 'product': 'battery', 'quantity': '120'}
{'time': '10', 'product': 'tire', 'quantity': '2'}
{'time': '11', 'product': 'tire', 'quantity': '3'}
{'time': '30', 'product': 'sparkplug', 'quantity': '35'}If we only want to iterate through a subset of rows that match a condition, we can use theNavigator.filtermethod:fromcsvnavimportNavigatornav=Navigator('./inventory.csv',header=True,delimiter=',')defwhen_few_tires(row):ifrow['product']=='tire'andint(row['quantity'])<=3:returnTrueelse:returnFalseforrowinnav.filter(when_few_tires):print(row)nav.close()will produce the output:{'time': '10', 'product': 'tire', 'quantity': '2'}
{'time': '11', 'product': 'tire', 'quantity': '3'}Another usage of the class is to group pointers by column name (assumingNavigator.headeris set). This can be done with theNavigator.registermethod.
The following code will then group rows by product and show how this data can be accessed:fromcsvnavimportNavigatornav=Navigator('./inventory.csv',header=True,delimiter=',')nav.register('product')# can also provide a list of columns to register eachprint(nav.fields)print(nav.keys('product'))fork,vinnav.items('product'):print(k,list(v))nav.close()will print out the following groups (list of dict or list):dict_keys(['product'])
dict_keys(['tire', 'sparkplug', 'battery'])
tire [{'time': '5', 'product': 'tire', 'quantity': '4'}, {'time': '10', 'product': 'tire', 'quantity': '2'}, {'time': '11', 'product': 'tire', 'quantity': '3'}]
sparkplug [{'time': '8', 'product': 'sparkplug', 'quantity': '20'}, {'time': '30', 'product': 'sparkplug', 'quantity': '35'}]
battery [{'time': '2', 'product': 'battery', 'quantity': '120'}]Note that groups are then accessed by two "indexes", namely the column name and the key.TheNavigatorclass should be thread safe and an instance can be shared between threads.Navigatorhas some more functionality that I have not described here but this covers the basics. Refer to the docstrings of the various methods of theNavigatorclass for more information.