Package: casanova
casanovaIf you often find yourself processing CSV files using python, you will quickly notice that, while being more comfortable,csv.DictReaderremains way slower thancsv.reader:# To read a 1.5G CSV file:
csv.reader: 24s
csv.DictReader: 84s
casanova.reader: 25scasanovais therefore an attempt to stick tocsv.readerperformance while still keeping a comfortable interface, still able to consider headers (even duplicate ones also, something thatcsv.DictReaderis incapable of) etc.casanovais thus a good fit for you if you need to:Stream large CSV files without running out of memoryEnrich the same CSV files by outputing a similar file, all while adding, filtering and editing cells.Have the possibility to resume said enrichment if your process exitedResume even if your output does not have the same order as the inputcasanovaalso packs exotic utilities able to read csv files in reverse (without loading the whole file into memory and in regularO(n)time), so you can, for instance, fetch useful information at the end of a file to restart some aborted process.Finally,casanovacan be used as a command line tool able to evaluate python expressions (that can be parallelized if required) for each row of a given CSV file to produce typical results such as adding a column based on others etc.The command line tool documentation can be foundhere.For more generic task that don't require python evaluation, we recommend the very performantxsvtool instead, or our ownforkof the tool.InstallationYou can installcasanovawith pip with the following command:pip install casanovaIf you want to be able to feed CSV files from the web tocasanovareaders & enrichers you will also need to install at leasturllib3and optionallycertifi(if you want secure SSL). Nnote that a lot of python packages, including the popularrequestslibrary, already depend on those two, so it is likely you already have them installed anyway:# Installing them explicitly
pip install urllib3 certifi

# Installing casanova with those implicitly
pip install casanova[http]Usagereaderreverse_readerheaderswriterenricherindexed_enricherbatch_enricherresumersRowCountResumerIndexedResumerBatchResumerLastCellResumerLastCellComparisonResumercountlast_cellset_defaultsxsv selection mini DSLreaderStraightforward CSV reader yielding list rows but giving some information about potential headers and their ipositions.importcasanovawithopen('./people.csv')asf:# Creating a readerreader=casanova.reader(f)# Getting header informationreader.fieldnames>>>['name','surname']reader.headers>>>Headers(name=0,surname=1)name_pos=reader.headers.namename_pos=reader.headers['name']'name'inreader.headers>>>True# Iterating over the rowsforrowinreader:name=row[name_pos]# it's better to cache your pos outside the loopname=row[reader.headers.name]# this works, but is slower# Interested in a single column?fornameinreader.cells('name'):print(name)# Need also the current row when iterating on cells?forrow,nameinreader.cells('name',with_rows=True):print(row,name,surname)# Want to iterate over records# NOTE: this has a performance costforname,surnameinreader.records('name','surname'):print(name,surname)forrecordinreader.records(['name','age']):print(record[0])forrecordinreader.records({'name':'name','age':1}):print(record['age'])# No headers? No problem.reader=casanova.reader(f,no_headers=True)# Note that you can also create a reader from a pathwithcasanova.reader('./people.csv')asreader:...# And if you need exotic encodingswithcasanova.reader('./people.csv',encoding='latin1')asreader:...# The reader will also handle gzipped files out of the boxwithcasanova.reader('./people.csv.gz')asreader:...# If you have `urllib3` installed, casanova is also able to stream# remote CSV file out of the boxwithcasanova.reader('https://mydomain.fr/some-file.csv')asreader:...# The reader will also accept iterables of rowsrows=[['name','surname'],['John','Moran']]reader=casanova.reader(rows)# And you can of course use the typical dialect-related kwargsreader=casanova.reader('./french-semicolons.csv',delimiter=';')# Readers can also be closed if you want to avoid context managersreader.close()Argumentsinput_filestr or Path or file or Iterable[list[str]]: input file given to the reader. Can be a path that will be opened for you by the reader, a file handle or even an arbitrary iterable of list rows.no_headersbool, optional[False]: set toTrueifinput_filehas no headers.encodingstr, optional[utf-8]: encoding to use to open the file ifinput_fileis a path.dialectstr or csv.Dialect, optional: CSV dialect for the reader to use. Check python standardcsvmodule documentation for more info.quotecharstr, optional: quote character used by CSV parser.delimiterstr, optional: delimiter characted used by CSV parser.prebuffer_bytesint, optional: number of bytes of input file to prebuffer in attempt to get a total number of lines ahead of time.totalint, optional: total number of lines to expect in file, if you already know it ahead of time. If given, the reader won't prebuffer data even ifprebuffer_byteswas set.multiplexcasanova.Multiplexer, optional: multiplexer to use. Readthisfor more information.strip_null_bytes_on_readbool, optional[False]: before python 3.11, thecsvmodule will raise when attempting to read a CSV file containing null bytes. If set toTrue, the reader will strip null bytes on the fly while parsing rows.reversebool, optional[False]: whether to read the file in reverse (except for the header of course).Propertiestotalint, optional: total number of lines in the file, if known through prebuffering or through thetotalkwarg.headerscasanova.Headers, optional, optional: CSV file headers ifno_headers=False.emptybool: whether the given file was empty.fieldnameslist[str], optional: list representing the CSV file headers ifno_headers=False.row_lenint: expected number of items per row.Methodsrows: returns an iterator over the reader rows. Same as iterating over the reader directly.cells: take the name of a column or its position and returns an iterator over values of the given column. Can be givenwith_rows=Trueif you want to iterate over avalue, rowtuple instead if required.enumerate: resuming-safe enumeration over rows yieldingindex, rowtuples. Takes an optionalstartkwarg like builtinenumerate.enumerate_cells: resuming-safe enumeration over cells yieldingindex, cellorindex, row, cellif givenwith_rows=True. Takes an optionalstartkwarg like builtinenumerate.wrap: method taking a list row and returning aRowWrapperobject to wrap it.close: cleans up the reader resources manually when not using the dedicated context manager. It is usually only useful when the reader was given a path and not an already opened file handle.MultiplexingSometimes, one column of your CSV file might contain multiple values, separated by an arbitrary separator character such as|.In this case, it might be desirable to "multiplex" the file by making a reader emit one copy of the line with each of the values contained by a cell.To do so,casanovaexposes a specialMultiplexerobject you can give to any reader like so:importcasanova# Most simple case: a column named "colors", separated by "|"reader=casanova.reader(input_file,multiplex=casanova.Multiplexer('colors'))# Customizing the separator:reader=casanova.reader(input_file,multiplex=casanova.Multiplexer('colors',separator='$'))# Renaming the column on the fly:reader=casanova.reader(input_file,multiplex=casanova.Multiplexer('colors',new_column='color'))reverse_readerA reverse CSV reader might sound silly, but it can be useful in some scenarios. Especially when you need to read the last line from an output file without reading the whole thing first, in constant time.It is mostly used bycasanovaresumersand it is unlikely you will need to use them on your own.importcasanova# people.csv looks like this# name,surname# John,Doe,# Mary,Albert# Quentin,Goldwithopen('./people.csv','rb')asf:reader=casanova.reverse_reader(f)reader.fieldnames>>>['name','surname']next(reader)>>>['Quentin','Gold']headersA class representing the headers of a CSV file. It is useful to find the row position of some columns and perform complex selection.importcasanova# Headers can be instantiated thuslyheaders=casanova.headers(['name','surname','age'])# But you will usually use a reader or an enricher's one:headers=casanova.reader(input_file).headers# Accessing a column through attributesheaders.surname>>>1# Accessing a column by indexing:headers['surname']>>>1# Getting a columnheaders.get('surname')>>>1headers.get('not-found')>>>None# Getting a duplicated column namecasanova.headers(['surname','name','name'])['name',1]>>>2casanova.headers(['surname','name','name']).get('name',index=1)>>>2# Asking if a column exists:'name'inheaders:>>>True# Retrieving fieldnames:headers.fieldnames>>>['name','surname','age']# Iterating over headersforcolinheaders:print(col)# Couting columns:len(headers)>>>3# Retrieving the nth header:headers.nth(1)>>>'surname'# Wraping a rowheaders.wrap(['John','Matthews','45'])>>>RowWrapper(name='John',surname='Matthews',age='45')# Selecting some columns (by name and/or index)):headers.select(['name',2])>>>[0,2]# Selecting using xsv mini DSL:headers.select('name,age')>>>[0,2]headers.select('!name')>>>[1,2]For more info about xsv mini DSL, checkthispart of the documentation.writercasanovaalso exports a csv writer. It can automatically write headers when needed and is able to resume some tasks.importcasanovawithopen('output.csv')asf:writer=casanova.writer(f,fieldnames=['name','surname'])writer.writerow(['John','Davis'])# If you want to write headers yourself:writer=casanova.writer(f,fieldnames=['name','surname'],write_header=False)writer.writeheader()Argumentsoutput_filefile or casanova.Resumer: target file.fieldnamesIterable[str], optional: column names.strip_null_bytes_on_writebool, optional[False]: whether to strip null bytes when writing rows. Note that on python 3.10, there is a bug that prevents acsv.writerwill raise an error when attempting to write a row containing a null byte.dialectcsv.Dialect or str, optional: dialect to use to write CSV.delimiterstr, optional: CSV delimiter.quotecharstr, optional: CSV quoting character.quotingcsv.QUOTE_*, optional: CSV quoting strategy.escapecharstr, optional: CSV escaping character.lineterminatorstr, optional: CSV line terminator.write_headerbool, optional[True]: whether to automatically write header if required (takes resuming into account).Propertiesheaderscasanova.Headers, optional, optional: CSV file headers if fieldnames were providedfieldnameslist[str], optional: provided fieldnames.ResumingAcasanova.writeris able to resume through aLastCellResumer.enrichercasanovaenrichers are basically a smart combination of both a reader and a writer.It can be used to transform a given CSV file. This means you can transform its values on the fly, select some columns to keep from input and add new ones very easily.Note that enrichers inherits from bothcasanova.readerandcasanova.writerand therefore keep both their properties and methods.importcasanovawithopen('./people.csv')asinput_file,\open('./enriched-people.csv','w')asoutput_file:enricher=casanova.enricher(input_file,output_file)# The enricher inherits from casanova.readerenricher.fieldnames>>>['name','surname']# You can iterate over its rowsname_pos=enricher.headers.nameforrowinenricher:# Editing a cell, so that everyone is called John nowrow[name_pos]='John'enricher.writerow(row)# Want to add columns?enricher=casanova.enricher(f,of,add=['age','hair'])forrowinenricher:enricher.writerow(row,['34','blond'])# Want to keep only some columns from input?enricher=casanova.enricher(f,of,add=['age'],select=['surname'])forrowinenricher:enricher.writerow(row,['45'])# Want to select columns to keep using xsv mini dsl?enricher=casanova.enricher(f,of,select='!1-4')# You can of course still use #.cells etc.forrow,nameinenricher.cells('name',with_rows=True):print(row,name)Argumentsinput_filefile or str: file object to read or path to open.output_filefile or Resumer: file object to write.no_headersbool, optional[False]: set toTrueifinput_filehas no headers.encodingstr, optional[utf-8]: encoding to use to open the file ifinput_fileis a path.addIterable[str|int], optional: names of columns to add to output.selectIterable[str|int]|str, optional: selection of columns to keep from input. Can be an iterable of column names and/or column positions or a selection string writting inxsv mini DSL.dialectstr or csv.Dialect, optional: CSV dialect for the reader to use. Check python standardcsvmodule documentation for more info.quotecharstr, optional: quote character used by CSV parser.delimiterstr, optional: delimiter characted used by CSV parser.prebuffer_bytesint, optional: number of bytes of input file to prebuffer in attempt to get a total number of lines ahead of time.totalint, optional: total number of lines to expect in file, if you already know it ahead of time. If given, the reader won't prebuffer data even ifprebuffer_byteswas set.multiplexcasanova.Multiplexer, optional: multiplexer to use. Readthisfor more information.reversebool, optional[False]: whether to read the file in reverse (except for the header of course).strip_null_bytes_on_readbool, optional[False]: before python 3.11, thecsvmodule will raise when attempting to read a CSV file containing null bytes. If set toTrue, the reader will strip null bytes on the fly while parsing rows.strip_null_bytes_on_writebool, optional[False]: whether to strip null bytes when writing rows. Note that on python 3.10, there is a bug that prevents acsv.writerwill raise an error when attempting to write a row containing a null byte.writer_dialectcsv.Dialect or str, optional: dialect to use to write CSV.writer_delimiterstr, optional: CSV delimiter for writer.writer_quotecharstr, optional: CSV quoting character for writer.writer_quotingcsv.QUOTE_*, optional: CSV quoting strategy for writer.writer_escapecharstr, optional: CSV escaping character for writer.writer_lineterminatorstr, optional: CSV line terminator for writer.write_headerbool, optional[True]: whether to automatically write
header if required (takes resuming into account).Propertiestotalint, optional: total number of lines in the file, if known through prebuffering or through thetotalkwarg.headerscasanova.Headers, optional, optional: CSV file headers ifno_headers=False.emptybool: whether the given file was empty.fieldnameslist[str], optional: list representing the CSV file headers ifno_headers=False.row_lenint: expected number of items per row.output_headerscasanova.Headers, optional, optional: output CSV headers ifno_headers=False.output_fieldnameslist[str], optional: list representing the output CSV headers ifno_headers=False.added_countint: number of columns added to the output.ResumingAcasanova.enricheris able to resume through aRowCountResumeror aLastCellComparisonResumer.indexed_enricherSometimes, you might want to process multiple input rows concurrently. This can mean that you will emit rows in an arbitrary order, different from the input one.This is fine, of course, but if you still want to be able to resume an aborted process efficiently (using the`IndexedResumer), your output will need specific additions for it to work, namely a column containing the index of an output row in the original input.casanova.indexed_enrichermakes it simpler by providing a tailoredwriterowmethod and iterators always provided the index of a row safely.Note that such resuming is only possible if one row in the input will produce exactly one row in the output.importcasanovawithopen('./people.csv')asf,\open('./enriched-people.csv','w')asof:enricher=casanova.indexed_enricher(f,of,add=['age','hair'])forindex,rowinenricher:enricher.writerow(index,row,['67','blond'])forindex,valueinenricher.cells('name'):...forindex,row,valueinenricher.cells('name',with_rows=True):...ArgumentsEverything fromcasanova.enricherplus:index_columnstr, optional[index]: name of the automatically added index column.ResumingAcasanova.indexed_enricheris able to resume through aIndexedResumer.batch_enricherSometimes, you might want to process a CSV file and paginate API calls per row. This means that each row of your input file should produce multiple new lines, which will be written in batch each time one call from the API returns.Sometimes, the pagination might be quite long (think collecting the Twitter followers of a very popular account), and it would not be a good idea to accumulate all the results for a single row before flushing them to file atomically because if something goes wrong, you will lose a lot of work.But if you still want to be able to resume if process is aborted, you will need to add some things to your output. Namely, a column containing optional "cursor" data to resume your API calls and an "end" symbol indicating we finished the current input row.importcasanovawithopen('./twitter-users.csv')asinput_file,\casanova.BatchResumer('./output.csv')asoutput_file:enricher=casanova.batch_resumer(input_file,output_file)forrowinenricher:forresults,next_cursorinpaginate_api_calls(row):# NOTE: if we reached the end, next_cursor is Noneenricher.writebatch(row,results,next_cursor)ArgumentsEverything fromcasanova.enricherplus:cursor_columnstr, optional[cursor]: name of the cursor column to add.end_symbolstr, optional[end]: unambiguous (from cursor) end symbol to mark end of input row processing.ResumingAcasanova.batch_enricheris able to resume through aBatchResumer.resumersThrough handyResumerclasses,casanovalets its enrichers and writers resume an aborted process.Those classes must be used as a wrapper to open the output file and can assess whether resuming is actually useful or not for you.All resumers act like file handles, can be used as a context manager using thewithkeyword and can be manually closed using theclosemethod if required.Finally know that resumers should work perfectly well withmultiplexingRowCountResumerTheRowCountResumerworks by counting the number of line of the output and skipping that many lines from the input.It can only work in 1-to-1 scenarios where you only emit a single row per input row.It works inO(2n) => O(n)time andO(1)memory,nbeing the number of already processed rows.It is only supported bycasanova.enricher.importcasanovawithopen('input.csv')asinput_file,\casanova.RowCountResumer('output.csv')asresumer:# Want to know if we can resume?resumer.can_resume()# Want to know how many rows were already done?resumer.already_done_count()# Giving the resumer to an enricher as if it was the output fileenricher=casanova.enricher(input_file,resumer)IndexedResumercasanovaexports an indexed resumer that allows row to be processed concurrently and emitted in a different order.In this precise case, couting the rows is not enough and we need to be smarter.One way to proceed is to leverage the index column added by the indexed enricher to compute a set of already processed row while reading the output. Then we can just skip the input rows whose indices are in this set.The issue here is that this consumes up toO(n)memory, which is prohibitive in some use cases.To make sure this still can be done while consuming very little memory,casanovauses an exotic data structure we named a "contiguous range set".This means we can resume operation inO(n + log(h) * n)) => O(log(h) * n)time andO(log(h))memory,nbeing the number of already processed rows andhbeing the size of the largest hole in the sorted indices of those same rows. Note that most of the timeh << nsince the output is mostly sorted (albeit not at a local level).You can read more about this data structure inthisblog post.Note finally this resumer can only work in 1-to-1 scenarios where you only emit a single row per input row.It is supported bycasanova.indexed_enricheronly.importcasanovawithopen('input.csv')asinput_file,\casanova.IndexedResumer('output.csv')asresumer:# Want to know if we can resume?resumer.can_resume()# Want to know how many rows were already done?resumer.already_done_count()# Giving the resumer to an enricher as if it was the output fileenricher=casanova.indexed_enricher(input_file,resumer)# If you want to use casanova ContiguousRangeSet for whatever reasonfromcasanovaimportContiguousRangeSetBatchResumertodo...LastCellResumerSometimes you might write an output CSV file while performing some paginated action. Said action could be aborted and you might want to resume it where you left it.TheLastCellResumertherefore enables you to resume writing a CSV file by reading its output's last row using acasanova.reverse_readerand extracting the value you need to resume in constant time and memory.It is only supported bycasanova.writer.importcasanovawithcasanova.LastCellResumer('output.csv',value_column='user_id')asresumer:# Giving the resumer to a writer as if it was the output filewriter=casanova.writer(resumer)# Extracting last relevant value if any, so we can properly resumelast_value=resumer.get_state()LastCellComparisonResumerIn some scenarios, it is possible to resume the operation of an enricher if you can know what was the last value of some column emitted in the output.Fortunately, usingcasanova.reverse_reader, one can read the last line of a CSV file in constant time.As such theLastCellComparisonResumerenables you to resume the work of an enricher inO(n)time andO(1)memory, withnbeing the number of already done lines that you must quickly skip when repositioning yourself in the input.Note that it only works when the enricher emits a single line per line in the input and when the considered column value is unique across the input file.It is only supported bycasanova.enricher.importcasanovawithopen('input.csv')asinput_file,\casanova.LastCellComparisonResumer('output.csv',value_colum='user_id')asresumer:# Giving the resumer to an enricher as if it was the output fileenricher=casanova.enricher(input_file,resumer)countcasanovaexposes a helper function that one can use to quickly count the number of lines in a CSV file.importcasanovacount=casanova.count('./people.csv')# You can also stop reading the file if you go beyond a number of rowscount=casanova.count('./people.csv',max_rows=100)>>>None# if the file has more than 100 rows>>>34# else the actual count# Any additional kwarg will be passed to the underlying reader as-iscount=casanova.count('./people.csv',delimiter=';')last_cellcasanovaexposes a helper function using areverse_readerto read only the last cell value from a given column of a CSV file.importcasanovalast_cell=casanova.last_cell('./people.csv',column='name')>>>'Quentin'# Will return None if the file is emptylast_cell=casanova.last_cell('./empty.csv',column='name')>>>None# Any additional kwarg will be passed to the underlying reader as-islast_cell=casanova.last_cell('./people.csv',column='name',delimiter=';')set_defaultscasanova.set_defaultslets you edit global defaults forcasanova:importcasanovacasanova.set_defaults(strip_null_bytes_on_read=True)# As a context manager:withcasanova.temporary_defaults(strip_null_bytes_on_read=True):...Argumentsstrip_null_bytes_on_readbool, optional[False]: should readers and enrichers strip null bytes on read?strip_null_bytes_on_writebool, optional[False]: should writers and enrichers strip null bytes on write?prebuffer_bytesint, optional: default prebuffer bytes for readers and enrichers.xsv selection mini DSLxsv, a command line tool written in Rust to handle csv files, uses a clever mini DSL to let users specify column selections.casanovahas a working python implementation of this mini DSL that can be used by theheaders.selectmethod and the enrichersselectkwargs.Here is the gist of it (copied right from xsv documentation itself):Select one column by name:
    * name

Select one column by index (1-based):
    * 2

Select the first and fourth columns:
    * 1,4

Select the first 4 columns (by index and by name):
    * 1-4
    * Header1-Header4

Ignore the first 2 columns (by range and by omission):
    * 3-
    * '!1-2'

Select the third column named 'Foo':
    * 'Foo[2]'

Re-order and duplicate columns arbitrarily:
    * 3-1,Header3-Header1,Header1,Foo[2],Header1

Quote column names that conflict with selector syntax:
    * '"Date - Opening","Date - Actual Closing"'