Package: recordparser
Python recordparserParse delimited text file into a classSynopsisThis module will parse a delimited text file, transforming each record
of text fields into a given class with fields and data types.
Only return records that conform to this description will be returned.For example, given an input file like this:$ cat test_data/items.csv
id_,name,price
1,foo,3.25
2,bar,.43
3,baz,4.01We can create a class calledItemto describe a record with the following
definition:from typing import NamedTuple, TextIO


class Item(NamedTuple):
    id_: int
    name: str
    price: floatTherecordparser.parse()function will return a generator of records that
will be of the typeItem.
Various problems may cause this to fail which are currently raised
with exceptions, so usetry/catch:try:
    parser = recordparser.parse(fh=open('test_data/items.csv'),
                                cls=Item,
                                delimiter=',',
                                quiet=False)

    for rec in parser:
        print(rec)

except Exception as err:
    print(err)This would print the following:Item(id_=1, name='foo', price=3.25)
Item(id_=2, name='bar', price=0.43)
Item(id_=3, name='baz', price=4.01)Input file delimiterThe default field delimiter is a comma, but you can set the "delimiter"
to any valid character.
Thecsvmodule is used to parse the input text files.Ignore undefined fieldsThe source file may name columns which the given class does not describe.
Only the columns from the class will be used, so the extra fields will be
ignored.
Here the "can_discount" field is skipped:$ cat test_data/extra_fields.csv
id_,name,price,can_discount
1,foo,3.25,True
2,bar,.43,False
3,baz,4.01,TrueSo the program would emit the same output:Item(id_=1, name='foo', price=3.25)
Item(id_=2, name='bar', price=0.43)
Item(id_=3, name='baz', price=4.01)Modify column namesOften an input file will have column names you need to map to your class'
fields.
You can provide a "mapping" dictionary of the source name to a field name.
In this file, the columns "id," "item," and "cost" need to be mapped to "id_,"
"name," and "price," respectively:$ cat test_data/other_names.csv
id,item,cost
1,foo,3.25
2,bar,.43
3,baz,4.01Here is the code:mapping = {'id': 'id_', 'item': 'name', 'cost': 'price'}
parser: Iterable[Item] = recordparser.parse(fh=args.file,
                                            cls=Item,
                                            mapping=mapping
                                            delimiter=args.delimiter)The field will be mapped to their correct counterparts in theItemclass:Item(id_=1, name='foo', price=3.25)
Item(id_=2, name='bar', price=0.43)
Item(id_=3, name='baz', price=4.01)Parsing files without column namesIf you have an input file that does not list the columns in the first
line, you can set the "fieldnames" option to pass tocsv.DictReader.
Be sure not to pass this option torenameexisting columns; instead use
the above "mapping" option.$ cat test_data/no_names.csv
1,foo,3.25
2,bar,.43
3,baz,4.01This is the code:parser: Iterable[Item] = recordparser.parse(
    fh=args.file,
    cls=Item,
    fieldnames=['id_', 'name', 'price'])To produce our familiar output:Item(id_=1, name='foo', price=3.25)
Item(id_=2, name='bar', price=0.43)
Item(id_=3, name='baz', price=4.01)Failure on missing fieldsIf the input file is missing field definitions, the module will throw
an exception (which may not be great):$ cat test_data/missing_price.csv
id_,name
1,foo
2,bar
3,bazGiven theItemclass, this file would be rejected:$ ./rc.py test_data/missing_price.csv
Missing field: priceOptional fieldsYou can use theOptionaltype to allow a field to be missing:class Item(NamedTuple):
    id_: int
    name: str
    price: Optional[float]And now the "price" will beNone:$ ./rc.py test_data/missing_price.csv
Item(id_=1, name='foo', price=None)
Item(id_=2, name='bar', price=None)
Item(id_=3, name='baz', price=None)Skipping data that does not matchIf a record cannot be parsed according to the class definition, the
record will be skipped and a message printed to STDERR (unless you
passTruefor the "quiet" option).
It's hard to see in this example (which is the point), but the "price"
of the "bar" item has a capital "O" instead of a "0":$ cat test_data/bad_item.csv
id_,name,price
1,foo,3.25
2,bar,O.43
3,baz,4.01If the "price" is a required (notOptional) field, the record will be skipped:$ ./rc.py test_data/bad_item.csv
Item(id_=1, name='foo', price=3.25)
1: Cannot convert "O.43" to "<class 'float'>"
Item(id_=3, name='baz', price=4.01)If the "price" isOptional, then the record will be present but the value
will be aNone:$ ./rc.py !$
./rc.py test_data/bad_item.csv
Item(id_=1, name='foo', price=3.25)
1: Cannot convert "O.43" to "typing.Union[float, NoneType]"
Item(id_=2, name='bar', price=None)
Item(id_=3, name='baz', price=4.01)Union typesSometime the data in a column may be a mix of values.
In this case, you can use aUniontype to declare the various typein the order you wish to try.
For instance, we first want to try to convert "price" to afloatand then
usestrfor a backup:class Item(NamedTuple):
    id_: int
    name: str
    price: Union[float, str]So that this data:$ cat test_data/mixed_types.csv
id_,name,price
1,foo,3.25
2,bar,NA
3,baz,4.01Will parse like so:Item(id_=1, name='foo', price=3.25)
Item(id_=2, name='bar', price='NA')
Item(id_=3, name='baz', price=4.01)Base class must be NamedTupleYour class must be derived from aNamedTuple.
Future versions may allow for the use of aTypedDict.
This means that the objects/records will beimmutable(which, IMHO,
is a Good Thing).MotivationI spend too much time parsing messy delimited files/Excel spreadsheets.
Often there are some values that need to be coerced to something like number.
If these values cannot be cast, I should skip the data.
So I kept writingforloops withtry/catchto handle exceptions
fromfloat(val)and so forth.With this module, I candescribewhat the resulting data should look like
using aclass.
Therecordparserwill handle missing and optional fields, multiple data
types, and the mapping of the file's column names to the fields in my class.Please let me know how I can improve this code!https://github.com/kyclark/python-recordparser/issuesAuthorKen Youens-Clarkkyclark@gmail.com