Package: penny
Penny=====Inspect your data. Find the truth.~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~.. figure:: http://www.martianwatches.com/wp-content/uploads/2013/10/InspectorGadget.jpg:alt: alt tagalt tagUncle Gadget was great and all, but when it came to real detective work,we all know Penny did the heavy lifting. Hence, Penny, the Python modulethat inspects stuff. Feed it rows or columns from a dataset, and getinformation about the column types -- including whether or not a givencolumn represents a category or date. Penny also finds column headers(waaaay more reliably than the ``Sniffer`` class in to the standard``csv`` module).Why?~~~~If you're working with a few datasets, it's easy to figure out whichcolumns are supposed to be dates, integers and even categories just bylooking at the raw csv files. But if you need to programmatically dealwith lots of datasets, this gets tedious fast.Setup~~~~~Grab the package.::pip install pennyOr grab the code from GitHub.::git clone https://github.com/gati/pennycd pennypip install -r requirements.txtGetting Started~~~~~~~~~~~~~~~Guess the headers of a csv file... code:: pythonfrom penny.headers import get_headerswith open('your-awesome-file.csv') as csvfile:has_header, headers = get_headers(csvfile)# Prints True/False depending on whether or not headers were foundprint has_header# Prints column headers or placeholders if real headers weren't foundprint headers # ['Example Header A', 'Example Header B']Guess the data type of a column in your dataset... code:: pythonfrom penny.inspectors import column_types_probabilitiesfileobj = open('your-awesome-file.csv')rows = list(csv.reader(fileobj))# Get the values from column 0column_0 = [x[0] for x in rows]probs = column_types_probabilities(column_0)# Prints something like {'date': 1, 'int': .75, 'category': 0 ...}print probsOr get type guesses for all the rows in your dataset at once... code:: pythonfrom penny.inspectors import rows_types_probabilitiesfileobj = open('your-awesome-file.csv')rows = list(csv.reader(fileobj))probs = rows_types_probabilities(rows)Penny checks for a lot of data "types," not just the standard ``int``,``str``, etc. Here's the list (for now):-  **date** something ``dateutil.parser`` can parse into a ``datetime``object-  **int** a whole number-  **bool** y/n or yes/no or something true/falsey-  **float** a number with a decimal-  **category** something you might want to group records by-  **text** string longer than 90 characters (something you could getnames/places/sentiment/etc from)-  **id** unique for each row-  **coord** a float that might be a latitude or longitude-  **coord\_pair** string that looks like "coord,coord"-  **proportion** column where all values sum to 1 or 100-  **street** house number + street name-  **city** one of the world's 80,000 largest cities-  **region** smaller than a country, bigger than a city. state,province, etc-  **country** a country name on the `ISO 3166list <http://en.wikipedia.org/wiki/ISO_3166-1#Current_codes>`__-  **phone** a phone number-  **email** an email address-  **url** web address with or without http:// (so http://google.com orgoogle.com)-  **address** a full address you could geocode with a service likeGoogle MapsLast but not least, you can also inspect a column for a single type... code:: pythonfrom penny.list_check import column_probability_for_typefileobj = open('your-awesome-file.csv')rows = list(csv.reader(fileobj))# Get the values from column 0column_0 = [x[0] for x in rows]prob = column_probability_for_type(column_0, 'date')# Prints something like 0.78print probContributing & Credits~~~~~~~~~~~~~~~~~~~~~~This is a work in progress, so pull request at will. Some of this workwas inspired by `messytables <https://github.com/okfn/messytables>`__,which looks great for xls files but wasn't quite what I needed. Thanksto `Chris Albon <http://twitter.com/chrisalbon>`__ for putting togethera `repo of useful testdatasets <https://github.com/chrisalbon/Variable-Type-Identification-Test-Datasets>`__.Questions, concerns, devoted fan mail to[@jonathonmorgan](http://twitter.com/jonathonmorgan) on Twitter.