Package: organize
# organizeReal world data is messy:1. the [IMF eLibrary Data](http://www.imf.org/external/data.htm) uses the ``.xls`` extension for TSV,2. files on [data.gov](https://www.data.gov/) sometimes include preambles which break straightforward parsing,3. if you've been using public data sources, then you have horror stories of your own.4. (If all your data comes from coworkers or colleagues then undoubtedly it's always perfectly formatted.)``organize`` aims to make it easy to eliminate the hand-scrub phase from working with real-world data files:1. Read CSV, TSV and Excel formats, even if they are poorly labeled or missing a filename.2. Skip over preambles lines which would otherwise require cleaning up by hand.3. Ignore lines with whitespace or where every column is empty.In most cases it should be as simple as:from organize import organizewith open('myfile.csv|myfile.tsv|myfile.xls|myfile', 'r') as fin:for row in organize(fin):for column_name, column_value in row:print column_name, column_valueFor best performance, also supply the filename or mimetype to helpprioritize the most likely parsers:for row in organize(fin, filename='myfile.csv', mimetype='text/csv'):# and so on``organize`` owes a spiritual debt to [BeautifulSoup](http://www.crummy.com/software/BeautifulSoup/),which similarly offered a sane abstraction over broken HTML, saving programmers great swaths of timeand energy.Developed against Python 2.7, and hosted on [Github](https://github.com/lethain/organize).## InstallationSimplest is to install via pip from [PyPi](https://pypi.python.org/pypi?name=organize):pip install organizeAlternative, you can install it from Github if you want an unreleased branch or commit:pip install -e git+https://github.com/lethain/organize#egg=organizeNote that installing from Github will also download the <5MB of test datasetswhich are used when running the test suite. These are not installed when youinstall the package via ``setup.py build install``.For development:git clone git@github.com:lethain/organize.gitcd organizemake envAnd then you can run the tests and stylechecks via:make test style## UsageFor normal usage, ``organize`` presents a very simple interface:from organize import organizewith open('myfile', 'r') as fin:for row in organize(fin):for name, val in row:print "%s: %s" % (name, val)If possible, you should also pass the filename (or mimetype) tothe ``organize.organize`` function in order to prioritize theparsers. Supplying the filename or mimetype will improve performancein the normal case but will not impact correctness:from organize import organizefilename = 'myfile.csv'mimetype = 'text/csv'with open('myfile', 'r') as fin:for row in organize(fin, filename=filename, mimetype=mimetype):for name, val in row:print "%s: %s" % (name, val)In most cases there is no benefit to passing both filename and mimetype,but for cases where you're reading a bunch of files, generally try tosupply as much information as possible.Note that both the row and columns are generators, so you cannotiterate through the returned generator multiple times. If youwanted to process the same file twice, you would need to dosomething like this:from organize import organizewith open('myfile', 'r') as fin:for row in organize(fin):print rowfin.seek(0)for row in organize(fin):print row### Exposes duplicate columns, and unnamed columnsBecause we don't want to accidentally drop data we do notuse a dictionary to store columns. This means by default youwill see all columns even if they are unnamed or have duplicate names.If you do **not** want duplicates, you can convert the rows into dictionariessimply by calling ``dict`` on each row:with open('myfile', 'r') as fin:for row in organize(fin):row_dict = dict(row)print row_dict.keys()If you want uniqueness and ordering, then you can use [collections.OrderedDict](https://docs.python.org/2/library/collections.html#collections.OrderedDict):from collections import OrderedDictwith open('myfile', 'r') as fin:for row in organize(fin):row_dict = OrderedDict(row)print row_dict.keys()In this case you'll get the first value for each name.### Only read certain rowsFor some reason you may only want a subset of rows.For getting a subset of rows we recommend using [itertools.islice](https://docs.python.org/2/library/itertools.html#itertools.islice)from the Python ``itertools`` module:with open('myfile', 'r') as fin:for row in islice(organize(fin), 5, 25):print rowNote that you won't get the literal rows 5 through 25 from the document,but rather rows 5 through 25 from the dataset extracted from the document.I honestly have no idea why you'd actually want to do this. If you do havea usecase, please let us know.## ExamplesThis section briefly describes the contents of the ``examples`` directory.### [transform_directory](organize/examples/transform_directory.py)Walks through slurping up all files in a directory, organizing them,and then writing the cleaned up versions into another directory.### [Using with Pandas](organize/example/with_pandas.py)[Pandas](http://pandas.pydata.org/) is one of the core Pythondata science packages, and a commonly used tool. While it providesmany [tools for loading data](http://pandas.pydata.org/pandas-docs/stable/io.html),it doesn't place as much emphasis on handling poorly formatted data.from organize import organizeimport pandas as pdpd.DataFrame.from_records(organize(open('myfile', 'r')))The example file is a bit better formatted, but expresses the same idea.## What it doesn't do``organize`` tries to parse tabular data files, and any tabular data file it doesnot parse is a bug. However, there are many things it does not do.These are not necessarily the final say, but represent our best current thinking.### Does not read directly from databasesWe do not plan to support reading directly from databases (MySQL, PostgreSQL, SQLite, MongoDB, ...).### Does not generate or enforce schemasWe do not intend to create or enforce schemas on the organized data itself.Data will always be what the underlying format parser returns. For TSV, CSV,Excel this means a string, for more helpful formats like JSON it will be astring or integer or whatnot.We like this functionality, but believe it would be best suited toa different library built on top of ``organize`` as opposed to includingit directly within the library itself.## Contribution and developmentThis section includes some commentary for those interested and willing to contributeadditions to this codebase. First, and most importantly: thank you!Successful pull requests will pass pep8, pylint and tests, as well as including newtests for any additional functionality (or fixed bugs). You can verify they are workingvia:cd ~/path/to/organizemake env test styleIt's OK to disable certain pylint checks within the files you edit if it's not feasibleto resolve the pylint complaint. (For example, when it believes a given attribute doesn'texist which does but it can't lint properly for whatever reason.)### Areas for future developmentWe're using the [Github issue tracker](https://github.com/lethain/organize/issues) totrack specific projects for future development, but broadly there are two areas we'dlike to continue improving upon:1. File parsing should be as robust as possible for existing formats.2. File parsing should support as many formats as useful.Anything along those lines that doesn't introduce significant complexity orperformance degradation will probably be viewed as a very good thing.### ApproachOur implementation approach is:1. The library should do the intended thing, even if it requires a hackor underwhelming heuristic.2. Deal with streams, return streams. Many files are huge and we don't wantto be needlessly wasteful.3. Don't rely on filenames to identify data formats. Files are often mislabled ornot labeled at all.### Implemntation notesThis section includes a variety of implementation notes which may become somewhat relevantto you in some odd edge-cases, but generally are not important for using the library.#### Looking ahead a bitWe need to look ahead a bit at the beginning of files in order to exclude preamblerows within a given dataset. This means that we will read some rows of the loadedfile before you explicitly load them.From a user's point of view, this *should* be transparent as we will replay the rowswe've read in advance as if we're reading them when you read them. However, I would notbe shocked if in some rare and bizarre scenarios this leaks.