Hey Aaron,

I will look again more deeply tonight, but wanted to respond quickly to see
if you had comments on the aggregation piece of my issues (last paragraph
below).

Yes, I downloaded the github code and hacked around for inspiration but
didn't actually set up all the filepaths to make the code run.  I should
try that tonight - I did read through all of that code at some point and
think I'm doing the same things they are to read the csv but no one seemed
to note the extra columns issue.

Thanks to the pointer to the csv module instead of pandas.read_csv.  Makes
sense for debugging. I will also look more closely at dtype for
pandas.read_csv.  Digging into the details, I'll try the usecols option too
since I am not using all the variables in my analysis and taking a subset
will help with dtype specification.

Regarding the suitability of the data, I do feel like ranking is still over
my head.  I'm attempting to reduce to a lower dimensional problem and use
various customer and search characteristics to classify book vs. no book
for a given search (rather than developing a complete hotel sort order).
I'm aggregating all of the hotel level information so sorting should not
apply.  I'm not sure what we'll get, but it will be interesting to see
which of these variables mean anything.

More importantly though, is there a simple answer to which function (in
Python or R) that I should be using to aggregate my data?  I can get my
data to 11 columns, 6 of which create an (almost) unique "key" and 5 that
need to be summed, averaged or counted.  I could melt and recast these in
R, but can't piece out how to treat the sum/mean/count variables
differently.  I think I got what I needed with my clunky aggregate/merge
method in R, but I'm still stumped by getting a tidy dataframe in Python.

Thanks,
Holly


On Mon, Apr 28, 2014 at 11:59 AM, Aaron Schumacher <notifications@github.com
> wrote:

> In logistic_assignment/hollyexcuses.rmd:
>
> > +
> > +# Stage 1 - Getting the raw data
> > +
> > +The full 2.4GB training set and variable documentation may be found here: http://www.kaggle.com/c/expedia-personalized-sort/data
> .  To make the dataset manageable, I sliced out the first 100K of 9.9M rows in the shell.
> > +
> > +```
> > +$ wc train.csv
> > +$ touch train100K.csv
> > +$ head -100000 train.csv > train100K.csv
> > +```
> > +
> > +There was no happiness when I tried to read into Python - around line 38000 there start to be some entries with commas instead of periods within some of the decimal values - hence csv readers are unhappy.  I really don't know how to solve this.
> > +
> > +### Q1: How could I use pd.read_csv to read in a few extra columns to catch the overflow variables in the buggy lines with extra commas?  Another path - how do you specify datatypes to expect in each column?
> > +
> > +If I had that, I'd then delete out all lines that included those extra columns (after testing for more systematic issues).  For now, let's just read in 35K lines.
>
> Common way to solve buggy-CSV issues is to use the csv module and check
> each row for useability / fixing if possible.
>
> —
> Reply to this email directly or view it on GitHub<https://github.com/ajschumacher/gadsdc1/pull/128/files#r12053013>
> .
>