Used to generate the main storage tables for the newer set of data.
Using the table format so we can append additional data to the table
at will.

Note that I had to mess the files list up.  2001 - 2003 ran fine but
then eurostat changed the name from 'product_nc' to 'product' and
that messed me up.
CALC: Used to generate the shares and prices. Took about 45 min?
BUG:  Seems like pandas as handling the zero division errors,
replacing with np.inf.  Now I'll check for zero ahead of time.
Perission before forgiveness wins here.
ENH: Cleaning up indecies here.  Removes stat and flow since they're
now redundent and moves years from YYYY52 > YYYY.
BUG**: Call the function eh.  This version was used to generate
the prices (and shares).  I reused shares from an earlier version
but no differences for them.
Another bug.  Obviously should check quantity, not value.
Running for Full.
Failed after '001' so I'm temporaily skipping over that one.
One minor change to incorporate EU as well.
Minor change to include EU.
BUG: Forgot to log.  One line of code... several hours of compute.
May as well do the crossing and squaring here to save on IO.
Some formatting and functionallity improvements.
Fixing up hist2 formatting.
ENH: Allow for outlier filtering in plots.
Various changes to get up to date with the new_data style.

Prices, shares already squared. Cleanup.  Adding a counter.
Starting support for weighting.
Implements weighting for gmm_with_weighting (finally).

Notes: This is slow(!) and doesn't add one term from B&W
(see gen_params).  It just takes care of calculating and using the
weighting matrix.

Two new functions are added to do the calculations: f and gen_weight.

Some changes made to the calling convention to accomodate these
changes.
Reworked weighting matrix calls to do some work outside that function.

This will avoid duplicate calculation of the (1/q) + L(1/q) term.
For ipython paste :()
To make Ray happy.