BrandonFors/pandas-github-issues huggingface dataset:
BUG: sql_schema does not generate dialect-specific DDL
State: closed

Closes #8697 .

sqlalchemy `CreateTable` needs to be called slightly differently to create dialect-specific SQL.  See http://stackoverflow.com/questions/2128717/sqlalchemy-printing-raw-sql-from-create .


Labels:
name: IO SQL
description: to_sql, read_sql, read_sql_query

Comments:
@artemyk Thanks! This seems indeed the fix.

I wanted to comment that we should maybe explicitely test for a certain dialect that a dialect-specific name is used (eg for postgresql that it used `timestamp without time zone` instead of `datetime`) to be sure it is dialect specific. But I was thinking now: does your test already cover that because the dialect-agnostic "create table" statement would fail to execute on the different dialects?

@jorisvandenbossche  Yes, the `CREATE TABLE` is executed under the different dialects in the tests, and then we test that the created table is the same as the dataframe.  I think this is the best way to check that the right schema is getting generated (as opposed to checking for specific strings, for example).  Before this patch, for example, postgres fails the test.
Added a release note and squashed.

Thanks a lot!
ENH: Allow get_dummies to return SparseDataFrame
State: closed

For dataframes with a large number of unique values, `get_dummies` can use enormous amounts of memory.  This provides a `sparse` flag to `get_dummies` which returns a much more memory-efficient structure.

For example:

```
import pandas as pd
df1 = pd.get_dummies(pd.DataFrame({'a':range(1000)}, dtype='category').a, sparse=False)
print 'dense:', df1.memory_usage().sum()
df2 = pd.get_dummies(pd.DataFrame({'a':range(1000)}, dtype='category').a, sparse=True)
print 'sparse:', df2.memory_usage().sum()
pd.util.testing.assert_frame_equal(df1, df2)
```

returns

```
dense: 8000000
sparse: 8000
```

Performance could probably be improved a lot.

Fails `pandas/tests/test_reshape.py:TestGetDummiesSparse.test_include_na` due to #8822 .


Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Sparse
description: Sparse Data Type

Comments:
@artemyk can you rebase this. and you will need to compare vs Sparse (or not) in the tests.

@jreback Rebased on the #8822 branch

@jreback This should be ready once #8822 is merged.  
Regarding tests for comparing sparse vs. not sparse --- there are a battery of tests that compare `get_dummies` results against 'expected' versions.  Since both sparse and non-sparse versions of `get_dummies` get compared to the same 'expected' dataframes, doesn't this effectively test that they are returning the same things? 

@jreback Ready to merge?

merged via https://github.com/pydata/pandas/commit/467322551c4bd8ad8b7e4e12db00c0d0194c91a7

thanks!
VIS: plot with `use_index=False` shouldn't plot the index name
State: closed

Fixes #8558. As far as I can tell the two plot types where this applies are line and bar plots, if there any others they should also be easy to fix. Bar and line plots no longer include the index name:

``` python
import pandas as pd
from pandas import Series
import matplotlib.pyplot as plt

s = Series([1, 2, 3], index=['a', 'b', 'c'])
s.index.name = 'The Index'

ax = s.plot(use_index=False)
plt.show()
```

￼
![image](https://cloud.githubusercontent.com/assets/1460294/5041001/6095c37e-6c0c-11e4-86ca-8e1c4ec83b3c.png)

``` python
ax2 = s.plot(kind='bar', use_index=False)
plt.show()
```

![image](https://cloud.githubusercontent.com/assets/1460294/5041003/657a45d6-6c0c-11e4-92a2-642e5df917a0.png)


Labels:
name: Visualization
description: plotting

Comments:
@TomAugspurger ?

Great, thanks @onesandzeroes . FYI there's also AreaPlot but your fix catches it through LinePlot.

Mind rebasing, and I'll merge?

@TomAugspurger I rebased but got a build error (looks like Travis was having connection issues), not sure how to force a recheck.

nvm, had to fix up something anyway so it's rebuilding now.

Thanks. I think there's a button in the Travis GUI to force a rebuild FYI.
DOC: Added get_group() example to grouby.html
State: closed

Added a get_group example in the Iterating through groups section of grouby.html to demonstrate selecting
single group from a DataFrameGrouBy object

Fixed typo in README


Labels:
name: Docs
name: Groupby

Comments:
Looking good. Can you just squash the commits into one?

can u make this into a sub-section as well 

rebase again. We had an unrelated failure.

Thanks a lot!
GH3508 added basic documentation of google analytics
State: closed

closes #3508 


Labels:
name: Docs

Comments:
Thanks! This was so much needed.

I would add the `filters` parameter as well on the code block.

@manugarri filters added.

Would it be better suited to put this in remote_data.rst instead of io.rst?

hmm, I thought I mentioned that previously.....

@benoitpointet this should go in `remote_data.rst`

Yep indeed, that's what you said. However GBQ is in `io.rst`. What's the rule there? 

gbq allows full IO

ga is just a remote data pull

so remote_data is more appropriate

closing in favor of #8835. FYI for the future, you can simply `push -f` to overwrite these commits rather than opening another pr, but ok for this one.
[ENH] Add ability to enforce specific filter on Google Analytics query
State: closed

Fix for #3505

This PR allows to directly write filters to ga.py `read_ga`  function. 

Currently only **OR** filters are supported. with this PR we can support directly writing the filters as a string, allowing for example **AND** filtering.

**Example here**

![example](http://i.imgur.com/28FojWh.png)


Labels:

Comments:
It seems that you can already do this, by simply specifying the `filters` as a compound string, no?

have multiple keywords for this does not seem like a good idea.

Further, this needs tests (well the entire module needs some tests, against a mock GA).

So currently you can only add filters that get added with a comma separating them (see https://github.com/pydata/pandas/blob/master/pandas/io/ga.py#L386).

comma separated filters are interpreted as being added with an **OR**. The Google analytics way of adding an **AND** to the filters is by adding the filter with a semicolon separating them.

My implementation doesn't affect the current API (afaik), and allows anyone with more specific filtering needs to write a custom filter.

and is their anything stopping you from simply passing the entire correct string as the `filter` argument? 

ok, so just noticed that my implementation was a totla overkill.

you can just send the raw_filter as `filter = ['filter1;ga:filter2,ga:filter3...]`

Will close
TST: Add test for get_data_google dtypes
State: closed

closes #3995 


Labels:
name: Bug
name: Testing
description: pandas testing functions or related to the test suite
name: Dtype Conversions
description: Unexpected or buggy dtype conversions

Comments:
@jreback  I couldn't determine when it got fixed.  I'm guessing it was a change in read_csv that inferred data types.

np.appreciate the testing....ping on green

@jreback Green ping.

thanks!
Fix unrecognized 'Z' UTC designator
State: closed

closes [#8771](https://github.com/pydata/pandas/issues/8771)


Labels:
name: Datetime
description: Datetime data dtype
name: Regression
description: Functionality that used to work in a prior pandas version

Comments:
ok, looks simple enough! cross-fingers other tests pass!
pls add a release note as well (referencing the original issue) (put in bug-fixes, though you can say regression)

Hi, do you need me to do other modifications before merging? Let me know

looks good. pls rebase and squash. ping when green.

done!

merged via https://github.com/pydata/pandas/commit/c03e92f4e3c1ab978b3111556019a502ae55f1e4

@broessli hmm, I think this was changed before the latest travis build. Thought it was green.

Can you see what the problem is and and do a follow up PR? thanks (pls submit a new PR, can't push to this one anymore)

https://travis-ci.org/pydata/pandas/builds/41039139

Yes, sorry about that. The issue seems related to `dateutil` behavior, which is the fallback when parsing dates with `...Z00`, depending on the version we can get a time zone equal to tzutc() or tzlocal(). Not sure how to handle this (besides dropping the test on `Z00`...)

By the way, checking ISO 8601 I am not sure what to expect exactly with `Z00`, is it really legit?

so it give one answer on dateutil 1.5 and a different on 2.1/2.2?

you can do the test like this:

```
from distutils.version import LooseVersion

.....


if LooseVersion(dateutil.__version__) >= '2.2':
    #.....
else:
    # ....
```

I am not sure of the defined behavior of `...Z00`

cc @rockg ?

Actually, testing on my machine, I always get `tzutc()` regardless of `dateutil`'s version, cannot figure out why this is different on Travis 

hmm, I think this might have to do with the cache not updating because only the .c files were changed (and not the cython files). let me see.

reverted for now: https://github.com/pydata/pandas/commit/6f48d4a18cf92ffaceff3f6c53766004895194c9

got ahead and rebase on master and put up a new PR (close this one)

cc @dbew  

any thoughts on the above? 

https://travis-ci.org/pydata/pandas/builds/41039139

Nothing jumps out, I'm afraid.

There were definitely differences between python 3 and python 2 when I did the original dateutil work but the issues I had were all around UTC and the implementation of the timezones, not tzlocal and parsing. 

Looking at the parse code in dateutil, I think it reads the timezone name and first tests if it matches the local timezone (`tzname in time.tzname`) then tests if it matches UTC - so if 'Z' matches the local timezone then it wouldn't be reported at UTC.
TST: Add additional RemoteDataError check to io.data.Options
State: closed

Per comment in #8761 


Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
thanks!

any thoughts in this one: https://travis-ci.org/jreback/pandas/jobs/40731908

Strange.  Looks like bad data from yahoo - apparently they added an extra day between January 15th and January 20th, 2012. Appears to be resolved now - I can't replicated. All the Travis failures occurred in the same hour..

weird - ok thanks
Do not convert series when looking at sub-daily frequencies.
State: closed

Fixes #8772 by not converting values from UTC to local timezone when trying to infer sub-daily frequencies.  Across a DST transition this will result in a non-unique deltas and an inferred frequency of None.


Labels:
name: Frequency
description: DateOffsets
name: Timezones
description: Timezone data dtype

Comments:
pls add a release note, minor corrections, otherwise looks good

cc @ischwabacher  you happy with this? any other tests needed?

Otherwise LGTM.

I am going to add more tests over both DST transitions (fall and spring) and then some more granular frequencies.  This is why we are in the trouble we are in for all the DST issues...the tests don't cover the transitions.

I can add that new one as well.  Will be later this evening.  Thanks.

Yeah, I've started using ranges including 2013-11-03 or 2014-03-09 as interactive test cases whenever I start to mock up some new feature involving times in ipython.  Which occasionally leads me to think "oh, I can't do that" when the problem is actually "oh, that doesn't work over a DST boundary".

Also I really like the approach taken in `test_nanops.py`, where a large number of arrays are constructed and then passed to each of the different tests.  If we have a menagerie of weird time zone corner cases easily available, it'll make testing so much easier.

@jreback, should be good now.

thanks!
Allowing empty slice in multi-indices
State: closed

This solves #8737 in the case where there is at least one boolean indexer. The patch of  #8739 does not allow the following:

``` python
import pandas as pd
import numpy as np
multi_index = pd.MultiIndex.from_product((['foo', 'bar', 'baz'], ['alpha', 'beta']))
df = pd.DataFrame(np.random.randn(5, 6), index=range(5), columns=multi_index)
df = df.sortlevel(0, axis=1)
df.loc[:, (['foo'], [])]
```


Labels:
name: Bug
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: MultiIndex

Comments:
The Travis failure seems unrelated to this change. 

pls add your example as a test (and put it after the 8737 test)

Done. 

don't worry about that failing test - sep issue

also pls add a release note in v0.15.2 (u can ref this issue)

here it is

can you sqush the commits? thanks.

Rebased. 

thank you for the fix!
ENH: Add categorical support for Stata export
State: closed

Add support for exporting DataFrames containing categorical data.

closes #8633
xref #7621 


Labels:
name: Categorical
description: Categorical Data Type
name: IO Stata
description: read_stata, to_stata

Comments:
Some questions about categoricals:
- Are the underlying data types always ints?
- Can they every be `int64`?

Could also probably use some feedback to simplify the Python2/3 bytes/strings code.

A few small things left to do:
- [x] Test that errors are correctly raised
- [x] Add notes
- [x] Add something to docs

`categories` usually strings, but can be ints/datetimes/floats, really anything.
`codes` are always ints, can be a size from 8-64 bits (a function of the number of categories)
In theory can be `in64` but for a practial manner you can hold `np.iinfo(np.int32).max` in an `int32` so will never happen.

@jreback Should be pretty much ready.  

In the end I decided to stringify categoricals and provide a warning to check the Stata file.

@bashtage otherwise looks good. ping when pushed and green.

@jreback ready

minor comment. ping when green.

@jreback Should be ready

@jorisvandenbossche  Good idea - I had only been checking that the files are correct in Stata, which is probably more important but cannot be automated. This showed there is a bug in the reader code that doesn't correctly handle missing values.

In general writing and reading it back in isn't that useful since both the writer and reader can agree but still be incorrect (this happened in the past).

So @jreback hold off on this one for a while until I can fix the reader.

ping when ready 

@jreback ready

thanks! this is excellent!
BUG: fontsize parameter of plot only affects one axis.
State: closed

fixes #8765 

``` python
import pandas as pd
df = pd.DataFrame(np.random.randn(10, 9), index=range(10))
df.plot(fontsize=7)
```

![fig_fonts_all_same_size](https://cloud.githubusercontent.com/assets/5957850/4969745/86e42902-686b-11e4-959c-e22b544e63e2.png)


Labels:
name: Bug
name: Visualization
description: plotting

Comments:
@TomAugspurger have a look at this as well

Agreed that this should be the default.

@dstephens99 Could you rebase? And while you're rebasing, could you change the docstring for fontsize to make it clear that the parameter affects x and y ticklabels?

@TomAugspurger  Done.  Also specified that rotation only happens to the x ticks on a vertical plot and the y ticks on a horizontal plot.

@dstephens99 can you rebase

@TomAugspurger @jorisvandenbossche ?

Apparently someone else's pull request was merged to fix this (#8940).

ahh I c; I think was done at the london hackathon ....

next time :)

@dstephens99 Ah, sorry, didn't see your PR. 
However, if you still want to rebase it, that would be valuable. The actual fix is the same, but you also fixed some docstrings and tests that were not in the other PR that still can go in (only the whatsnew entry you will have to remove, the rest should be ok after a rebase).

@dstephens99 can ypu rebase / update according to @jorisvandenbossche comment. thanks

@jreback Green and good to go on this one.

@dstephens99 Thanks!
BUG: Fix plots showing multiple sets of  axis labels if index is a timeseries.
State: closed

Fixes #3964


Labels:
name: Bug
name: Visualization
description: plotting

Comments:
can you post the updated plots that result?

Here it is:

![figure_1](https://cloud.githubusercontent.com/assets/5957850/4969615/1e944112-6865-11e4-9808-dc0e1e5f3669.png)

cc @michaelaye you raised this issue (a while ago!) 

can you have a look - see if this is the expectation

@TomAugspurger 

This is another issue, but it seems as if the font size only affects the x axis.. (Confirmed it also happens on master).

@dstephens99 ok, pls create a separate issue for that (as fix will be independent).

Here's the code that generated the plot.  Made it a bit easier to see with a bigger font and sharing the yaxis as well.

``` python
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt

fig, axes = plt.subplots(3, 3, sharex=True, sharey=True)
plt.subplots_adjust(left=0.05, right=0.95, hspace=0.1, wspace=0.1)
df = pd.DataFrame(np.random.randn(10, 9), index=pd.date_range(start='2014-07-01', freq='M', periods=10))
for i, ax in enumerate(axes.ravel()):
    df[i].plot(ax=ax, fontsize=8)
```

![figure_2](https://cloud.githubusercontent.com/assets/5957850/4969636/060daf42-6866-11e4-85d0-45ed1f898913.png)

Checked out this branch. Looks good to me, thanks!

I'll be able to check this out in a week or so. 

> On Nov 10, 2014, at 13:59, K.-Michael Aye notifications@github.com wrote:
> 
> Checked out this branch. Looks good to me, thanks!
> 
> —
> Reply to this email directly or view it on GitHub.

@dstephens99 can you rebase

@TomAugspurger ?

Just the one note about x vs y. Then lets merge it. May need one more rebase, but I promise I won't let it sit around waiting to be merged this time!

@TomAugspurger ready to go?

can rebase on the merge if needed

Yeah, I think so. I'll do one more once-over and then rebase on merge.

Merged via https://github.com/pydata/pandas/commit/7d13fdd8a1ad67113f4255e5b715f48faae827d0

Thanks @dstephens99!
TST: Raise remote data error if no expiry dates are found (Options)
State: closed

Fixes #8761.


Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
I was able to replicate that issue if I set the expiry dates from yahoo to an empty list.   This raises RemoteDataError in that case, so skips tests.

Does this need a mention in what's new?

yes, give a mention in bug fixes with the issue.

lmk when you repush

@jreback  Added comment in bug fixes.  Green and good to go.

beautiful! thanks
BUG: Fix groupby methods to include *args and **kwds if applicable.
State: closed

Fixes #8733 (and any other bug that would result from trying to send an arbitrary keyword to a groupby method).


Labels:
name: Bug
name: Groupby

Comments:
pls add a release note v0.15.2 as well

Added a release note and tests for all the methods that use **kwargs.   I couldn't find any that use *args for testing.  

perfect thanks!
API: allow negative steps for label-based indexing
State: closed

This should fix #8716.

Some of this refactoring may be useful for #8613, so I'd like someone to look through this.

cc'ing @shoyer, @jreback and @jorisvandenbossche .

TODO:
- add tests for label slicing with negative step (ix, loc)
  - [x] int64index
  - [x] float64index
  - [x] objectindex
- [x] fix datetime/period indices
-  add tests for label slicing with negative step (datetime, string, native pandas scalar) x (ix, loc)
  - [x] datetimeindex
  - [x] periodindex
  - [x] timedeltaindex
- [x] clean up old partial string code?
- [x] benchmark


Labels:
name: Bug
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves

Comments:
generally seems like a nice refactor to me :+1: 

This PR will consolidate `Index.slice_locs` with that of Datetime-/Period-/TimedeltaIndex, which means that the non-present lookups on `is_monotonically_decreasing` will start working on those classes.

Mostly done here, except for
- maybe old partial string slice code can be cleaned up
- per-level slices on MultiIndex don't work and I don't yet see a clear way to support steps for it
- have to check performance

Again, this looks very nice. Thanks!

@immerrr maybe you could have a look at https://github.com/pydata/pandas/blob/master/doc/source/internals.rst to see if this is still up to date after your PR (although I am not fully sure this is the best way to document this complex indexing part of the codebase)

going to have a look tomorrow.

I've been doing some benchmarking and found the strangest thing: I couldn't get a stable result from `panel_methods.panel_pct_change_major` benchmark, it reported anything from 2000ms to 8000ms. It appears that the task is memory intensive (easily several gbs of ram) and leaves a lot of objects in its wake because timeit disables garbage collection during the benchmarks. The problem is that it doesn't do any cleanup afterwards and this slows down subsequent runs because malloc has harder time finding chunks of memory when requested.

Given usually there's no such thing as "run code before each iteration, but don't benchmark it" in benchmark libraries, I wonder what's the proper way to fix this... Maybe we should just reduce panel sizes in panel benchmarks.

@immerrr yep that vbench is odd, never really had time to look, feel free to submit a pr to reduce it in size.

@immerrr looks good. minor case that needs testing. Which might be an open issue in any event (e.g. supporting < second, e.g. microsecond reso in partial_strings for DatetimeIndex).

I devoted some time to this at the weekend:
- enabled second/microsecond-resolution partial based indexing.
- partial string lookups are going to stay for now, they're used in `get_loc` in seemingly a bit inconsistent way and fixing it may alter the API, so the cleanup will be a separate PR.

ready & green

thanks!

keep-em coming!

Thank you @immerrr !
BUG: Fix io.data.Options quote time for DST.
State: closed

Updated the second test html so that it uses a DST sample.

Changed failure to get quote time from raising RemoteDataError to setting to np.nan

@jorisvandenbossche This fixes the doc error you noted on #8631 


Labels:
name: Docs

Comments:
The doc errors are indeed fixed!

Is it possible to include a test for what you fixed?

@dstephens99 also add this issue number to the release notes (just put it after the original issue #8631) 

@jorisvandenbossche 
There is a test - There are two sample pages in the tests/data directory.  Part of this commit was changing one of them it was an EST version.  I updated the comment on the test to clarify.

@jreback  I added the issue number to the release notes.

@dstephens99 ah, I hadn't noticed the change in the html file, ok then!

thanks!
BUG: Bug in slicing a multi-index level with an empty-list (GH8737)
State: closed

closes #8737 

```
In [5]: multi_index = pd.MultiIndex.from_product((['foo', 'bar', 'baz'], ['alpha', 'beta']))

In [6]: df = DataFrame(np.random.randn(5, 6), index=range(5), columns=multi_index).sortlevel(0,axis=1) 

In [7]: df
Out[7]: 
        bar                 baz                 foo          
      alpha      beta     alpha      beta     alpha      beta
0 -0.725086 -0.715498  1.141087 -0.586018  0.906581 -0.260218
1 -1.111794 -0.393149  2.101567  0.648850  0.366892  0.554206
2  1.258239  1.472659 -0.174970 -0.355639 -0.843127  1.101716
3 -0.775790  0.419361 -0.521444  1.245519  0.759791  0.183877
4 -0.001001 -1.526741 -1.721501  0.527531  0.407099 -0.415245

In [8]: df.loc[:, ([], slice(None))] 
Out[8]: 
Empty DataFrame
Columns: []
Index: [0, 1, 2, 3, 4]

In [9]: df.loc[:, ([], slice(None))].columns
Out[9]: 
MultiIndex(levels=[[u'bar', u'baz', u'foo'], [u'alpha', u'beta']],
           labels=[[], []])
```


Labels:
name: Bug
name: MultiIndex

Comments:
merged via https://github.com/pydata/pandas/commit/1887ba5e0d3c75d784633caf7f2fac54ede2eacd
ENH: read_csv dialect parameter can take a string (GH8703)
State: closed

Closes #8703

I wasn't sure where to put the test, hopefully this is right.

Follow-up of #8718


Labels:
name: Bug
name: Docs
name: IO Excel
description: read_excel, to_excel
name: Compat
description: pandas objects compatability with Numpy or Python functions

Comments:
Looks good. Pls add a release note in v0.15.1 bug fix section referencing the original issue. Ping when pushed and green

@thrasibule pls add a release note

thanks! this was gr8!
merged via https://github.com/pydata/pandas/commit/0f72d3d6e7e60a8c567bb723f6ea9ad430765845