BrandonFors/pandas-github-issues huggingface dataset:
BUG: GH9428 promote string dtype to object dtype for empty DataFrame
State: closed

Addresses [GH9428](https://github.com/pydata/pandas/issues/9428).

When constructing an empty DataFrame with a string dtype (e.g. `str`, `np.unicode_`, `U5`), the dtype is now promoted to the `object` dtype. This is now consistent with Series and avoids the confusing behaviour described in the original issue.


Labels:
name: Bug
name: Dtype Conversions
description: Unexpected or buggy dtype conversions

Comments:
can u add some tests 

pls add a test for this.

No problem - I can add the test to `test_frame.py` as soon as I get off work later today.

I've added the test, but please do let me know if anything about the PR looks like it should be improved or re-done.

ok, looks good. pls rebase on master and squash

ping when green.

@jreback: it's green, but I now realise I may not have squashed the commits properly.

Would it be best to rebase/squash again locally and then push again? (Also do I need to squash _all_ commits together or just my own commits?)

need small change

then squash locally and make sure to push with -f

@jreback: green again... hopefully things are in order this time (I can push again if not!)

thanks @ajcr !

keep em coming!

Thank you! I'll see what else I can do...
BUG: #10652 google-api-python-client minimum version check
State: closed

Fixes [bug](https://github.com/pydata/pandas/issues/10652) with version check for google-api-python-client


Labels:
name: Bug

Comments:
this is not tested on travis, not really sure how to evaluate this.

cc @jacobschaer
cc @sean-schaefer

What do you mean? it seems the [tests passed](https://travis-ci.org/pydata/pandas/builds/72177378) . Am I missing something?

and they pass on current master (w/o this change). So you will have to show an environment that fails first.

to replicate:

```
$ pip install google-api-python-client==1.2.0
```

and then:

```
>>> from pandas.io import gbq
>>> gbq.read_gbq('foo', 'spam')
....
ImportError: pandas requires google-api-python-client >= 1.2.0 for Google BigQuery support, current version 1.2
```

gr8. so modify one of the ci environments to load that version. have it fail and then use your change to have it pass. It is run in `ci/requirements-2.7.txt` now.

I remember this issue... seems a shame to have to resort to using regular expression matching. @parthea mentioned using `StrictVersion` instead of `LooseVersion` might resolve the issue - I haven't really looked into the differences, but his example in the discussion for: https://github.com/pydata/pandas/pull/10857 seems to suggest it will work.

replaced by #10857
BUG: to_dense now preserves dtype in SparseArray
State: closed

Also fixes values and get_values.

fixes #10648


Labels:
name: Bug
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: Sparse
description: Sparse Data Type

Comments:
@jreback sorry for being slow in getting back to this, I'll make the changes requested before the week is out.

can you update

@ebolyen can you update according to comments. need a whatsnew note as well.

can you update according to comments

closing, but pls reopen if you'd like to work on this again.
ENH: allow gzip de-compression for files specified by a url
State: closed

Address #8685 .  The file in the test `test_url_gz` currently points to my fork to demonstrate that the tests pass, but should be changed to point to the real pydata master after merging.


Labels:
name: Enhancement
name: IO CSV
description: read_csv, to_csv

Comments:
This works for `compression="infer"` now as well by looking at the HTTP `Content-Encoding` header like @jseabold suggested.  The test `test_url_gz_infer` uses a file hosted on an s3 bucket that I own where I set it to return `Content-Encoding=gzip`.  Assuming this gets merged, we'd want to move that into the s3 bucket that pandas uses elsewhere in the tests.

Any admins have time to look at this PR?

at the end pls squash to a single commit. pls add a release note in whatsnew/0.17.0.

Here are the contribution guidelines for your [reference](http://pandas.pydata.org/pandas-docs/stable/contributing.html)

@jreback Once the tests all pass, I'll squash into a single commit.  There's still the issue of changing the test url's to point to the official github repo and s3 bucket.  Not sure the best way to handle that.

@jreback Okay.  This should be good to go.  Thanks for the comments!

I pushed the .gz file, so you can use this as a stable address: https://github.com/pydata/pandas/blob/master/pandas/io/tests/data/salary.table.gz

pls change, squash, and ping when green

@jreback Thanks!  What about the s3 file?  Are you able to post that too and set s3 to return the Content-Encoding gzip header?

so is a that a public bucket you setup? any problem leaving it for now?

Yep.  It's public.  I've got no problem leaving it.  Just didn't know if you guys wanted to have it look more "official."  I'll make that other change to the github link.

we don't have an official s3 bucket for this ATM. But I think I can get access to one where we can do stuff like this. 

Ah.  Okay.  I saw `s3://nyqpug/` in the tests and assumed that was official.  Happy to leave mine in there for awhile until you figure it out.

hmm, that is some old stuff IIRC. Wll try to set this up for later today.

ok, going to merge this then fix it with a new bucket. thanks!

Oh shoot.  I didn't get a chance to change the regular .gz github link.

no worries, going to merge this shortly: https://github.com/jreback/pandas/commit/2f0c344191be435b0302ad8e8af4ca3183f2fcf5

This is a public bucket for pandas (though read-only)
BUG: (GH10408, GH10412) in vectorised setting of timestamp columns
State: closed

closes #10408  
closes #10412 

Fix setting values with python datetime.date and numpy datetime64.


Labels:
name: Bug
name: Datetime
description: Datetime data dtype
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves

Comments:
done. and travis is green already. 
thanks for quick replies on PRs :)

thank you sir!
BUG: Fixed typo-related bug to resolve #9266
State: closed

Resubmitting pull request #10576 to resolve issue #9266 .

@jreback — This should be good to merge.


Labels:
name: Bug
name: IO CSV
description: read_csv, to_csv

Comments:
merged via https://github.com/pydata/pandas/commit/384eb45587172b67c5de358b3e3c05cb0f61cdbd

thanks!

Indeed, thanks!
ENH: allow duplicate column names if they are not merged upon
State: closed

Currently merge columns are not automatically inferred when there are duplicate column names. Instead a `MergeError` is raised. This pull request enables merging with duplicate column names as long as they are not merged upon.

```
df = DataFrame({'key': [1, 2, 3],
                'v1': [4, 5, 6],
                'v2': [7, 8, 9]})
df.columns = ['key', 'foo', 'foo']
df2 = DataFrame({'key': [1, 2, 3],
                'v1': [4, 5, 6],
                'v2': [7, 8, 9]})
df2.columns = ['key', 'bar', 'bar']

# worked before
merge(df, df2, on=['key'])
# now works as well since 'foo' and 'bar' don't interfere
merge(df, df2)
```

If the common cols are not unique a `MergeError` is still raised as in #2649


Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: API Design

Comments:
pls add a release note.

@jreback I added the note. I hope I put it in the right spot

merged via https://github.com/pydata/pandas/commit/7559522f4c599874cd05718acee452afe87e53c4

thanks
BUG: made behavior of operator equal for CategoricalIndex consistent,…
State: closed

… improved unit tests

this is a follow-up to https://github.com/pydata/pandas/pull/9947

@jreback 


Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Error Reporting
description: Incorrect or improved errors from pandas

Comments:
pls add this issue number (the PR) to the same place as in #9947 (just list it in the whatsnew after)

sure, just updated the whatsnew 

couple of doc fixes

thanks, updated 

@jreback it's green now, please take a look

ty!
Remove Categorical.name
State: closed

Probably work in progress... Up to now only tests locally with 'not slow' (whoever wrote  `nosetests --with-id --failed`  should be praised! :-) )

See here: https://github.com/pydata/pandas/issues/10482


Labels:
name: Categorical
description: Categorical Data Type
name: Deprecate
description: Functionality to remove in pandas

Comments:
Hey, a first: `+50 −84` :-)

Thanks. Added it to deprecation list, and I'll close the original issue.

lgtm. couple of minor import changes. Pls add a release note.

I also added the stacklevel arg to the other args. Hopefully getting it right...

Can you add a small whatsnew notice for this?

done

can u add a test that asserts the deprecation in the name

@jreback added a test for the warnings

thanks!
TST: windows compat for testing / msgpack
State: closed

- skipping a couple of parsing routines that cause odd test failures on windows
- msgpack compat for windows (not tested before, prob not working correctly). Slight internal API change where writing the `dtype.name` rather than `dtype.num`, but back-compat is ok on this. (and forward compat is better).
- skipping a buggy test for ujson on windows on 2.7 only


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: CI
description: Continuous Integration
name: Windows
description: Windows OS
name: Compat
description: pandas objects compatability with Numpy or Python functions

Comments:
cc @kawochen 

turns out the windows msgpacks were incorrectly being read. thanks for the creation scripts!
DOC Add more examples to Series.Resample docstring
State: closed

Partial fix for https://github.com/pydata/pandas/issues/5023
flake8 doesn't do a very good job at checking code in the docstrings for PEP8 compliance, so I'm just eyeballing it at the moment and my PEP8 knowledge leaves something to be desired. Apologies if I missed something really obvious.


Labels:
name: Docs

Comments:
Will squash everything into one commit once I add an example of using a custom `how` method.

@jorisvandenbossche @sinhrks 
this one is done and good to go from my side!
comments/corrections welcome, of course!

Nice examples!

Looks nice! No further comments from me. 

@Winterflower Do you have some time to update? (it's only a very small comment)

@jorisvandenbossche apologies for the delay! I will add the how='right' example tonight after work.

Still need to fix line lengths.

At the moment, the doctest for the string is failing because of the `custom_resampler` function defined in the docstring

```
Trying:
    def custom_resampler(array_like):
Expecting:
            return np.sum(array_like)+5
**********************************************************************
File "pandas/core/generic.py", line 3367, in generic.NDFrame.resample
Failed example:
    def custom_resampler(array_like):
Exception raised:
    Traceback (most recent call last):
      File "/home/winterflower/anaconda/lib/python2.7/doctest.py", line 1315, in __run
        compileflags, 1) in test.globs
      File "<doctest generic.NDFrame.resample[9]>", line 1
        def custom_resampler(array_like):
                                        ^
    SyntaxError: unexpected EOF while parsing
```

The solution, based on [this](http://stackoverflow.com/questions/13510698/python-how-can-i-define-a-class-in-a-doctest) is to include the ellipsis in the docstring. Will add that to make the doctests pass.

```
def custom_resampler(array_like):
...     return np.sum(array_like)+5
```

This is good to go on my part! 

@Winterflower gr8 thanks!

Further I think portions of your narrative my be a nice addition to the actual resample docs; add where appropriate
Very minor refactoring and edits to code comments
State: closed

Just some miscellaneous improvements on things that tripped me up a bit.


Labels:
name: Docs
name: Clean

Comments:
@Garrett-R The changes to the docs and conf.py seem good to me.

For the clean-up in the code, I will let someone else review, as I am not familiar with that part of the code.

By the way, to make our (easier to review) and your (easier to update, quicker merged) life easier, it is often times a good approach to do separate PRs for clearly separate things (here it is not yet a big problem, as it are not that much of lines that are changed, but just giving the tip for possible future contributions)

@jorisvandenbossche, right, good call.  I'll do that next time.

merged via https://github.com/pydata/pandas/commit/a7437430b5cb62e49a79b64d18eccfb2b4d6367f

thanks!
Fix bug in outer_indexer where the special case of an empty right array resulted in bogus return data.
State: closed

Fixes bug described in issue #10618.


Labels:
name: Bug
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves

Comments:
tests pls

pls add a release note in whatsnew/0.17.0

any tests at a higher level that hit this code?

As far as I can tell this boundary case is already tested at the higher level (https://github.com/pydata/pandas/blob/master/pandas/tests/test_index.py#L919). Those tests never hit the low level cython code though, since these boundary cases are preempted in the high-level code (e.g., https://github.com/pydata/pandas/blob/master/pandas/core/index.py#L1419).

once you fix the testing function, pls squash, and ping.

Done.

thanks you sir!
support sql transactions
State: closed

this allows the `con` argument of `pd.read_sql`/`pd.read_sql_query` to be a sqlalchemy [`Session`](http://docs.sqlalchemy.org/en/rel_1_0/orm/session.html) object.

this allows for the use of temporary tables.


Labels:
name: IO SQL
description: to_sql, read_sql, read_sql_query

Comments:
@adamgreenhall Thanks for the contribution! 
Can you provide a bit more context? (I never used the ORM part of SQLAlchemy, only the Core part) But as a `Session` has a `begin` method and `execute` method, I suppose `read_sql` would just work regardless you provide it a Session or connection? (as the support for connections was just added in #10105, which should already allow temporary tables when using a Connection)

Why does it need sqlalchemy 1.0.0 ?

The reason your tests fail, is because it seems to run the tests with the fallback mode (although you put it in the correct test class). But this seems to indicate that it does not recognizes it as an sqlalchemy engine/connection/session, and therefore tries to use it as a DBAPI connection in fallback mode (hence the error 'has no attribute 'cursor'')

@jorisvandenbossche - #10105 does the trick with `sqlalchemy.engine.Connectable`. I didn't check the docs on master before adding this - learned my lesson for next time. Thanks!

ps. Not sure what the sqlalchemy recommended way to handle transactions is now - I think sessions got added in 1.0 - that was just the first thing I found in the docs, but the `with engine.begin():` code seems simpler.

But

```
In [46]: isinstance(session, sqlalchemy.engine.Connectable)
Out[46]: False
```

so the current code is not going to work for a Session. 
(I never used Sessions, but I can imagine that support for it would be useful for people using it)
TST: test_read_famafrench fails with HTTP 404
State: closed

Closes #10591. The same fix as pydata/pandas-datareader#53.

Though it is not a bug, added it to the bug section because users likely to check it when they meet 404.


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: IO Data
description: IO issues that don't fit into a more specific label

Comments:
looks good. merge on green.
DOC: 9789 Added missing letter, fixed link and Examples formatting.
State: closed

Fix for https://github.com/pydata/pandas/issues/9789. 

How do you feel about direct URLs in the doctrings?
I saw them in a few other docstrings. Pandas maintainers, any preferences?
Still need to fix the Examples at the bottom of the page.


Labels:
name: Docs

Comments:
lgtm. is there a standard to putting links in doc-strings so that they render when viewed in html (and text otherwise)? @jorisvandenbossche 

@jorisvandenbossche @jreback 
I found some examples in the pandas API docs where links are rendered with nicer markup and the format seems to be  ``Link text <URL>`_`. This also follows the [Sphinx documentation style guide for links and references](http://documentation-style-guide-sphinx.readthedocs.org/en/latest/style-guide.html#links-and-references)so unless you have any objections, I'll change the markup in my pull to reflect the Sphinx recommendations.

Hope the squashing worked.

About the url. In principle this can even be made shorter by using the internal links. Here, this would be something like:

```
:ref:`here <timeseries>`
```

The disadvantage of this is that when looking at the docstring in text format (in completion box eg) and not online, you don't see a url you can copy paste. 
So I think in this case your approach is a good compromise: use the full url (for when looking in text format), but with a shorter version in the html docs.

thanks!
Improve categorical concat speed by ~20x
State: closed

closes #10587

before (current master):

``` python
In [1]: s = pd.Series(list('aabbcd')*1000000).astype('category')

In [2]: timeit pd.concat([s,s])
1 loops, best of 3: 573 ms per loop
```

after (this PR):

``` python
In [1]: s = pd.Series(list('aabbcd')*1000000).astype('category')

In [2]: timeit pd.concat([s,s])
10 loops, best of 3: 30.1 ms per loop
```


Labels:
name: Enhancement
name: Performance
description: Memory or execution speed performance
name: Categorical
description: Categorical Data Type

Comments:
put the the before/after comparison at the top of the PR when you have a chance.

guess i should change to 20x :)

![image](https://cloud.githubusercontent.com/assets/953992/8712983/dda4ba3a-2b30-11e5-9182-95cc67b35ca5.png)

release note and squash :)

looks like i'm failing

hm weird ... fama_french url is throwing a 404

yeh, rebase on master i just took that test out

@jreback any more comments? (assuming travis passes)

joy that assertion just caused 6 failures

axis is ignored - only here for compat

maybe `axis in [0, -1]`?

@shoyer there's all sorts of crazy function level imports of the same function name and passing around the axis parameter. i think that consistency should be addressed in another PR

merge on green. 

FYI, the axis parameter is not used in _any_ of the `_concat_compats`. These are all 1-d. Not 100% sure why it was originally passed in.

hm i'm going to put the last two commits in a new PR so i can fix the category + category concatenation as well

thanks @cpcloud 

if you want to do another PR with those other changes ok
DOC-10371 Add note regarding supported interpolation methods for Series/DF
State: closed

Closes #10371

Suggested improvement for https://github.com/pydata/pandas/issues/10371


Labels:
name: Docs

Comments:
Looks good! 
About the formatting, for me this is good as well, or otherwise make it look like code with ```method='linear'```

Thanks @jorisvandenbossche!
Incorporated your suggestions.
Local build of the docstring looks ok.

Yep, looks good. Can you just squash the two commits into one? 

Sounds like a recipe for one of my famous git disasters, but I'll give it a go.

it is totally not difficult :-), just `git rebase -i master` and then force push

Merging now, thanks a lot!

@jorisvandenbossche did the squashing really work?  Wow, that must be the first time I do something like that in git without borking a million other files at the same time! Thanks!

I don't know _what_ exactly you did, but in any case, it was OK! 
BTW, really nice to see contribution to the docs! Hopefully there will follow some more :-)
DOC: Small improvement to convert_objects doc
State: closed

Fix small issues in convert_objects doc

[skip ci]


Labels:
name: Docs

Comments:
I think this is what I wanted (more indent of other stuff, not less of ipython blocks).  Does it seem right?

other way. The ipython blocks can be put on the left margin (as this is an entire separate section)

I think indentation is correct now.

yep, better! only one line was still incorrect I think

Thanks!
Fixed bug where read_csv ignores dtype arg if input is empty.
State: closed

The `CParser` implementation for `pd.read_csv` ignores argument `dtype` if the input is empty.  This pull request fixes this so that a DataFrame with the expected column types is returned.

``` Python
import pandas as pd                                                                                           
import cStringIO as stringio                                                                                  

data, dtype = 'a,b', 'i'

df = pd.read_csv(stringio.StringIO(data), dtype={'a': dtype})                                                                                                                                                               
assert df.dtypes[0].kind == dtype, "df.types[0].kind = %r, dtype = %r" % (df.dtypes[0].kind, dtype)
```

``
Traceback (most recent call last):
  File "issue_1.py", line 9, in <module>
    assert df.dtypes[0].kind == dtype, "df.types[0].kind = %r, dtype = %r" % (df.dtypes[0].kind, dtype)
AssertionError: df.types[0].kind = 'O', dtype = 'i'

```
```


Labels:
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: IO CSV
description: read_csv, to_csv

Comments:
this is going to fail when you have duplicate column names (although this is a separate issue), I think that you could deal with it when constructing the dtype dict (where the dtype would only be specificed once as its a dict)

e.g.

`"one,one"`

add this issue (the pr number) right in-line with the #10467 comment in the whatsnew/v0.17.0 (as its quite related)

I made the changes you requested and added some tests for duplicate column names.  I found that `read_csv` appends a suffix to the duplicate name so I don't think there's a problem (unless I misunderstood you).

Please let me know if there's something else before the PR can be merged (I can't get the build of my branch to go green right now due to HTTP 404 errors, which are unlikely to do with my changes).

try setting `mangle_dupe_columns=False` and (in another test) use `names=['one','one']`

Build is green @jreback - please let me know if there's something I've missed.

looks good. typo, then pls squash.

Good catch!  Apologies; should be okay now.

Ps. Did you want me to squash just the typo or all commits?

to a single commit.thxs

Done...

ok lgtm. ping when green.

@jreback - green! :-)

green on this page here. (somethimes takes a while)

@jreback - definitely green now.

thanks!
BUG: Fix typo-related bug to resolve #9266
State: closed

closes #9266 

Tried replicating the test case provided in the original issue. Let me know if it needs any changes or fixes.


Labels:
name: Bug
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: IO CSV
description: read_csv, to_csv

Comments:
pls add a note in whatsnew/v0.17.0

Thanks for the comments — will try to get to these changes tonight PT or early tomorrow PT. 

so this lgtm. pls squash and ping when green.

sorry. pls add a release note in whatsnew/0.17.0. refernce the original issue.

Poops. I messed the `rebase` up badly — I'm gonna close this pull request and open another. Sorry. :/
BUG: pd.eval with numexpr engine coerces 1 element numpy array to scalar
State: closed

Closes #10546. Used `assert_numpy_atray_equivalent` to guarantee array comparison, not scalar. This should be prior to #10542 and being changed to use `assert_numpy_array_equal` in #10542.


Labels:
name: Bug
name: Numeric Operations
description: Arithmetic, Comparison, and Logical operations

Comments:
@jreback Can you take a look for this?

looks ok. just a suggestion I put above (but may not be exactly correct)