BrandonFors/pandas-github-issues huggingface dataset:
ENH: accept dict of column:dtype as dtype argument in DataFrame.astype
State: closed

Continued in https://github.com/pydata/pandas/pull/13375

---

closes #7271

By passing a dict of {column name/column index: dtype}, multiple columns can be cast to different data types in a single command. Now users can do:

`df = df.astype({'my_bool', 'bool', 'my_int': 'int'})`

or:

`df = df.astype({0, 'bool', 1: 'int'})`

instead of:

```
df['my_bool'] = df.my_bool.astype('bool')
df['my_int'] = df.my_int.astype('int')
```


Labels:
name: Enhancement
name: Dtype Conversions
description: Unexpected or buggy dtype conversions

Comments:
astype currently has the `copy` kw. This is a big bogus as this almost always actually does copy if its changing to a new dtype (the exception is when its going to the same dtype).

related to #5769. It think we should change this to `inplace` (and deprecate `copy`) as its a bit more clear whats actually going on.

This could be done in another PR (before this one), or in this one is ok too.

further I wouldn't use `.ix` here as that would interpret numeric values as positional indexers for columns as this is a potentially confusing situation (and we don't do this anywhere else). If a non-matching column is passed then it will simply raise a `KeyError`. Instead use `[]`.

So pls add some more tests, including some invalid columns.

@StephenKappel can you rebase/update

@StephenKappel can you rebase / update

Sorry for neglecting this. The semester is finally over, so I'll get this cleaned up today :-)

@jreback - Rebased and updated based on your earlier feedback. Please let me know if any further updates are needed.

## [Current coverage](https://codecov.io/gh/pydata/pandas/pull/12086?src=pr) is **84.15%**

> Merging [#12086](https://codecov.io/gh/pydata/pandas/pull/12086?src=pr) into [master](https://codecov.io/gh/pydata/pandas/branch/master?src=pr) will increase coverage by **<.01%**

``` diff
@@             master     #12086   diff @@
==========================================
  Files           137        137          
  Lines         50287      50309    +22   
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
+ Hits          42311      42333    +22   
  Misses         7976       7976          
  Partials          0          0          
```

> Powered by [Codecov](https://codecov.io?src=pr). Last updated by [af7bdd3...0ac5354](https://codecov.io/gh/pydata/pandas/compare/af7bdd3883c8d61e9d9388d3aa699930eee7fff8...0ac535454f50784e1c35bc651eacf03ac48815d3)

can you rebase / update?

Ahh. Not sure how, but rebasing added everyone else's commits to this diff. Closing this PR and opening a clean one.

you don't need to make a new PR just rebase and force push 

I created a clean branch off of prod, cherry-picked my commits, and force pushed to this branch, but for some reason GitHub isn't recognizing it. Perhaps it's because I already closed the PR? But, I don't think I have access to reopen the PR...

Oh. It seems the PR can't be reopened because the branch has been subsequently force pushed :-(
Fix issue with merge-pr.py script
State: closed

The commented line shown produces an error on fresh clones of pandas, and is not needed AFAICT.


Labels:
name: Build
description: Library building on various platforms

Comments:
this gave me an error after it pushed the merged branch to master

it said: `Would you like to pick %s into another branch?`

and I said `Y`, at which point `latest_branch` was undefined. Not really even sure what that's supposed to do. I presume it would have cherry-picked to my local copy of master, as the remote master was already updated. 

The script exited and all was ok.

What is the purpose of that section?

You say no to that. We can also remove that section

@jreback I removed the cruft. If looks good merge away. If we find ourselves doing much backporting to maintenance branches in the future we can add some tools for that

merged. thanks.
BUG: GH12071 .reset_index() should create a RangeIndex
State: closed

closes #12071 


Labels:
name: Bug
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves

Comments:
Fixes #12071 

add this issue number to right after #939 in `whatsnew/v0.18.0`

Thanks. Updated the pull request. Not sure if I put the reference to the issue in the right place in "whatsnew" (this is my first PR to pandas).

add a test for Series as well

merged via https://github.com/pydata/pandas/commit/16a7799702f43d58ee2c2b3582c5a2c92b34fc0c

thanks!

Thanks for the help @jreback
PEP: pandas/core round 5 (nanops, ops, panel*)
State: closed


Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
Looks good, but:

```
$ flake8 pandas/core/nanops.py 
pandas/core/nanops.py:401:41: E226 missing whitespace around arithmetic operator
pandas/core/nanops.py:491:16: E226 missing whitespace around arithmetic operator
pandas/core/nanops.py:491:42: E226 missing whitespace around arithmetic operator
pandas/core/nanops.py:492:16: E226 missing whitespace around arithmetic operator
pandas/core/nanops.py:492:42: E226 missing whitespace around arithmetic operator
pandas/core/nanops.py:499:45: E226 missing whitespace around arithmetic operator
pandas/core/nanops.py:528:16: E226 missing whitespace around arithmetic operator
pandas/core/nanops.py:528:42: E226 missing whitespace around arithmetic operator
pandas/core/nanops.py:529:16: E226 missing whitespace around arithmetic operator
pandas/core/nanops.py:529:42: E226 missing whitespace around arithmetic operator
pandas/core/nanops.py:530:17: E226 missing whitespace around arithmetic operator
pandas/core/nanops.py:530:43: E226 missing whitespace around arithmetic operator
pandas/core/nanops.py:545:33: E226 missing whitespace around arithmetic operator

$ flake8 pandas/core/ops.py 
pandas/core/ops.py:94:41: E226 missing whitespace around arithmetic operator
```

Well this is fun:

```
(development)roch@roch ~/Software/pandas $ flake8 pandas/core/nanops.py
(development)roch@roch ~/Software/pandas $ flake8 pandas/core/ops.py
(development)roch@roch ~/Software/pandas $ git status
On branch pep8-round5
Untracked files:
  (use "git add <file>..." to include in what will be committed)

    pandas/version.py

nothing added to commit but untracked files present (use "git add" to track)
(development)roch@roch ~/Software/pandas $ 
```

that's pretty old file. how did that slip in?

I was just doing that to show no local changes.  Seems in pep8 version 1.6 they no longer enforce whitespace around the operator. I'm using 1.6.2.

@rockg that's super fun! for the preservation of our sanity let's settle on the >= 1.6 behavior; `**` isn't all that common anyhow

thank you!
CLN: fix all flake8 warnings in pandas/tseries
State: closed


Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
Looks good from a quick glance! (and for the diffs I could see)

Cool, I can merge after Travis passes if we're all good

If someone can +1 this and also review #12082 I can merge both of those

I fixed it! 

lgtm otherwise

@jreback I nixed the resample.py diff, will merge when the build passes

Cherry-picked in 1acdb31d9348f70f1cc77790fbc7bee568366cdd

merge away
ENH: GH12034 RangeIndex.__floordiv__ returns RangeIndex if possible
State: closed

xref #12034


Labels:
name: Enhancement
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves

Comments:
Not sure if I should always return `RangeIndex` when length < 2. 

@kawochen I had the same thoughts for `n=0/1` for the default constructor for a `NDFrame`, but thought its more consistent to always get a `RangeIndex`. Ideally would do the same if at all possible.

pls add this issue number to the whatsnew note as well.

ping when green

updated

merged via https://github.com/pydata/pandas/commit/8e2936172bdab24d6f5ad9722ec7bd0f249c2d6d

thanks!
CI: use wheels on all numpy dev builds
State: closed

- remove pydata 2.7 test build
- change 2.7 numpy_dev build to use wheels
- remove need for BUILD_TYPE, all builds are now conda

closes #12057 


Labels:
name: CI
description: Continuous Integration

Comments:
not sure we really need 2.7 & 3.5 numpy-dev builds, though they are faster now. and removed all original pydata builds.
CI: add 3.5 build with numpy_dev installed by wheel from master, #12057
State: closed

- adds back some 3.5 deps previously missing in conda
- closes #12057 (adds 3.5 wheel from nump)


Labels:
name: Build
description: Library building on various platforms
name: CI
description: Continuous Integration
name: Compat
description: pandas objects compatability with Numpy or Python functions

Comments:
this is adding a build to the allowed tests. But once we get a 2.7 wheel, total build time will be about the same. Of course we _could_ only have a single numpy_dev build (e.g. 3.5 or 2.7), but ATM this is done in allowed failures (so its 'after' the other builds).
BUG in .groupby for single-row DF
State: closed

No KeyError was raised when grouping by a non-existant column

Fixes #11741

Xref issue #11640, PR #11717


Labels:
name: Bug
name: Groupby

Comments:
merged via https://github.com/pydata/pandas/commit/e9e85988ec1240803379ae35ce8590e94537e747

thanks!
PEP: pandas/core round 3 (generic, groupby, index)
State: closed

yapf with manual modifications for error strings and bad yapf formatting.


Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
No github diffs again!  

There are some pieces of code that I think can be removed but want to confirm:
[index.py](https://github.com/pydata/pandas/blob/master/pandas/core/index.py#L1029) i/j are not used anywhere.
[groupby.py](https://github.com/pydata/pandas/blob/master/pandas/core/groupby.py#L1717) how can this work?  `agg_func` is not defined.

`index` i/j ok to remove

yeh that looks not tested, it for a transform on a Panel or greater. I think just raise `NotImplemented` (if the ndim >= 3) & remove the 2nd clause (where you noted). If nothing breaks in the test suite then ok!

I assume it should be `transform_func(result[:, :, i], chunk, comp_ids, accum)`

k, I can do that.

@rockg I would keep style changes and code clean-up (as in removal of unused code or a change of actual code) strictly separated (at least in a separate commit). Just to be sure, in case we overlooked something, then it is not burried in a gigantic commit for which you can't see the diff.

The main goal of this exercise is to make flake8 pass and cleanup some style were necessary.  What I cleaned out were unused variables reported by flake8 and, as a result, I don't see a problem including that here.

@rockg I've been putting `# noqa` on these lines and adding a `TODO` comment to come back and investigate

This is especially important in unit tests where I suspect that there are some test cases that may have skipped some intended checks

Okay, that seems reasonable.

ok, rebasing `core/groupby.py` is going to be painful for #11841 but go ahead (my touches are mostly reorg anyhow)

Is this good to merge?

I'll leave this to @jreback with the pending rebase

if u want to remove groupby.py from this would be great (I'll catch that in resample fixes)

otherwise lgtm

thanks! we're almost there =)
COMPAT: numpy compat with NaT != NaT, #12049
State: closed

closes #12049 


Labels:
name: Build
description: Library building on various platforms
name: Compat
description: pandas objects compatability with Numpy or Python functions

Comments:
cc @shoyer 

so just a minor change actually. So ok from me to leave numpy as is I think. (though you still might want to deprecate / do next release in any event)

I think that people will be unhappy if numpy issues a release that breaks the latest release of pandas.... so I think numpy needs to wait either way.

ok should be good now.

if its changed to a `DeprecationWarning` I _think_ we shouldn't show it, but will have to see

LGTM

now if Travis-CI would just work :< notice its been red all day
BUG: GH12050 Setting values on Series using .loc with a TZ-aware DatetimeIndex fails
State: closed

Fixes https://github.com/pydata/pandas/issues/12050 .

Taking just the values of an index object looses the timezone information.


Labels:
name: Bug
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Timezones
description: Timezone data dtype

Comments:
Fix has been derived from similar code in the method _getitem_iterable around line 980 of indexing.py .

Somebody with a better knowledge about the interplay of all the functions should comment before merging.

tests!

Is this one good enough? Or should I also test for variations?

pls also add a release note (bug fixes)

- Fixed the tests.
- Added release notes.
- Rebased onto current master.
- Squashed

Ready for merging from where i sit.

merged via https://github.com/pydata/pandas/commit/41821050049de766b07ed068b32839718544e39e

thanks!
ENH : Allow to_sql to recognize single sql type #11886
State: closed

Continued in #13252 

---

This solves #11886 
It checks whether the passed dtype variable is a dictionary.
If not, it creates a new dictionary with keys as the columns of the dataframe.
It then passes this dictionary  to the pandasSQL_builder.


Labels:
name: Enhancement
name: IO SQL
description: to_sql, read_sql, read_sql_query

Comments:
tests!

Update: Writing the tests has been more difficult that I thought it would be. I am currently working on it though.

@RahulHP For the test, best have a look at the existing `test_dtype` test (https://github.com/RahulHP/pandas/blob/to_sql-dictionary/pandas/io/tests/test_sql.py#L1502). You can see there how it is checked that the dtype is actually used in the database. 
And maybe you can just add your test to that function.

Update: I added the new test and it passes the Travis build.

Edits done

can you rebase / squash

Done, I followed the Contributing guide as best as I could but I think I may have messed it up.

@RahulHP There indeed slipped in some changes from other PRs. Normally, by doing exactly this, it should be solved:

```
git checkout to_sql-dictionary
git fetch upstream
git rebase upstream/master
git push -f origin to_sql-dictionary
```

In the rebase step, you will possibly need to resolve some conflicts

can you rebase/update

Sorry, I'll work on this in a few days.

pls rebase / update

I have created a new pull request here https://github.com/pydata/pandas/pull/13252
remove read_excel keyword NotImplemented error, update documentation #11544
State: closed

Followed up in #14326

---

Towards fixing bug #11544, with respect to comments on merged PR #11870.

I've removed the `NotImplemented` error for `parse_dates` and `date_parser` keywords in `read_excel`, seeing that they are implemented (see discussion in #11870). I've added an example of the intended functionality in the documentation


Labels:
name: IO Excel
description: read_excel, to_excel

Comments:
@grahamjeffries Thanks for working on this!

I am only not sure why you changed many of the tests by adding `parse_dates=True`. Why is that needed? 
Besides that, I think it would be good to add a test that actually tests the `parse_dates` keyword behaviour.

Okay, I'll change the warning so that it is raised only when `parse_dates is True`, add a test, and and update the documentation example.

Re: changes to the tests, in PR #11870 I removed those arguments in the test to avoid raising the `NotImplemented` error which is raised by the `read_excel` modifications in that same PR. Now, I've removed those modifications so I thought I'd add the arguments back. It turns out that it makes no difference to the testing outcomes, so I'm happy to go either way. Please advise which route is preferable.

For the tests, if it does not make a difference, you don't need to add this back. Rather try to add a test where it _does_ make a difference, so we test the functionality explicitly. 
Eg for the test1.xlsx, the first column is formatted as datetime in excel, not as a string, so indeed in those cases `parse_dates=True` is not needed.

@grahamjeffries can you update

can you update according to @jorisvandenbossche comments (and rebase)

@grahamjeffries Do you have time to update this?

this would be nice for 0.18.0, but not going to hold us up

@grahamjeffries can you rebase / update

@grahamjeffries Do you have some time to update/rebase this?

status of this?

needs a rebase

Either this should be merged, or I should fix the docs that use this keyword (now error during doc build), so I would keep it at the 0.19.0 milestone for now to remember that

can you rebase / update?

@grahamjeffries can you rebase / update this? 

I rebased and updated this in https://github.com/pydata/pandas/pull/14326
BUG: .plot modifing `colors` input
State: closed

Closes https://github.com/pydata/pandas/issues/12039

Just added a defensive copy in `_get_standard_colors` on list-likes.


Labels:
name: Visualization
description: plotting

Comments:
Travis had a (seemingly very unrelated) failure on one of the [allowed to fail builds](https://travis-ci.org/pydata/pandas/jobs/102507481). If anyone wants to merge this tonight feel free, otherwise I'll restart the travis build tomorrow and assume it was a random failure.
Copy on write using weakrefs (part 2)
State: closed

Continuation of #11500 


Labels:
name: Copy / view semantics

Comments:
@shoyer @jreback Ballpark timeline on refactor? Like, >1.5 years? 

If so, I think I'll set this aside.Trying to modify the DataFrame constructor just opened a huge can of worms I think it'd take a long time to work through. 

Till the refactor, any interest in making `SettingWithCopyWarning` into an exception?

I'd like to get it done in under a year; in combing through the codebase I've been making lists of things to do. It's a large effort but luckily we have a huge test suite to protect us and help keep the train on the rails. Will keep you posted

@wesm Great. Then will def set aside. :)

@nickeubank thanks for all the effort / commentary tests. I think these will be useful in the refactor.

@jreback thanks for the guidance! If nothing else, I now know a lot more about how references work in Python! :)

@nickeubank If you want to apply that knowledge for some internet karma, [this](http://stackoverflow.com/questions/34884536/what-is-the-point-of-views-in-pandas-if-it-is-undefined-whether-an-indexing-oper) came up yesterday ;)

@TomAugspurger on it, thanks Tom. 

Is there an open issue for this? I found a bunch of issues related to this, but didn't see any open ones.

there are refeneced issues

See https://github.com/pydata/pandas-design/issues/10
CI: lint for rest of pandas
State: closed

see output at the end: https://travis-ci.org/jreback/pandas/jobs/102213682

this only shows the summary stats. Once these are much lower we can remove the `-q` and 
show the actual errors.

As more files are fixed can move the checks to `lint.sh`


Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
@wesm
@jorisvandenbossche 
cc @rockg

any objections?

:+1:
MAINT: Make take_1d accept readonly buffers.
State: closed

This is a port of @ogrisel's fix in  #10070 to deal with 1d arrays. There's absolutely nothing new here beyond what's in the original patch.

closes #11502 


Labels:
name: Compat
description: pandas objects compatability with Numpy or Python functions

Comments:
ok, pls add a release note (you can put in bug fixes).

ping on green.

@jreback release note added

merged via https://github.com/pydata/pandas/commit/5d8cbb2745693eaa8e009edcd8d0511741a037e9

thanks!
ENH issue#11977 Added ValueError to Timestamp p…
State: closed

closes #11977


Labels:
name: Datetime
description: Datetime data dtype
name: Error Reporting
description: Incorrect or improved errors from pandas

Comments:
pls don't keep opening and closing pull requests

simply push to the same one

Question. How to I embed a graph in an issue?

https://guides.github.com/features/mastering-markdown/ search for images.

drag and drop

ideally like some tests that `assert_raises_regex` about the error message (add in `tseries/test_timeseries`)

@MattRijk can you update according to comments

@MattRijk can you update

pls reopen if you'd like to update

@jreback. Sorry for not getting back to you. I worked on the update a couple of times for a while. I couldn't get the message to show correctly
ENH issue#11977 Added ValueError(year is out of range) to Timestamp p…
State: closed

closes #11977 


Labels:
name: Datetime
description: Datetime data dtype
name: Error Reporting
description: Incorrect or improved errors from pandas

Comments:
Does the caught exception carry a message?  I don't think you can say categorically that the year is out of range.

@MattRijk simply pass thru message of the caught exception of the error
BUG: Raise ImportError when provided URI, but sqlalchemy is not installed.
State: closed

Rather than presume that it is an sqlite database and provide a confusing error.

Running (without sqlalchemy installed):

``` python
import pandas

db_uri = "postgresql://user:pass@localhost/db"
sql_query = "SELECT * FROM test"

pandas.read_sql(sql_query, db_uri)
```

Produces:

``` python
pandas/io/sql.pyc in execute(self, *args, **kwargs)
   1532             cur = self.con
   1533         else:
-> 1534             cur = self.con.cursor()
   1535         try:
   1536             if kwargs:

AttributeError: 'str' object has no attribute 'cursor'
```

Which links to:

https://github.com/pydata/pandas/blob/master/pandas/io/sql.py#L1534

Zooming into the problem:
1. https://github.com/pydata/pandas/blob/master/pandas/io/sql.py#L631-L637
2. https://github.com/pydata/pandas/blob/master/pandas/io/sql.py#L612-L619
3. https://github.com/pydata/pandas/blob/master/pandas/io/sql.py#L64-L68

So it looks like the URI is passed through and simply assumed to be a sqlite/mysql database object (using the old method), even if it isn't.

Raising an import error when a `URI string` is passed back by `_engine_builder` should solve the issue.


Labels:
name: Duplicate Report
description: Duplicate issue or pull request
name: IO SQL
description: to_sql, read_sql, read_sql_query

Comments:
@anirudhjoshi Thanks a lot for your contribution! 
However, there is already a recent PR that tries to solve the same issue: https://github.com/pydata/pandas/pull/11920. That is not merged yet, but I will let the author of that PR the chance first to update it.

Sure thing. I did a search, must have missed it.

Although I believe his error raise is in the wrong area.
BUG: remove millisecond field that raises valueerror
State: closed

This would appear in the `dir` and cause tab completion to a field that would raise a `ValueError`. When I brought this up in gitter it seemed like it was safe to remove it.

I couldn't find any tests that hit this descriptor.


Labels:
name: Datetime
description: Datetime data dtype
name: API Design

Comments:
yeh, I think this was historicaly left in. can you add a test (assert that it raises), so when someone tries to add it back it will fail!. ping on green.

Added the test, build is now passing

thanks!