BrandonFors/pandas-github-issues huggingface dataset:
Revising the HDF5 docs a bit, added another level to the toc
State: closed

- makes the toc 4 levels
- increases the width of the side bar a bit
- increases the overall width just a bit

![io tools text csv hdf5 pandas 0 16 2 557 g23b9715 documentation](https://cloud.githubusercontent.com/assets/953992/9696820/5bbb9528-5346-11e5-8e52-15759d02257a.png)
![io tools text csv hdf5 pandas 0 16 2 557 g23b9715 documentation 1](https://cloud.githubusercontent.com/assets/953992/9696821/5d955190-5346-11e5-9e3f-3025c232f166.png)


Labels:
name: Docs

Comments:
@jorisvandenbossche 

Looks good!

+1 on making it a bit wider (certainly for the side bar), but maybe 800 is enough (now the increase is quite big, from 720 to 840; 800 is what sphinx readthedocs theme uses)
ENH: Added warning when excel file contains duplicate column names
State: closed

See #10970 and #10982. Users face bugs but no informative errors are given. This should warn them and inform them what might be wrong. 


Labels:

Comments:
rather than a warning, should be able to handle this.  of course a tests would help

So like automatic renaming duplicate column names?
On Sep 4, 2015 1:52 PM, "Jeff Reback" notifications@github.com wrote:

> rather than a warning, should be able to handle this. of course a tests
> would help
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/pydata/pandas/pull/10991#issuecomment-137805257.

no, they should simply be handled.
ENH: Added skipcols option for CParser
State: closed

See #10882 
But I am having trouble testing this change. When I run `nosetests pandas/io/tests/test_parsers.py`, it says `Segmentation fault: 11` and this pops up `Python quit unexpectedly while using the parser.so plug-in.`

Any ideas of how to deal with this?


Labels:

Comments:
@jreback Do you know how to fix this? Thanks.

you need to debug what's happening in cython prob. I usually just insert print statements, but you can also use `gdb` (though haven't done it in a while)

Ok. Thanks. I thought it's a problem of not having installed some neccessary dependencies. 

closing, pls reopen if you can fix according to comments
TST/DOC #10846 Test and document use of SQLAlchemy expressions in read_sql()
State: closed

This provides 2 tests using SQLALchemy expressions to read SQL data into a dataframe and accompanying documentation.

It attempts to address #10846 by explaining how this feature is already supported by pandas.

Closes #10846


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Docs
name: IO SQL
description: to_sql, read_sql, read_sql_query

Comments:
@stephenpascoe Sorry for the late reply. So this actually already just worked.

Thanks a lot for the tests. And nice docs!

Two questions:
- Can you squash?
- Maybe also add to the `read_sql`/`read_sql_query` docstrings that it can also accept an SQLAlchemy Select statement

Squashed, docstrings added and rebased.

Thanks a lot!
CLN: removes BinGrouper kind of cython methods
State: closed


Labels:
name: Groupby
name: Clean

Comments:
prob the most lines removed at one time!

can you do an asv run and see if any anomalies.

you can kill that bin grouper test (that you are now skipping) as well.

I checked with vbench and benchmarks were stable. [this line](https://github.com/pydata/pandas/pull/10976/files#diff-720d374f1a709d0075a1f0a02445cd65R1799) is the only added overhead which is pretty efficient

gr8 thanks!

Very nice! I was wondering why we had all these binned groupby routines in Cython.
BUG: DataFrame constructor should not promote complex64 dtypes (GH10952)
State: closed

Addresses [GH10952](https://github.com/pydata/pandas/issues/10952).

When passing an array with complex64 dtype to the DataFrame constructor, that array will no longer be automatically promoted to the complex128 dtype (different complex dtypes are now allowed).


Labels:
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: API Design
name: Complex
description: Complex Numbers

Comments:
very simple! nice!

merged via https://github.com/pydata/pandas/commit/b5b06cf7615a8c8f956aa80ef023a3748e786df5

tip: to avoid merge conflicts in the whatsnew, put the entry not at the end, that's why I leave blank spaces in the Bug fixes (though this is a slight api change).

Thanks! I'll remember to leave spaces in the file next time.
DOC: Consistent variable names (sheetname vs sheet_name, issue 10559)
State: closed

See issue #10559. Done testing after the changes. 


Labels:

Comments:
@terrytangyuan the tests are failing because you changed the keyword names as well in the implementation, where this depends on the libraries used by pandas (xlrd, openpyxl, etc). So you also have to check there what name they are using.

@jorisvandenbossche  Thanks. I tried but failed again. It's hard to deal with those internal usages. Might just leave this task to people who are more familiar with them.
TST: refactored Excel reader tests
State: closed

This PR implements more flexible Excel (spreadsheet) reader tests. The work was originally part of the larger but unsuccessful PR #9070. 

This PR includes:
- Excel reader base class tests to facilitate adding the similar tests for other readers
- not all Excel file formats (xls, xlsm, xlsx) had a corresponding test file. This PR adds the missing ones. Consequently, all Excel data files are now tested against all 3 Excel file formats.
- rename `test` to `test1` (for xls, xlsm, xlsx extensions) for consistency with `test2..4` test data files. This also means that before merging (but after Travis has ran the tests) the URL of Excel test file `test1.*` has to be changed from `davidovitch/pandas` to `pydata/pandas` in `pandas/io/tests/test_excel.py` at line [378](https://github.com/davidovitch/pandas/blob/master/pandas/io/tests/test_excel.py#L378).


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: IO Excel
description: read_excel, to_excel

Comments:
can you report the number of tests on current master (for the excel stuff) and number on this branch. Trying to figure out if anything is left out.

With this PR I have the following:

```
# pandas/io/tests/test_excel.py
Ran 283 tests in 9.364s
OK (SKIP=10)
```

and with pydata/pandas@9bdae60eca392877a0fbca4e671466f28bc8a0d9 (same as above minus the 2 commits of this PR) I get:

```
# pandas/io/tests/test_excel.py
Ran 242 tests in 6.969s
OK (SKIP=10)
```

I have 10 skipped tests with openpyxl=1.8.6 being indicated as incompatible (the same for both this PR and pydata/pandas@9bdae60eca392877a0fbca4e671466f28bc8a0d9). With openpyxl=2.2.4 I get failures that I haven't look into yet, but I also get these failures with pydata/pandas@9bdae60eca392877a0fbca4e671466f28bc8a0d9. These failures are most likely due to my system since Travis is not complaining about them, but I will have to investigate this further to figure out what exactly is going on.

@chris-b1 can you have a look here

Added a couple comments, overall approach seems pretty reasonable.

@chris-b1 does this sufficiently address your comments?

@davidovitch pls squash.

ok, going to merge this, @chris-b1 you will prob need to rebase your excel pr

merged via https://github.com/pydata/pandas/commit/98b3a5df5b8919eeeabd025b872bfc9c2e3721c3

thanks!

@jreback: Sorry, was too late for squashing and correcting one small thing: the URL of the renamed test file was still pointing my repo instead of pydata/pandas. See PR #11009.

I fixed this in my other PR when I rebased.
On Sep 5, 2015 3:14 PM, "David Verelst" notifications@github.com wrote:

> @jreback https://github.com/jreback: Sorry, was too late for squashing
> and correcting one small thing: the URL of the renamed test file was still
> pointing my repo instead of pydata/pandas. See PR #11009
> https://github.com/pydata/pandas/pull/11009.
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/pydata/pandas/pull/10964#issuecomment-137990573.

ah ok, great! No need for PR #11009, closed now.
BUG: DataFrame subplot with duplicated columns output incorrect result
State: closed

When `DataFrame` has duplicated column name, each subplot will contain all the lines with same name.

```
import pandas as pd
import numpy as np
df = pd.DataFrame(np.random.rand(5, 5), columns=list('aaaaa'))
df.plot(subplots=True)
```

![figure_1](https://cloud.githubusercontent.com/assets/1696302/9609047/7acba4d0-510c-11e5-872d-1fcc1fcf8ce6.png)


Labels:
name: Bug
name: Visualization
description: plotting

Comments:
minor comment, lgtm, merged on pass

Thanks for review. Fixed and merging.
BUG: Fixed bug that Timedelta raises error when slicing from 0s (issue #10583)
State: closed

closes #10583 


Labels:
name: Bug
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Timedelta
description: Timedelta data type

Comments:
ping on green

@jreback looks like all the PRs on Travis are pending (none of them is running now)? BTW, how can I cancel/drop one of the tasks that I created? (In case in the future when I need to fix my commits)

things can take a while on travis. pandas only has a limited amount of capacity (across pandas master and other accounts), so could be a while as I pushed a bunch of things recently.

you can go to travis and cancel things manually if you want. but the green button is run by a different process.

I see. On green now!  @jreback 

thank you sir!
BUG: Fixed bug that Timedelta raises error when slicing from 0s (issue #10583)
State: closed

closes #10583 


Labels:
name: Bug
name: Timedelta
description: Timedelta data type

Comments:
lgtm. pls add a whatsnew note. ping when green.

I just messed up my last push. It combined some of your previous commits...would this still be ok? I am not sure how to make it back...

see how to here: http://pandas.pydata.org/pandas-docs/stable/contributing.html

you need to rebase on master

I've created a new PR [here](https://github.com/pydata/pandas/pull/10960). Sorry for any inconvenience. 

FYI you probably would have been able to recover with `git reflog`, which records all the states your branch has been in. You'd use that to find which hash to `git reset` back to.

Got it. Thanks!
On Sep 1, 2015 8:55 AM, "Tom Augspurger" notifications@github.com wrote:

> FYI you probably would have been able to recover with git reflog, which
> records all the states your branch has been in. You'd use that to find
> which hash to git reset back to.
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/pydata/pandas/pull/10957#issuecomment-136707696.
Add support for math functions in eval()
State: closed

closes #4893

Extends the eval parser to accept calling math functions.


Labels:
name: Enhancement
name: API Design
name: Numeric Operations
description: Arithmetic, Comparison, and Logical operations

Comments:
looks good. can you add a whatsnew note in 0.17.0 docs. pls squash as well. ping when green.

once I merged this: https://github.com/pydata/pandas/pull/10975 (or you can rebase on top)

pls update the tests as you indicated 

@sklam can you update the tests

@sklam can you update

hmm, it appears `numexpr` automatically upcasts complex64 -> complex128?
is that a bug? or in the spec?

```
In [1]: df = DataFrame({'a': np.random.randn(10).astype(np.complex64)})

In [2]: df.dtypes
Out[2]: 
a    complex64
dtype: object

In [3]: df.eval('b = sin(a)')

In [4]: df
Out[4]: 
                a                     b
0  (-0.570414+0j)  (-0.539980304291+0j)
1   (0.623834+0j)   (0.584150916712+0j)
2     (1.6842+0j)   (0.993577225836-0j)
3    (1.54063+0j)   (0.999545174164+0j)
4   (0.788171+0j)   (0.709064831257+0j)
5  (-0.916847+0j)  (-0.793687403046+0j)
6   (0.775991+0j)   (0.700423483759+0j)
7  (-0.654412+0j)  (-0.608692614148+0j)
8   (-1.62652+0j)  (-0.998447583421-0j)
9   (0.152154+0j)   (0.151567580954+0j)

In [5]: df.dtypes
Out[5]: 
a     complex64
b    complex128
dtype: object
```

thanks @sklam
PERF: improves performance in SeriesGroupBy.count
State: closed

BUG: closes bug in Series.count when index has nulls

``` python
In [4]: ts
Out[4]:
a  1      0
   2      1
b  2      2
   NaN    3
c  1      4
   2      5
dtype: int64

In [5]: ts.count(level=1)
Out[5]:
1    2
2    4          # <<< BUG!
dtype: int64

In [6]: from string import ascii_lowercase

In [7]: np.random.seed(2718281)

In [8]: n = 1 << 21

In [9]: df = DataFrame({
   ...:     '1st':np.random.choice(list(ascii_lowercase), n),
   ...:     '2nd':np.random.randint(0, n // 100, n),
   ...:     '3rd':np.random.randn(n).round(3)})

In [10]: df.loc[np.random.choice(n, n // 10), '3rd'] = np.nan

In [11]:

In [11]: gr = df.groupby(['1st', '2nd'])['3rd']

In [12]: %timeit gr.count()
The slowest run took 6.67 times longer than the fastest. This could mean that an intermediate result is being cached
1 loops, best of 3: 86.4 ms per loop

In [13]: %timeit gr.count()
10 loops, best of 3: 87 ms per loop
```

on branch:

``` python
In [5]: ts.count(level=1)
Out[5]:
 1     2
 2     3
NaN    1
dtype: int64

...

In [12]: %timeit gr.count()
The slowest run took 12.29 times longer than the fastest. This could mean that an intermediate result is being cached
1 loops, best of 3: 43.1 ms per loop

In [13]: %timeit gr.count()
10 loops, best of 3: 43.5 ms per loop
```


Labels:
name: Groupby
name: Performance
description: Memory or execution speed performance

Comments:
@behzadnouri pls rebase and make the changes as above

@behzadnouri can you update according to comments

@jreback the added test will fail if i change it what u suggest.

@behzadnouri I just don't think its worth it to support this kind of behavior in Series.count which is inconsitent with all other stat functions on how they handle levels, they just dispatch to groupby.

then plz go ahead and change it

https://github.com/pydata/pandas/issues/443 would fix this, e.g. we need an option like:

`s.count(level=1, skipna=False)`

hmm, looks like my way would break some existing tests....ok will merge as is, and when we eventually add nan group handling in groupby can simplify this code.

need rebase?

I just did it. thxs.

merged via https://github.com/pydata/pandas/commit/33723f9577d451c20db68b4d10ed90e81f54e03e
BUG: Index name lost in conv #10875
State: closed

Addresses #10875, when `Index` is converted via `to_datetime`, `to_timedelta`
part of #9862 master issue


Labels:
name: Bug
name: Datetime
description: Datetime data dtype
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Timedelta
description: Timedelta data type

Comments:
@jreback moved where you noted

@chris-b1 thank you sir!

your contributions have been awesome! keep em coming!
Fixing column_format argument passing, #9402
State: closed

closes #9402 


Labels:
name: Bug
name: IO LaTeX
description: to_latex

Comments:
pls add a release note (bug fix section).

pls squash, then ping when green.

I'm in transit today, no internet. I'll do that back home tonight or tomorrow. Best

Le 31 août 2015 01:53:26 GMT+01:00, Jeff Reback notifications@github.com a écrit :

> pls squash, then ping when green.
> 
> ---
> 
> Reply to this email directly or view it on GitHub:
> https://github.com/pydata/pandas/pull/10944#issuecomment-136224665

## 

Envoyé de mon appareil Android avec K-9 Mail. Veuillez excuser ma brièveté.

Found a connection :) 

I messed up the rebase somewhat, so there's an extra useless commit, but for the rest I think I did what you asked.

I plan on fixing some more `to_latex` bugs this week, shall I keep on doing this on that branch and PR, or is it better to make a branch for each bug?

I'll squash, don't worry.

always make new features on separate branches. thxs

merged via https://github.com/pydata/pandas/commit/fb42766b3eb55155d67548d364672d254be7ffb6

thanks!
BUG: passing columns and dict with scalar values should raise error
State: closed

Fixes [GH10856](https://github.com/pydata/pandas/issues/10856).

``` Python
>>> pd.DataFrame({'a':0.1}, columns=['b'])
ValueError: If using all scalar values, you must pass an index
```

Trying to raise this error was slightly trickier than I anticipated - this was the only way that didn't break existing tests. If no index is passed to the constructor, `extract_index` is called to check whether the dictionary contacts only scalar values (and raises the ValueError if so).

This check now happens _prior_ to preselecting any columns. If people are happy with this approach I can write tests for the PR.


Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Error Reporting
description: Incorrect or improved errors from pandas

Comments:
seems reasonable. Pls add the test in of course! 
and a release note. squash / ping me when green.

@jreback: added the test and note and checks are green.

@jreback the discussion we had during the sprint is the following: although this patch fixes the actual reported issue (raising a ValueError when a scalar dict is passed), this also introduces another inconsistency in some way:

```
In [25]: pd.DataFrame({'a':[0.1]}, columns=['b'])
Out[25]:
Empty DataFrame
Columns: [b]
Index: []
```

In the above case, **first** the 'reindexing' of the dict on the columns is done, ending up with an empty dict, leading to an empty dataframe with columns but **no** index (so ignoring the index that would have been created from the original data). 
So if you apply the same logic on the case in this issue, you also end up with an empty dict, but in this case the index is not ignored, as you still raise an error because the _original_ data would cause an error when constructing an index from that.

But there is not really a clear correct solution I think (and it is also rather a cornercase), so I am fine with merging this.

thanks guys. catching all of these corner cases is tricky.
ENH: Add Series.dt.total_seconds GH #10817
State: closed

Implements a Series.dt.total_seconds method for timedelta64 Series.

closes #10817


Labels:
name: Enhancement
name: API Design
name: Timedelta
description: Timedelta data type

Comments:
does timedelta.total_seconds() provide fractional second as wel?

Yes, fractional seconds are included:

```
In [1]: s = Series(pd.timedelta_range('1day',periods=5,freq='1ms'))

In [2]: s
Out[2]: 
0          1 days 00:00:00
1   1 days 00:00:00.001000
2   1 days 00:00:00.002000
3   1 days 00:00:00.003000
4   1 days 00:00:00.004000
dtype: timedelta64[ns]

In [3]: s.dt.total_seconds
Out[3]: 
0    86400.000
1    86400.001
2    86400.002
3    86400.003
4    86400.004
dtype: float64
```

not what I mean

is the actual Python timedelta.total_seconds have fractions (I think yes) just confirming

Ah, I understand. Yes, it does:

```
In [38]: t = timedelta(days=1,microseconds=40)

In [39]: t.total_seconds()
Out[39]: 86400.00004
```

Indeed, didn't see that, but +1 making it a method instead of a property for consistency with datetime.timedelta/pd.Timedelta

Further, @sjdenny can you add a whatsnew notice (see doc/source/whatsnew/v0.17.0.txt, somewhere in 'other enhancements')

In extending the (nanosecond-precision) tests, I've come across this behaviour:

```
In [18]: td = pd.Timedelta(nanoseconds=50)

In [19]: td.nanoseconds
Out[19]: 50

In [20]: td.total_seconds()
Out[20]: 0.0

In [21]: td.total_seconds() == 0.0
Out[21]: True

In [22]: td2 = pd.Timedelta(microseconds=50)

In [23]: td2.total_seconds()
Out[23]: 5e-05
```

It appears (e.g. [here](https://docs.python.org/2/library/datetime.html#datetime.timedelta.total_seconds)) that timedelta.total_seconds() aims for microsecond accuracy only. Is this the behaviour we want to reproduce in Series.dt.total_seconds()? Nanosecond precision appears preferable, but then the scalar and Series functions would have slightly different behaviour.

It should be consistent between both in any case, but maybe we can also see the `Timedelta` behaviour as a bug?

I suspect that this method is just inherited from datetime.timedelta, so maybe we will have to overwrite it ourselves in the subclass.

BTW, the reason of this difference is that datetime.timedelta does not support nanoseconds, while pd.Timedelta does

the pandas functions should work with ns precision in all cases

so need to override the scalar function as well

@sjdenny This should probably be added somewhere here: https://github.com/pydata/pandas/blob/master/pandas/tslib.pyx#L2415 (I think the `self.value` will give you the nanoseconds, and then you can use the same approach)

I think also add the method `.total_seconds()` on the `NaTType` object (returns `np.nan`) for compat. (and pls add a test as well).

@sjdenny awesome job!

@sjdenny Thanks a lot!
consistent imports in text.rst
State: closed


Labels:
name: Docs

Comments:
lgtm. can you squash. ping when pushed / green.

@SebastianRubbert docs still build fully? (note that we don't actually test this on travis, except after it is merged).

I rebuilt the doc and I did not have errors in the files I edited. 

merged via https://github.com/pydata/pandas/commit/5803e4c7417130536e4c12412c03c5bbaac60219

thanks!
updating docs for the new sorting mechanisms - GH #10886
State: closed

closes #10886 

I didn't notice warnings or errors associated with those modifications when building the docs.

No more `FutureWarning` either (except in a the v. 0.13 what's new, which is legit)... 


Labels:
name: Docs

Comments:
thank you sir!

My pleasure!

Le 31 août 2015 02:03:32 GMT+01:00, Jeff Reback notifications@github.com a écrit :

> thank you sir!
> 
> ---
> 
> Reply to this email directly or view it on GitHub:
> https://github.com/pydata/pandas/pull/10937#issuecomment-136226446

## 

Envoyé de mon appareil Android avec K-9 Mail. Veuillez excuser ma brièveté.
BUG: tests that can't fail
State: closed

tm.equalContent does not raise an exception, it returns a Bool.  Therefore these tests need wrapping in self.assertTrue()


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: IO SQL
description: to_sql, read_sql, read_sql_query

Comments:
maybe better to make this `assertEqualContents` (e.g. rename and put the assert in `utils/testing.py`)

does this have a specific issue associated?

No, he just discovered it now while looking at the code

@stephenpascoe can you update this?

closing, but if you would like to continue working on this pls reopen
DOC: consistent imports (GH9886) part V
State: closed

I made a few alterations to the consistent imports in the merging.rst file. 

Does anyone have any objections to what I did?

I notice a few problems with the rebase though first - that's 12 commits rather than one. 


Labels:

Comments:
Ok updated the commit by rebasing.

Ok this should be all the changes done in the merging.rst file. 

@springcoil if behooves you to keep your master up to date and clean, e.g. don't make changes on it.

so

```
git checkout master
git reset upstream/master --hard
```

then when checking out

```
git checkout master
git pull
git checkout -b mybranch --track master
```

will keep things clean

Reopening this branch. 

This is deprecated by [#10941](https://github.com/pydata/pandas/pull/10941)