BrandonFors/pandas-github-issues huggingface dataset:
BUG: Intersection buggy for non-monotonic non-unique indices (GH8362)
State: closed

Fixes intersection bug in the case of non-monotonic non-unique indexes. 
closes #8362 


Labels:
name: Bug
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves

Comments:
needs a test (e.g. your example from the issue)

also pls add a release note in v0.15.0.txt

Okay I think it's fixed now.

looks good
pls squash and rebase 
ping in green

merged via https://github.com/pydata/pandas/commit/68fdacd12e7963706f0ac1b859058bddb6b814bd

thanks!
ENH: Added file path existence check for read_hdf
State: closed

closes  #7715 

Added simple check for file path existence. It might be more advisable to deprecate this function and separate file buffer and file path read functionality, but the current form matches the rest of the API.

I wasn't sure if there was any tests I needed to add, but if there is, let me know.


Labels:
name: API Design
name: IO HDF5
description: read_hdf, HDFStore

Comments:
needs a test. You need to check that this raises appropriately..

Okay, where should the test be placed? Or should I create a new test file?

`pandas/io/tests/test_pytables.py`

you can create a new function; model off some of the other ones (e.g. search for `read_hdf` and put it somewhere after)

also will need a release note in v0.15.0.txt API section

Oh duh, sorry. I was looking in the wrong place for tests. Will do! Thanks.

minor comment, pls squash as well. looks good otherwise. ping when ready/green.

@jreback I think everything should be good. Let me know if I need to fix anything else. Thanks for your patience! :)

thanks!
DataFrame.dropna bug fix
State: closed

Bug fix for #8303.  Now raise KeyError if any non-existent columns are passed in the subset argument.


Labels:
name: Bug
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate

Comments:
need a test for this

pls add a release note in v0.15.0 Bug Fix section as well (and squash when you are done). ping when green. Git references are here: https://github.com/pydata/pandas/wiki/Using-Git

I added the release note and changed the fix to use the more natural pandas/numpy syntax.  I submitted a new pull request with all of this in one commit.  (I did something bad to my 8303 branch when I tried to rebase and I couldn't push from it - Hopefully I get the hang of rebasing soon.)

On Sep 23, 2014, at 7:23 AM, jreback notifications@github.com wrote:

> pls add a release note in v0.15.0 Bug Fix section as well (and squash when you are done). ping when green. Git references are here: https://github.com/pydata/pandas/wiki/Using-Git
> 
> —
> Reply to this email directly or view it on GitHub.
BUG: OLS with clustering and nw_lags does not error
State: closed

Added statement to raise ValueError when OLS is run with clustering and
Newey-West adjustments. Closes #5884.  This is a re-attempt at
GH8170.


Labels:
name: Error Reporting
description: Incorrect or improved errors from pandas

Comments:
Tests are passing now--I think this is done.

can u squash into 1 commit 

ping when green (after squash)

merged via https://github.com/pydata/pandas/commit/391e90b9939d8e912c3be16343b493c20f6ca55b

thanks!

Thanks for merging this
Changed round_trip converter to Python converter instead of strtod
State: closed

As @mdickinson noted in #8044.


Labels:
name: IO CSV
description: read_csv, to_csv
name: Compat
description: pandas objects compatability with Numpy or Python functions

Comments:
pls do a perf check, assuming ok, ping on green

@amras1 progress?

@jreback any idea what [this](https://travis-ci.org/amras1/pandas/jobs/36404385#L1602) could be?

looks like u rebased (which was the excel issue / that fixed a few days ago)

why was this failing on 2.6?

looks good now

Silly mistake when I wrote it, I had `<` instead of `>=` for the Python >= 2.7 check.

thanks for the fix!

And a big thank you from me, too!

No problem!
Bug on pivot_table with margins and dict aggfunc
State: closed

closes #8349

bug on pandas 0.14.1 pivot_table using dictionary aggfunc and margins.

Test code:

<pre>
<code>
df=pandas.DataFrame([ 
{'JOB':'Worker','NAME':'Bob' ,'YEAR':2013,'MONTH':12,'DAYS': 3,'SALARY': 17}, 
{'JOB':'Employ','NAME':'Mary','YEAR':2013,'MONTH':12,'DAYS': 5,'SALARY': 23}, 
{'JOB':'Worker','NAME':'Bob' ,'YEAR':2014,'MONTH': 1,'DAYS':10,'SALARY':100}, 
{'JOB':'Worker','NAME':'Bob' ,'YEAR':2014,'MONTH': 1,'DAYS':11,'SALARY':110}, 
{'JOB':'Employ','NAME':'Mary','YEAR':2014,'MONTH': 1,'DAYS':15,'SALARY':200}, 
{'JOB':'Worker','NAME':'Bob' ,'YEAR':2014,'MONTH': 2,'DAYS': 8,'SALARY': 80}, 
{'JOB':'Employ','NAME':'Mary','YEAR':2014,'MONTH': 2,'DAYS': 5,'SALARY':190} ])

df=df.set_index(['JOB','NAME','YEAR','MONTH'],drop=False,append=False)

df=df.pivot_table(index=['JOB','NAME'],
columns=['YEAR','MONTH'],
values=['DAYS','SALARY'],
aggfunc={'DAYS':'mean','SALARY':'sum'})
</code>
</pre>


All works fine but raise error using margins:

<pre>
<code>
df=df.pivot_table(index=['JOB','NAME'],
columns=['YEAR','MONTH'],
values=['DAYS','SALARY'],
aggfunc={'DAYS':'mean','SALARY':'sum'},
margins=True)
</code>
</pre>


df=df.pivot_table(index=['JOB','NAME'],columns=['YEAR','MONTH'],values=['DAYS','SALARY'],aggfunc={'DAYS':'mean','SALARY':'sum'},margins=True)
File "/usr/local/lib/python2.7/site-packages/pandas-0.14.1-py2.7-linux-x86_64.egg/pandas/util/decorators.py", line 60, in wrapper
return func(_args, *_kwargs)
File "/usr/local/lib/python2.7/site-packages/pandas-0.14.1-py2.7-linux-x86_64.egg/pandas/util/decorators.py", line 60, in wrapper
return func(_args, *_kwargs)
File "/usr/local/lib/python2.7/site-packages/pandas-0.14.1-py2.7-linux-x86_64.egg/pandas/tools/pivot.py", line 147, in pivot_table
cols=columns, aggfunc=aggfunc)
File "/usr/local/lib/python2.7/site-packages/pandas-0.14.1-py2.7-linux-x86_64.egg/pandas/tools/pivot.py", line 191, in _add_margins
row_margin[k] = grand_margin[k[0]]

KeyError: 'SALARY'


Labels:
name: Bug
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
always create the tests first!

you can use the test example you gave, put in `pandas/tools/tests/test_pivot.py` where the other margin items are calculated.

Did it, sorry for all, this is my first time.

ok, see docs here: https://github.com/pydata/pandas/wiki

no need to create a new PR for this.

just add a commit.

need a release note in v0.15.0 (bug fix section). and pls squash. ping when done.

Bug fix note added.

gr8. can you squash? https://github.com/pydata/pandas/wiki/Using-Git

uff... it was hard.
Hope all is ok now.

merged via https://github.com/pydata/pandas/commit/4074957c0933893afcb2ec785fcba6520789c1df

thanks!
DOC: move trellis under visualization.rst
State: closed

Closes https://github.com/pydata/pandas/issues/8351

Just moved it and changed a few heading levels. I could add a note about considering seaborn. I've never used rplot.


Labels:
name: Docs
name: Visualization
description: plotting

Comments:
ok merge when ready

@jreback The travis faliure was from the openpyxl stuff, which you fixed in https://github.com/pydata/pandas/commit/8069f4779cd1b0fe84abc53c73d8e6d0a44b0674

OK to merge still?

yep go ahead 

Thanks, I'll keep an eye on this build.
BUG: Work-around openpyxl-2.1.0 NumberFormat removal
State: closed

See #8342. This patch should allow all tests to pass with any supported openpyxl v1, and with openpyxl 2.1.0. Consider this an interim measure, not a complete fix.


Labels:
name: IO Excel
description: read_excel, to_excel
name: Compat
description: pandas objects compatability with Numpy or Python functions

Comments:
ok!

let me know when green (then will leave the issue open). If you have a more complete fix later can merge (other wise can simply document / put in some versioning checks)

@neirbowj saw your comment. I'll watch for the green. Have a nice flight!

@neirbowj - thanks for taking the time to work on this fix! I was really
hoping it would be a one-line fix-the-import kind of thing but I'm glad we
can still get openpyxl to work

On Mon, Sep 22, 2014 at 7:47 AM, jreback notifications@github.com wrote:

> @neirbowj https://github.com/neirbowj saw your comment. I'll watch for
> the green. Have a nice flight!
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/pydata/pandas/pull/8352#issuecomment-56383922.

closing in favor of #8356
BUG: _Openpyxl2Writer mishandles merged cell style
State: closed

Thanks to @jtratner for catching this.


Labels:
name: Bug
name: IO Excel
description: read_excel, to_excel

Comments:
can u include a test for that inner loop (as clearly no test exists now)

I'll see what I can do, perhaps tomorrow night.

It's stunning how much time can pass in a blink before getting to "tomorrow night."

@neirbowj this needs to go in 0.15.0, yes?

merged via https://github.com/pydata/pandas/commit/e0d9b1a7807c41374b0212c3b09d5b73c1fecec6

thanks!
CLN: pep8 clean up of sql.py
State: closed


Labels:
name: IO SQL
description: to_sql, read_sql, read_sql_query

Comments:
@jreback master is broken after merging this PR (https://travis-ci.org/pydata/pandas/jobs/35873580), but due to excel importing issues, so something totally unrelated. Any idea?

just looks like an import error which happens

I think Travis fs freezes and doesn't load the module -

I restarted 

if it repeats then can investigate again 

Hmm, I already restarted it once, but we'll see

hmm ok
assume it works locally for u (since u didn't touch that module)

it IS possible that openpyxl versions changed!
(as the 3.4 version does pull the latest)

hmmm

https://pypi.python.org/pypi/openpyxl

guess what they prob broke it - good thing we test for this

let's open an issue

cc @neirbowj

looks like openpyxl folks breaking things again 2.05 just came out
it's tested in. 3.4 build

have a look? (see link from above)

@jreback I'm just getting caught up here. Was out Beating the Blerch earlier. Will look into this.

thanks

it's now issue #8342
ENH: allow axis argument to append / move append code to generic.py (GH8295)
State: closed

closes #8295
- [x] Extract `append` method in `frame.py` into `generic.py` 
  - I left the original code just commented.
- [x] Extract test cases in `test_frame.py` into `test_generic.py` 
- [x] Extract test cases in `test_series.py` into `test_generic.py`
- [x] Specific test case for testing `axis=1` in `test_generic.py`
- [x] Generic method `append` in `generic.py`
- [x] `NotImeplementedError` for `Panel`
- [ ] Docstring for generic `append`
- [ ] Release note
- [ ] Documentation
- [ ] CI pass

One more thing, could `Series.append` use `axis=1` as well ?

``` python
pd.Series([1,2,3]).append(pd.Series(['a','b','c']), axis=1)

   0  1
0  1  a
1  2  b
2  3  c

# just like below
# pd.concat((pd.Series([1,2,3]), pd.Series(['a','b','c'])), axis=1)
```

It would be awesome if I could get your guidance to close this issue under the project's convention. Thank you ! 


Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: API Design

Comments:
@htkm the answer to your question is yes, `Series.append` CAN take `axis=1` to create a frame. Though that would need some separate tests.

I made some changes to follow your advice. And for generic `DataFrame.append` with `axis=1`, I found something interesting when I do the same thing by `pd.concat` with `axis=1`.

``` python
df = pd.DataFrame(np.random.randn(2,2), columns=list('AB'))
sr = pd.Series(range(2), index=[0,10], name='SR')

pd.concat((df, sr), axis=1)
           A         B  SR
0   0.376365 -0.830094   0
1   0.765837  1.001402 NaN
10       NaN       NaN   1

df.append(sr.to_frame(), axis=1)
           A         B  SR
0   0.376365 -0.830094   0
1   0.765837  1.001402 NaN
10       NaN       NaN   1

df.append(sr, axis=1)
           A         B   0  10   A   B   0  10
0   0.376365 -0.830094 NaN NaN NaN NaN NaN NaN
1   0.765837  1.001402 NaN NaN NaN NaN NaN NaN
SR       NaN       NaN NaN NaN NaN NaN   0   1
```

I think `df.append(sr, axis=1)` seems to return unexpected results. Do you think `append` method in `generic.py` should be modified to make it work like `pd.concat` ? Thank you.

@htkm I think you are right, the last `df.append(sr,axis=1)` should be equivalent to `df.append(sr.to_frame(),axis=1)`

can you rebase this on master and see where it is?

closing pls reopen if/when updated
TST: Fixed precision test error for read_csv
State: closed

This should hopefully fix #8324.
@jreback 


Labels:

Comments:
let me try

didn't fix. If want to setup a windows build, be my guest (though its a bit of a PITA): https://github.com/pydata/pandas/wiki/Building-On-Windows

Could skip on windows / change the check as well.

To me this is really strange, as I can't think of the reason why the converters should be at all platform-dependent (probably the same issue as in #8044). For the moment I amended this commit to a `<=` check, does this work for you?

sure that's fine

windows is just odd
it might be that a comparison is actually a float32 (rather than a float64) 

closed by https://github.com/pydata/pandas/commit/0005e0f78e6086b544257648c96e476708d7c3e1

just stuck it in, thanks !
ENH: add chunksize argument to read_sql (GH2908)
State: closed

Closes #2908 

Should probably add this also to `read_sql_table`. And try to reduce some repetitions (`_wrap_result` can maybe be a top-level (in the file) helper function).


Labels:
name: IO SQL
description: to_sql, read_sql, read_sql_query

Comments:
@jreback @artemyk Can you have a look? 
I should probably also implement it for `read_sql_table`

@jreback The implementation is not very clean (also some repetition in SQLTable and things like that), but I would like to get this in for 0.15 (functionality is in place, tests pass). And we can clean-up or refactor later if needed.

@jorisvandenbossche go for it. cleaning is for 0.15.1 !
BUG: to_clipboard output formatting
State: closed

closes  https://github.com/pydata/pandas/issues/8305

@jreback I made the simple patch as you suggested for fixing truncated columns, although problem seems to be https://github.com/pydata/pandas/blob/master/pandas/core/format.py#L2118 called from https://github.com/pydata/pandas/blob/master/pandas/core/format.py#L411
Perhaps `conf_max` should respect if caller passes `minimum` argument so it wont truncate?

Second issue, where `index=False` was ignored (or just any other argument when `excel=False`) was because arguments were not passed to underlying function `obj.to_string()` and that was easy fix.

I added test for checking if column width greater then `max_colwidth` passes clipboard test.


Labels:
name: Bug
name: Output-Formatting
description: __repr__ of pandas objects, to_string

Comments:
not sure I understand the first part of what you are saying?

> not sure I understand the first part of what you are saying?

you are right...

[L2118](https://github.com/pydata/pandas/blob/master/pandas/core/format.py#L2118) is where `to_string()` truncates to `max_colwidth`, but I wrongly assumed that `minimum` argument passed to `_make_fixed_width()` is the actual width of column value represented as string, while that value seems to deal with column header instead. I should have checked better...

merged via https://github.com/pydata/pandas/commit/8a169001b95f25095b619f5e0770fe15005909ee

thanks!
BUG: Exception raised when `to_sql` used to insert empty dataframe
State: closed

Right now, if inserting an empty table (no rows) using `to_sql`, the following exception is raised:

```
...
  File "/Users/artemy/dev/pandas/pandas/io/sql.py", line 1220, in to_sql
    table.insert(chunksize)
  File "/Users/artemy/dev/pandas/pandas/io/sql.py", line 659, in insert
    chunks = int(nrows / chunksize) + 1
ZeroDivisionError: division by zero
```

This fixes this bug and adds tests for it.


Labels:

Comments:
yep, thanks!
Allow preservation of object dtype when convert_numeric=True
State: closed


Labels:

Comments:
what's the utility here?

The current behavior hardly seems desirable, even though there are explicit tests for it:

```
In [26]: df = pd.DataFrame({'a': ['1', '2', '3'], 'b': ['a', 1, 'b']})

In [27]: df
Out[27]:
   a  b
0  1  a
1  2  1
2  3  b

In [28]: df.convert_objects(convert_numeric=True)
Out[28]:
   a   b
0  1 NaN
1  2   1
2  3 NaN

In [29]: df.convert_objects(convert_numeric=True).dtypes
Out[29]:
a      int64
b    float64
dtype: object
```

that's exactly the purpose
to clean a mixed series 
in effect it is a coerce a not-something to nan

but why does "clean" mean "clobber my data if it isn't numeric dtyped"?

well you have a mixed series 
if u are explicitly calling convert object and setting convert_numeric=True that is what you are asking

the reason for this is you read in data and it came out object dtype because the csv was off - so eliminate non-numbers

turned out this wasn't necessary
API: accept -1 for layout in plot functions
State: closed

Closes https://github.com/pydata/pandas/issues/8071#issuecomment-55906171

``` python
In [1]: df = pd.DataFrame(np.random.randn(20, 3))

In [2]: df.plot(subplots=True, layout=(-1, 2))
Out[2]: 
array([[<matplotlib.axes._subplots.AxesSubplot object at 0x116b134e0>,
        <matplotlib.axes._subplots.AxesSubplot object at 0x117819240>],
       [<matplotlib.axes._subplots.AxesSubplot object at 0x11786b6a0>,
        <matplotlib.axes._subplots.AxesSubplot object at 0x1178a5ba8>]], dtype=object)

```

![ex](https://cloud.githubusercontent.com/assets/1312546/4309486/b0f963f4-3e9c-11e4-965f-f0ed78c1412b.png)


Labels:
name: Visualization
description: plotting

Comments:
Just some python2 failures. I didn't realize `math.ceil` returned a float(!)

Ok all green now. Any comments @jreback @sinhrks?

@TomAugspurger ok then; pls add a release not in API (I don't think a whole example is needed, just a mention). merge at your discrection.

Thanks.

There was a build error on the first Travis job FYI. I restarted it.

All good: https://travis-ci.org/pydata/pandas/builds/35654530
ERR: DataFrame.from_records raises misleading exception on shape mismatch
State: closed

Currently, `DataFrame.from_records` raises a misleading exception when the shape of the input data do not match the shape implied by the axes:

``` python
In [1]: import pandas as pd

In [2]: pd.DataFrame.from_records([(0, 1), (0, 1), (0, 1)], index=[0, 1],
   ...:                           columns=['A', 'B'])
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-2-b8c920ac0f12> in <module>()
      1 pd.DataFrame.from_records([(0, 1), (0, 1), (0, 1)], index=[0, 1],
----> 2                           columns=['A', 'B'])

/Users/afni/homebrew/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pandas/core/frame.pyc in from_records(cls, data, index, exclude, columns, coerce_float, nrows)
    833 
    834         mgr = _arrays_to_mgr(arrays, arr_columns, result_index,
--> 835                              columns)
    836 
    837         return cls(mgr)

/Users/afni/homebrew/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pandas/core/frame.pyc in _arrays_to_mgr(arrays, arr_names, index, columns, dtype)
   4628     axes = [_ensure_index(columns), _ensure_index(index)]
   4629 
-> 4630     return create_block_manager_from_arrays(arrays, arr_names, axes)
   4631 
   4632 

/Users/afni/homebrew/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pandas/core/internals.pyc in create_block_manager_from_arrays(arrays, names, axes)
   3238         return mgr
   3239     except (ValueError) as e:
-> 3240         construction_error(len(arrays), arrays[0].shape[1:], axes, e)
   3241 
   3242 

/Users/afni/homebrew/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pandas/core/internals.pyc in construction_error(tot_items, block_shape, axes, e)
   3209         raise e
   3210     raise ValueError("Shape of passed values is {0}, indices imply {1}".format(
-> 3211         passed,implied))
   3212 
   3213 

ValueError: Shape of passed values is (2,), indices imply (2, 2)
```

This error message leads the user to believe that the error is internal (after all, the input data were _definitely_ 2-dimensional) and hides the true error, which is that the shape of the passed values is actually `(2, 3)`.

For some reason, this behavior was explicitly specified by a test.


Labels:
name: Error Reporting
description: Incorrect or improved errors from pandas

Comments:
pseudo related to #4464 

It looks to me more like these are totally unrelated issues in the same general set of systems.

Also, the test failure looks like bad math on py3.4, rather than anything to do with this patch.

@ischwabacher that's why I said _pseudo_ related :)

hmm, are you rebased on master? never saw those fail before

also pls squash

I made this fix on the GitHub site, so it should be directly from master.  I'll squash tonight.

Good to go.

thanks
BUG: DataFrame.to_html() fails when index=False and max_rows is specified
State: closed

This addresses #8273, but it doesn't actually fully fix the issue, since the same bug still occurs when we have a MultiIndex. I've got a fix for the single-index case, and I'm not sure if I have time to decipher what's happening with hierarchical indexes right now. I'll open a new issue for the MultiIndex case if that's OK, having a MultiIndexed dataframe and then not printing the index seems like a bit of an edge case to me anyway.

``` python
import pandas as pd
import numpy as np

df = pd.DataFrame({'a': np.arange(0, 10), 'b': np.arange(9, -1, -1)})
df.to_html(max_rows=3, index=False)
```

now produces:

<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th>a</th>
      <th>b</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td> 0</td>
      <td> 9</td>
    </tr>
    <tr>
      <td>...</td>
      <td>...</td>
    </tr>
    <tr>
      <td> 9</td>
      <td> 0</td>
    </tr>
  </tbody>
</table>


as expected.


Labels:
name: Output-Formatting
description: __repr__ of pandas objects, to_string
name: IO HTML
description: read_html, to_html, Styler.apply, Styler.applymap

Comments:
closing pls reopen if/when updated