BrandonFors/pandas-github-issues huggingface dataset:
BUG: to_latex() output broken when the index has a name (GH10660)
State: closed

closes #10660

first try at contributing to pandas. feedbacks will be greatly appreciated.


Labels:
name: Bug
name: Output-Formatting
description: __repr__ of pandas objects, to_string
name: IO LaTeX
description: to_latex

Comments:
looks good. pls squash.

thank you sir!
DOC: Add note about unicode layout
State: closed

Follow-up for #11102. Added note because doc output may look incorrect depending on font.


Labels:
name: Docs
name: Output-Formatting
description: __repr__ of pandas objects, to_string

Comments:
hmm maybe better to put this in a code block which has a mono spaced font?

OK, let me check code block results.

@sinhrks ?

@sinhrks ?

I may misunderstand something, but code block doesn't use monospaced font always? I tried both `::` and `.. code-block::`, but the results look same.

#### Markdown

```
::

    >>> df = pd.DataFrame({'a': ['xxx', u'¡¡'], 'b': ['yyy', u'¡¡']})
    >>> df
         a    b
    0  xxx  yyy
    1   ¡¡   ¡¡

.. code-block:: python

   >>> df = pd.DataFrame({'a': ['xxx', u'¡¡'], 'b': ['yyy', u'¡¡']})
   >>> df
        a    b
   0  xxx  yyy
   1   ¡¡   ¡¡
```

#### Output

![2015-10-06 21 11 45](https://cloud.githubusercontent.com/assets/1696302/10308208/17adb702-6c6f-11e5-9b29-83380359bdbb.png)

Do you know any way to force all characters to be monospaced? Alternative idea is attach the screenshot of the console...

CC: @jorisvandenbossche 

Hmm, don't think I can help here. So the 'easy' solution is indeed to use a picture that shows the effect (I think that is better than the note in any case, as the user can then visually see what is meant)

~~Some possible things: the output during the doc built is always decoded to utf-8 in a hack in the ipython directive (https://github.com/pydata/pandas/blob/master/doc/sphinxext/ipython_sphinxext/ipython_directive.py#L255)~~ -> your example above does not use the ipython directive 
It happens with the ambiguous width characters? And you needed to change something (the previous PR) to ensure the console output shows correcty? So maybe a similar issue is the case here with the sphinx directive

@jorisvandenbossche Thanks to confirm. The screen capture using ipython directive is attached on the bottom of #11102. Affected characters seems to depend on environment. At a quick glance, CSS doesn't specify monospaced font for highlight blocks.

As a workaround, updated to attach screen captures of each output as below.

![2015-10-07 23 11 07](https://cloud.githubusercontent.com/assets/1696302/10339931/b683acf4-6d48-11e5-89db-3cdc17b17c4f.png)

thanks @sinhrks
squeeze works on 0 length arrays
State: closed

fixes: https://github.com/pydata/pandas/issues/11229.
fixes: https://github.com/pydata/pandas/issues/8999.

Also a better implementation that avoids `ix`

Should I add to What's new for 0.17? Or is that closed now?


Labels:
name: Bug
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
@jreback tests added

What should the behavior be here:

``` python
empty_series=pd.Series([], name='five').squeeze()
```

numpy returns itself:

``` python
In [18]: pd.np.empty(0)
Out[18]: array([], dtype=float64)

In [19]: pd.np.empty(0).squeeze()
Out[19]: array([], dtype=float64)
```

...which is the current pandas implementation too.
But I could see `None` being acceptable here

no squeeze does not return a scalar always a Series (or high dim)

OK @jreback, test added for that too

Also closes https://github.com/pydata/pandas/issues/8999

their is a whatsnew for 0.17.1 pls add a release note.

@jreback what's new added, commits squashed, ready to go from this end

@jreback done!

merged via https://github.com/pydata/pandas/commit/aa94ae4d93f564a5dbba03255a0dfa5c73efb0e9

thanks!
defer or remove loading of openhub in docs
State: closed

Their service was slowing down the loading of our docs. I've added a defer tag so that our docs load first.

Or we can delete that entirely, anyone know why it's in there to begin with?


Labels:
name: Docs

Comments:
sure, no idea what that does?

http://blog.openhub.net/about/

@jreback if you're cool with just removing it altogether I think we should.

I also got rid of an unclosed `<p>` tag, which didn't seem to break everything.

sure let's try this, can always revert if necessary.
Use set literal syntax for set prettyprinting
State: closed


Labels:
name: Output-Formatting
description: __repr__ of pandas objects, to_string

Comments:
pls add a test in `tests/test_format.py` in `test_pprint`

pls add a whatsnew note in 0.17.1 as well

can you add a whatsnew note (api changes section)

done

can you add a test as well

`tests/test_format.py` in `test_pprint`

done

awesome. ping when green.

ping!

merged via https://github.com/pydata/pandas/commit/76d4c49a10b2606458d2622a05bb14aea578e9a5

thanks!
PERF: vectorized DateOffset with months
State: closed

This is a follow-up to https://github.com/pydata/pandas/pull/10744.  In that, vectorized versions of some offsets were implemented, mostly by changing to periods and back.  

The case of shifting by years/months (which is actually most useful to me) required some extra hoops and had poorer performance - this PR implements a special cython routine for that, for about a 10x improvement. 

```
In [3]: s = pd.Series(pd.date_range('1900-1-1', periods=100000))

# Master
In [4]: %timeit s + pd.DateOffset(months=1)
1 loops, best of 3: 140 ms per loop

# PR
In [2]: %timeit s + pd.DateOffset(months=1)
100 loops, best of 3: 14.2 ms per loo
```


Labels:
name: Performance
description: Memory or execution speed performance
name: Frequency
description: DateOffsets

Comments:
@jreback - pushed changes for your comments and added `MonthEnd` and `MonthBegin`

For your comment about using this routine in the `DateOffset` are you talking about cythonizing the actual `apply` method too?  On that front, I see there is an `offsets.pyx` file, but doesn't look it's actually used?

- do we have an asv benchmark for this? if not pls add one. otherwise minor change. lmk
- add to the perf section (e.g. just add this issue on in the whatsnew next to #10744) 

Made those doc changes.  Yeah,  there is an `asv` for `DateOffset`

```
    before     after       ratio
  [8ea8968 ] [c1d53f9 ]
-  121.83ms    18.49ms      0.15 timeseries.timeseries_datetimeindex_offset_fast.time_timeseries_datetimeindex_offset_fast
-  121.50ms    21.70ms      0.18  timeseries.timeseries_series_offset_fast.time_timeseries_series_offset_fast
```

@chris-b1 thanks! these pr's are awesome! keep em coming!
DOC: XportReader not top-level API
State: closed

@kshedden `XportReader` is not top-level API, so it was erroring in the doc build.

I now removed it from api.rst (the `TextFileReader` what you get back with chunksize for read_csv is also not included), but I can also adapt it to use `.. currentmodule:: pandas.io.sas` so it builds correctly if you want.


Labels:
name: Docs

Comments:
this is fine
API: raise on header=bool in parsers
State: closed

closes #6113

Passing `header=True|False` to `read_csv` (and brethren), `read_excel`, and `read_html` will now  raise a `TypeError`, rather than being coerced to an int.

I had thought about converting `False` to `None` but thought that could break someone's code in a very subtle way if they were somehow depending on the old behavior.  But happy to change that if it seems better.

If you want to push this in for 0.17 I can add a doc-note or this could easily wait.


Labels:
name: API Design
name: IO CSV
description: read_csv, to_csv

Comments:
@chris-b1 if you add a release note (put in the API section) can add for 0.17.0

@jreback added a doc note and testing for the True case.

tiny doc-change. ping when you push

@jreback thanks, made that change

thank you sir!

Nice!
improves groupby.get_group_index when shape is a long sequence
State: closed

closes #10161 

xref https://github.com/pydata/pandas/issues/10161#issuecomment-142701582

``` python
In [5]: df = DataFrame(np.random.randn(5000, 100).astype(str))

In [6]: %timeit df.duplicated()
1 loops, best of 3: 151 ms per loop

In [7]: %timeit df.T.duplicated()
1 loops, best of 3: 1.39 s per loop
```

part of this is because of taking the transpose (maybe cache locality). i.e. below performs better even though the shape is the same as `df.T` in above:

``` python
In [8]: df = DataFrame(np.random.randn(100, 5000).astype(str))

In [9]: %timeit df.duplicated()
1 loops, best of 3: 965 ms per loop
```


Labels:
name: Performance
description: Memory or execution speed performance

Comments:
are there asv benches for this?

can you add a doc-note in the performance section as well. thxs.

I tested this fix on the same dataframe and it looks like it solves the problem

``` python
In [8]: df.info()
<class 'pandas.core.frame.DataFrame'>
Int64Index: 5000 entries, 0 to 4999
Data columns (total 35 columns):
...
dtypes: float64(7), int64(12), object(16)
memory usage: 1.4+ MB


In [9]: %timeit -n 3 df.T.duplicated()
3 loops, best of 3: 549 ms per loop
```

There is still a slight regression from 0.12.0 but it is minimal.  Thanks for fixing this.

there already is a [`frame_duplicated`](https://github.com/pydata/pandas/blob/38ee8c701af98b9015a492096d4f8a6eae972cd6/asv_bench/benchmarks/frame_methods.py#L284) asv benchmark.

added the doc note.

@behzadnouri maybe add to that benchmark a case with the tranposed frame? (to catch this case with many columns)

merged via https://github.com/pydata/pandas/commit/3fb802a7f5b849385284c3ae3ce9c53ef27e7d6c

thanks!
DOC: header=None in SAS docs
State: closed

Fixed a typo where I had `header=False`

Side note, it might be nice to change the actual behavior, not the first time it's tripped me up.  I guess there already is an issue, #6113


Labels:
name: Docs

Comments:
Thanks!

And that is indeed a rather confusing feature. If we you want to push for #6113, certainly welcome!
ENH: Restore original convert_objects and add _convert
State: closed

Restores the v0.16 behavior of convert_objects and moves the new
version of _convert
Adds to_numeric for directly converting numeric data

closes #11116
closes #11133


Labels:
name: Dtype Conversions
description: Unexpected or buggy dtype conversions

Comments:
Still a few things to do:
-  [x] Update release notes to remove new docs and add deprecation note
-  [x] Make sure `copy` is passed to `_possibly_convert_objects`
-  [x] Add note about bugs fixed in `convert_objects`
-  [x] Restore docs for `convert_objects`
-  [x] Add tests for `to_numeric`
-  [ ] Add docs for `to_numeric`
-  [x] Move converters to a new file, `convert.py`

The important changes are:

`convert` in `ObjectManager`: https://github.com/pydata/pandas/pull/11173/files#diff-e705e723b2d6e7c0e2a0443f80916abfR1521

This change requires this to 'route' to either the old or new style converter. 

`to_numeric`: https://github.com/pydata/pandas/pull/11173/files#diff-40d1f19921aa9f2932951124fad07cc6R53

New function.

@bashtage Really thanks a lot for this! 

(I am still not fully convinced the deprecation is needed, but I will try to summarize my reasons tomorrow)

@bashtage 
- pls move the 2 converting functions either to a new module, `core/convert.py`. 
- let's finish up the documentation changes.

pls let me know asap.

Still no tests for `to_numeric` or docs for this method.

looks good! ping when tests for `pd.to_numeric`

@jorisvandenbossche ok with this

`to_numeric` needs some fixes (and tests): `lib` is not yet imported, `lib.maybe_convert_numeric` expects an array so it bugs on a series as arg, and the final result needs to be boxed into a series again if the arg was a series.

Yes - it definitely needs work.  It expects something with the buffer
interface, so I need to ensure that this works as expected.

On Sun, Sep 27, 2015 at 8:12 PM Joris Van den Bossche <
notifications@github.com> wrote:

> to_numeric needs some fixes (and tests): lib is not yet imported,
> lib.maybe_convert_numeric expects an array so it bugs on a series as arg,
> and the final result needs to be boxed into a series again if the arg was a
> series.
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/pydata/pandas/pull/11173#issuecomment-143606211.

@bashtage did you have a chance to add some tests here?

Should be ready when green I think

Can squash commits - wanted to make it easier to see changes at each stage since close to release

@jreback Green - let me know if I missed anything.  Only tick left is to add some docs although this can probably wait since `convert_objects` is still available.

All addressed

@jreback It is green - or should be since all required tests passed.

@bashtage thanks for all of the work on this!
ENH: make Series.ptp ignore NaN values (GH11163)
State: closed

Closes [GH11163](https://github.com/pydata/pandas/issues/11163).

`Series.ptp` will now ignore missing values when calculating the peak to peak difference.


Labels:
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: Numeric Operations
description: Arithmetic, Comparison, and Logical operations

Comments:
actually what I would is define this using `generic.py/_make_stat_function` (e.g. how `.min` and such are defined). though this would only apply to `Series` (so might need to move a couple of things around to deal with this).

Ah OK - no problem, I should be able to take a look at this and update in the next week or so.

can you update

Apologies - the past few weeks have been much busier than I'd anticipated. I hope to get back to update this very soon.

no prob.

@ajcr progress?

@ajcr pls update when you can

@jreback: my apologies for the delay once again, I've just got back and was  looking at this again just now.

As I understand it, I need to do the following:
- make a `nanptp` function in nanops.py (is implementing this as `nanmax - nanmin` sufficient?)
- add an entry for `ptp` to `_make_stat_function` in generic.py
- make sure that this doesn't become a method of DataFrame - i.e. retrict it to Series

The last part I'm not quite sure how to do best, because NDFrame is a parent class of both Series and DataFrame. Are there any other methods from NDFrame that are restricted to one or the other object in this way?

you don't really need to do step 1, simply define it in `_make_stat_function` to do this (as its sort of a trivial function)

if you look at the end of `core/series.py` there is a like `Series._add_numeric_operations()`, (and in `DataFrame`), so you just need to tag certain of these functions to be added only on certain classes. 

@jreback: made some changes and things are now green - let me know if you'd like me to improve anything or make any other changes.

thanks!

No problem (sorry it took a while) - thanks for your guidance!
Added density of a categorical #10949
State: closed

Not sure if this is the correct approach to the problem 

closes #10949.

 Assuming it is I will go ahead and add some documentation for this enhancement. Let me know if you have any feedback.


Labels:
name: Enhancement
name: Categorical
description: Categorical Data Type

Comments:
Great!

I know @jreback suggested to multiply by 100, but is that right? Shouldn't we let the user convert to percent? (a more general point than this PR)

I'm not sure you want integer division there.

I would test for an actual value, rather than just the code that you wrote

Happy to change this to a proportion, I agree that it seems a more sensible default. 

With regards to integer division, what is the preferred way to target Python 2 and 3 with float division, `from __future__ import division`?

Will update the tests in the next commit.

I'm not sure on the pandas standard for the division - the maintainers will opine - either `__future__` or coercing with `float(x)` would work.

And they'll also ask for a note in what's new & the docs.

Cheers!

further, can you do a tiny bit of research to see what `R` calls this (they call the `Categorical` a `Factor`.

I haven't had much luck searching for an equivalent field in R but I have found that the SAS community calls this number the cardinality ratio: http://www.sascommunity.org/wiki/Cardinality_Ratio

Should we consider preempting or handling `ZeroDivisionError`?

whoosh never realized we allow 0 categories!. Yes that should be a `NaN`

pls add a release note in 0.17.1 (enhancements)
add to api.rst as well

can you update

Apologies for the delay, I was trying to understand out the correct way to add documentation for this feature before other projects stole me away from this fix. 

I've made an updated commit. There remains one failing test due to the recommendation to switch from `tm.assert_numpy_array_equal(np.nan, s2.cat.density)` to `self.assertEqual(np.nan, s2.cat.density)`. Please advise on the best way to approach this failing test.

use self.assertTrue(... is np.nan)

closing as we closed the original issue #10949 

let's see an actual usecase (as I proposed the original issue, and don't really have one :)
PERF: Series.dropna with non-nan dtype blocks
State: closed

Minor improvement when `dropna` actually does nothing. If OK for 0.17, I'll add a release note.

```
import numpy as np
import pandas as pd

s = pd.Series(np.random.randint(0, 1000, 50000000))

# before
%timeit s.dropna()
#1 loops, best of 3: 1.58 s per loop

# after
%timeit s.dropna()
#1 loops, best of 3: 568 ms per loop
```


Labels:
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: Performance
description: Memory or execution speed performance

Comments:
sure, pls add a note in 0.17.1 whatsnew

@sinhrks looks good, can you rebase / add a release note 

Rebased, and added release note. Followings are asv results added in this PR.

```
    before     after       ratio
  [a89b96d4] [b2cb56a8]
    10.76ms    10.60ms      0.99  series_methods.series_dropna_datetime.time_series_dropna_datetime
-   12.07ms   726.61μs      0.06  series_methods.series_dropna_int64.time_series_dropna_int64
```

thanks!
PERF: nested dict DataFrame construction
State: closed

Part of #11084

http://pydata.github.io/pandas/#frame_ctor.frame_ctor_nested_dict_int64.time_frame_ctor_nested_dict_int64

```
    before     after       ratio
   [d28fd70 ] [48c588  ]
-  325.06ms    72.25ms      0.22  frame_ctor.frame_ctor_nested_dict.time_frame_ctor_nested_dict
-  364.35ms   106.67ms      0.29  frame_ctor.frame_ctor_nested_dict_int64.time_frame_ctor_nested_dict_int64
```

Two changes to get back to the old performance:
- skip attempting to box `Timedeltas`/`Timestamps` if not a `DatetimeIndex`/`TimedeltaIndex`
- only convert index to object once


Labels:
name: Performance
description: Memory or execution speed performance
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Regression
description: Functionality that used to work in a prior pandas version

Comments:
ping when green (and tests are moved)

@jreback - moved the `Period` test as suggested, travis is green. Thanks.

thanks@
(WIP) BUG: DatetimeTZBlock.fillna raises TypeError
State: closed

```
import pandas as pd
idx = pd.DatetimeIndex(['2011-01-01', pd.NaT, '2011-01-03'], tz='Asia/Tokyo')
s = pd.Series(idx)
s
#0   2011-01-01 00:00:00+09:00
#1                         NaT
#2   2011-01-03 00:00:00+09:00
# dtype: datetime64[ns, Asia/Tokyo]

# NG, the result must be DatetimeTZBlock
s.fillna(pd.Timestamp('2011-01-02', tz='Asia/Tokyo'))
# TypeError: putmask() argument 1 must be numpy.ndarray, not DatetimeIndex

# NG, it should be object dtype, as the same as when Series creation with mixed tz
s.fillna(pd.Timestamp('2011-01-02'))
# TypeError: putmask() argument 1 must be numpy.ndarray, not DatetimeIndex
```

Also, found existing `DatetimeBlock` doesn't handle tz properly. Closes #7095.

```
idx = pd.DatetimeIndex(['2011-01-01', pd.NaT, '2011-01-03'])
s = pd.Series(idx)

# OK, result must be DatetimeBlock
s.fillna(pd.Timestamp('2011-01-02'))
#0   2011-01-01
#1   2011-01-02
#2   2011-01-03
# dtype: datetime64[ns]

# NG, it should be object dtype, as the same as when Series creation with mixed tz
s.fillna(pd.Timestamp('2011-01-02', tz='Asia/Tokyo'))
#0   2011-01-01 00:00:00
#1   2011-01-01 15:00:00
#2   2011-01-03 00:00:00
# dtype: datetime64[ns]

# NG, unable to fill different dtypes
s.fillna('AAA')
# ValueError: Error parsing datetime string "AAA" at position 0
```


Labels:
name: Bug
name: Datetime
description: Datetime data dtype
name: Timezones
description: Timezone data dtype
name: Blocker
description: Blocking issue or pull request for an upcoming release

Comments:
this is #7095, yes?

Correct, this should fix #7095. I'll add test case for this. 

Also, I found current impl can't handle `inplace` option properly. `inplace` should be effective if target contains no NaN values in case of `DatetimeTZBlock`.

thanks @sinhrks nice catch
PERF: improves performance in groupby.size
State: closed

on master:

``` python
In [1]: n = 100000

In [2]: np.random.seed(2718281)

In [3]: dr = date_range('2015-09-19', periods=n, freq='T')

In [4]: ts = Series(np.random.choice(n, n), index=np.random.choice(dr, n))

In [5]: %timeit ts.resample('5T', how='size')
1 loops, best of 3: 538 ms per loop
```

on branch:

``` python
In [5]: %timeit ts.resample('5T', how='size')
10 loops, best of 3: 27.6 ms per loop
```


Labels:
name: Groupby
name: Performance
description: Memory or execution speed performance

Comments:
thanks!
BUG: axis kw not propogating on pct_change
State: closed

axis option was not correctly set on fillna in pct_change


Labels:
name: Bug
name: Numeric Operations
description: Arithmetic, Comparison, and Logical operations

Comments:
Thanks. Can you add a test for this?

@Tux1 pls add a test for this

Sorry. It is my first commit.
I made a simple test that shows the issue and my commit fixed it.
Is it okay ?
Thanks

pls see the contributing docs: http://pandas.pydata.org/pandas-docs/stable/contributing.html

tests / fix should be on a single pr.

Sorry... Moreover the test was wrong. 
I have made another test which is passed with my patch. That is committed in a single branch, this one.

pls add a whatsnew note for 0.17.1 (bug fix), use this PR number

@Tux1 can you rebase / update

Sorry for several commits. I think that is okay now ? whatsnew edited, commit rebased

some comments, pls squash when finished

pls rebase on master / squash, see docs [here](http://pandas.pydata.org/pandas-docs/stable/contributing.html#contributing-your-changes-to-pandas)

Sorry, my repo is a mess... I will be better next time

merged via https://github.com/pydata/pandas/commit/334b07641a1dc8c1e7390d01184e114ddde0a7da

thanks!