BrandonFors/pandas-github-issues huggingface dataset:
TST/BUG: DataFrame truncated repr with DatetimeTz and NaT column
State: closed

- [x] closes #12211
- [x] tests added / passed
- [x] passes `git diff upstream/master | flake8 --diff`


Labels:
name: Bug
name: Testing
description: pandas testing functions or related to the test suite
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Timezones
description: Timezone data dtype

Comments:
## [Current coverage](https://codecov.io/gh/pydata/pandas/pull/13039?src=pr) is **84.06%**

> Merging [#13039](https://codecov.io/gh/pydata/pandas/pull/13039?src=pr) into [master](https://codecov.io/gh/pydata/pandas/branch/master?src=pr) will decrease coverage by **-0.00%**

``` diff
@@             master     #13039   diff @@
==========================================
  Files           136        136          
  Lines         49994      49994          
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
- Hits          42028      42027     -1   
- Misses         7966       7967     +1   
  Partials          0          0          
```
1. File `...das/tseries/index.py` (not in diff) was modified. [more](https://codecov.io/gh/pydata/pandas/commit/44d2e2b132b837cdd5f2062aadb55ccaef6689c4/changes?src=pr#70616E6461732F747365726965732F696E6465782E7079) 
   - Misses `+1` 
   - Partials `0` 
   - Hits `-1`

> Powered by [Codecov](https://codecov.io?src=pr). Last updated by [84725fa...44d2e2b](https://codecov.io/gh/pydata/pandas/compare/84725fa936b4136657f5ea02c588790de6866899...44d2e2b132b837cdd5f2062aadb55ccaef6689c4)
TST/BUG: Categorical dtypes cause error when attempting stacked bar plot
State: closed

- [x] closes #13019
- [x] tests added / passed
- [x] passes `git diff upstream/master | flake8 --diff`


Labels:
name: Bug
name: Testing
description: pandas testing functions or related to the test suite
name: Visualization
description: plotting

Comments:
## [Current coverage](https://codecov.io/gh/pydata/pandas/pull/13038?src=pr) is **84.07%**

> Merging [#13038](https://codecov.io/gh/pydata/pandas/pull/13038?src=pr) into [master](https://codecov.io/gh/pydata/pandas/branch/master?src=pr) will not change coverage

``` diff
@@             master     #13038   diff @@
==========================================
  Files           136        136          
  Lines         49994      49994          
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
  Hits          42028      42028          
  Misses         7966       7966          
  Partials          0          0          
```

> Powered by [Codecov](https://codecov.io?src=pr). Last updated by [84725fa...dc57a6c](https://codecov.io/gh/pydata/pandas/compare/84725fa936b4136657f5ea02c588790de6866899...dc57a6c80b26d440381a08e92ffaae24568315f0)

👍  Not sure if we want to add a release note explicitly stating this was fixed (even if we don't know exactly what PR fixed it)

thanks!

whatsnew added here: https://github.com/pydata/pandas/commit/db35ff455227d1b705509966740d3d6607faa3d9
BUG: subclassed .align returns normal DataFrame
State: closed

- [x] closes #12983
- [x] tests added / passed
- [x] passes `git diff upstream/master | flake8 --diff`
- [ ] whatsnew entry

Also fixed a bug which `Series.align(DataFrame)` raises `AttributeError`.

```
pd.Series([1, 2], index=['a', 'b']).align(pd.DataFrame([1, 2], index=['a', 'c']))
# AttributeError: 'Series' object has no attribute 'columns'
```


Labels:
name: Bug
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Compat
description: pandas objects compatability with Numpy or Python functions

Comments:
## [Current coverage](https://codecov.io/gh/pydata/pandas/pull/13037?src=pr) is **84.07%**

> Merging [#13037](https://codecov.io/gh/pydata/pandas/pull/13037?src=pr) into [master](https://codecov.io/gh/pydata/pandas/branch/master?src=pr) will increase coverage by **+<.01%**

``` diff
@@             master     #13037   diff @@
==========================================
  Files           136        136          
  Lines         49994      49999     +5   
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
+ Hits          42028      42033     +5   
  Misses         7966       7966          
  Partials          0          0          
```

> Powered by [Codecov](https://codecov.io?src=pr). Last updated by [84725fa...32ced80](https://codecov.io/gh/pydata/pandas/compare/84725fa936b4136657f5ea02c588790de6866899...32ced80e3c3108da3e3ab89f1b961ee54df33323)

thanks @sinhrks
BUG: to_datetime when called with a unit and coerce is buggy
State: closed

closes #11758


Labels:
name: Datetime
description: Datetime data dtype
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate

Comments:
@jorisvandenbossche straightforward but lots of cases. So actually what I need is a bunch more tests. 

Stuff like mixed float/int/string (that are integer/float-like) with some NaT's thrown in (these should pass).
BUG/TST: Calling shift on a DatetimeIndex of length 0 returns an Index in…
State: closed

- [x] closes #9903 
- [x] tests added / passed
- [x] passes `git diff upstream/master | flake8 --diff`
- ~~whatsnew entry~~

No what's new as the fixed timing is unclear.


Labels:
name: Bug
name: Testing
description: pandas testing functions or related to the test suite
name: Datetime
description: Datetime data dtype
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves

Comments:
:+1: 

thanks @sinhrks
BUG:plot.bar misalignment when width=1
State: closed

- [x] closes #12979
- [x] tests added / passed
- [x] passes `git diff upstream/master | flake8 --diff`
- [x] whatsnew entry

Enabled true division. It also prevents differences between py2 / py3.


Labels:
name: Bug
name: Visualization
description: plotting

Comments:
lgtm. pls add a whatsnew note 0.18.1

lgtm. ping on green.

OK, added the release note and now green.

thank you sir!
Update truncate() documentation and testcases
State: closed

- [x] closes #11382 
- [x] tests added / passed
- [x] passes `git diff upstream/master | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
I'm not sure what to make of those codecov failures ...

just do the big selection across the 0th dimension 

The actual tests were OK (although I seen OutOfMemory exceptions ...), but the `_construct` wasn't.

For `Series/DataFrame` `self._construct(,index=,...)` vs for `Panel/Panel4D` `self._construct(,major_axis=,minor_axis=,...)` In this test I have to have the particular index.

How do you merge these?

Are you OK with this being under `Series`?

you need to make this generic, so no

@jreback Please offer some snippets on how you see this being make generic (I described what I ran into in the previous post). Thanks

@jreback How do you see this moving forward? Thanks

so in `test_generic.py` you can construct something generic like this:

```
shape = [int(1e6)] + ([1] * len(self._ndim-1)]
obj = self._construct(shape)
```

so will give you the first dim of 1e6 and the others 1

@jreback Thank you. I made the changes, the 2e6 takes a long time to run locally (in Panel) ...

Timing out in travis-ci too `No output has been received in the last 10m0s, this potentially indicates a stalled build or something wrong with the build itself.`

thanks @gliptak 

the panel comparator was doing things a very slow way. all fixed up.

@jreback Thank you for fixing Panel comparator.
BUG: .isin(...) now coerces sets to lists
State: closed

- [ ] closes #12988
- [ ] tests added, passed
- [ ] passes `git diff upstream/master | flake8 --diff`
- [ ] `comp.isin(values)` now coerces `values` from a set into a list. This was failing when `comp` is a datetime dtype and `values` is a set, where `.isin` will try to run `to_datetime(values)` on the unordered set and fail.


Labels:
name: Bug
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves

Comments:
tests!

I swear `git add` is playing games with me... sorry about that.

How to kill an underway Travis job?

## [Current coverage](https://codecov.io/gh/pydata/pandas/pull/13014?src=pr) is **83.91%**

> Merging [#13014](https://codecov.io/gh/pydata/pandas/pull/13014?src=pr) into [master](https://codecov.io/gh/pydata/pandas/branch/master?src=pr) will increase coverage by **+<.01%**

``` diff
@@             master     #13014   diff @@
==========================================
  Files           136        136          
  Lines         49908      49920    +12   
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
+ Hits          41876      41888    +12   
  Misses         8032       8032          
  Partials          0          0          
```

> Powered by [Codecov](https://codecov.io?src=pr). Last updated by 0294c48

@ch41rmn gr8. can you add a whatsnew note in bug fixes. ping when pushed.

looks good. can you add a whatsnew note. ping when pushed.

thanks @ch41rmn I merged this. hoping to get the release out shortly.

Thanks for merging @jreback. Apologies I didn't get to adding the whatsnew note in time, have been away for the weekend.
API, DOC: Clarify and Enforce Array to be 1-D during SparseArray Init Construction
State: closed

Title is self-explanatory.  Note that this behaviour was already implicitly enforced, but this makes the enforcement explicit.  Closes #12794.


Labels:
name: Sparse
description: Sparse Data Type
name: Compat
description: pandas objects compatability with Numpy or Python functions

Comments:
ping when green.

@jreback : Travis has given its green light of approval.  Ready to merge if there is nothing else.

thanks!
COMPAT: some compatibility fixes with new numpies
State: closed


Labels:
name: Compat
description: pandas objects compatability with Numpy or Python functions

Comments:
@tacaswell @mdboom, using mpl 1.5.0 here. I was turning on deprecation warnings. (we get a bunch of these in test suite), known issue?

cc @TomAugspurger 

```
======================================================================
ERROR: test_ts_area_lim (pandas.tests.test_graphics.TestSeriesPlots)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "pandas/tests/test_graphics.py", line 612, in test_ts_area_lim
    ax = self.ts.plot.area(stacked=False, x_compat=True)
  File "pandas/tools/plotting.py", line 3688, in area
    return self(kind='area', **kwds)
  File "pandas/tools/plotting.py", line 3564, in __call__
    **kwds)
  File "pandas/tools/plotting.py", line 2643, in plot_series
    **kwds)
  File "pandas/tools/plotting.py", line 2439, in _plot
    plot_obj.generate()
  File "pandas/tools/plotting.py", line 1032, in generate
    self._post_plot_logic_common(ax, self.data)
  File "pandas/tools/plotting.py", line 1160, in _post_plot_logic_common
    fontsize=self.fontsize)
  File "pandas/tools/plotting.py", line 1208, in _apply_axis_properties
    labels = axis.get_majorticklabels() + axis.get_minorticklabels()
  File "/Users/jreback/miniconda/lib/python2.7/site-packages/matplotlib/axis.py", line 1161, in get_majorticklabels
    ticks = self.get_major_ticks()
  File "/Users/jreback/miniconda/lib/python2.7/site-packages/matplotlib/axis.py", line 1290, in get_major_ticks
    numticks = len(self.get_major_locator()())
  File "/Users/jreback/miniconda/lib/python2.7/site-packages/matplotlib/dates.py", line 1007, in __call__
    return self._locator()
  File "/Users/jreback/miniconda/lib/python2.7/site-packages/matplotlib/dates.py", line 813, in __call__
    return self.tick_values(dmin, dmax)
  File "/Users/jreback/miniconda/lib/python2.7/site-packages/matplotlib/dates.py", line 830, in tick_values
    self.rule.set(dtstart=start, until=stop, count=self.MAXTICKS + 1)
  File "/Users/jreback/miniconda/lib/python2.7/site-packages/matplotlib/dates.py", line 724, in set
    self._rrule = rrule(**self._construct)
  File "/Users/jreback/miniconda/lib/python2.7/site-packages/dateutil/rrule.py", line 470, in __init__
    "raise an error.", DeprecationWarning)
DeprecationWarning: Using both 'count' and 'until' is inconsistent with RFC 2445 and has been deprecated in dateutil. Future versions will raise an error.
```

Thanks for pointing out.  Not a known issue (to me at least).  I'll see if I can reproduce locally.

@mdboom this is using pretty recently released `python-dateutil 2.5.3` FYI (might be new there)

There is a pr fixing this, but I am failing to find it.

https://github.com/matplotlib/matplotlib/issues/6155 / https://github.com/matplotlib/matplotlib/pull/6190

Thanks @tacaswell.  Unfortunately, it doesn't look there's much of a way to workaround that from the outside...

the fix will be it 1.5.2 / 2.0, which ever comes first

thanks @mdboom @tacaswell no worries. just trying to have clean test outputs.
PERF: Improved performance for .str.encode/decode
State: closed

I need such a patch to read huge `sas` tables encoded in `cp1251`. I'm not experienced enough to determine if such a patch is really needed here, but well.. it gives some speed in certain situations.

Optimize string encoding-decoding, leave default implementation for CPython optimized encodings,
(see https://docs.python.org/3.4/library/codecs.html#standard-encodings)

string

```
import pandas as pd
s1 = pd.Series(pd.util.testing.makeStringIndex(k=100000)).astype('category')
encs = 'utf-8', 'utf-16', 'utf-32', 'latin1', 'iso-8859-1', 'mbcs', 'ascii', 'cp1251', 'cp1252'
for enc in encs:
    s2 = s1.str.encode(enc).astype('category')
    print(enc)
    %timeit s1.str.encode(enc)
    %timeit s2.str.decode(enc)
```

unicode

```
import pandas as pd
s1 = pd.Series(pd.util.testing.makeUnicodeIndex(k=100000)).astype('category')
encs = 'utf-8', 'utf-16', 'utf-32'
for enc in encs:
    s2 = s1.str.encode(enc).astype('category')
    print(enc)
    %timeit s1.str.encode(enc)
    %timeit s2.str.decode(enc)
```

![image](https://cloud.githubusercontent.com/assets/53390/14850902/d7bf3ea2-0c84-11e6-88a8-fbdfc0f6b36f.png)
("10 loops, best of 3: xxx ms per loop")


Labels:
name: Performance
description: Memory or execution speed performance
name: Strings
description: String extension data type and string data

Comments:
@Winand in your example you miss the point of categorical encoding:

```
In [2]: pd.util.testing.makeUnicodeIndex(k=100000).nunique()
Out[2]: 100000
```

categorical in general make sense if k << n. slight overhead when k == n.

have you tried `read_sas`?

```
import pandas as pd
s1 = pd.Series(pd.util.testing.makeStringIndex(k=1000))
s1 = pd.concat([s1]*100).astype('category')
encs = 'utf-8', 'utf-16', 'utf-32', 'latin1', 'iso-8859-1', 'mbcs', 'ascii', 'cp1251', 'cp1252'
for enc in encs:
    s2 = s1.str.encode(enc).astype('category')
    print(enc)
    %timeit s1.str.encode(enc)
    %timeit s2.str.decode(enc)
```

![image](https://cloud.githubusercontent.com/assets/53390/14855912/9f0a0988-0c9e-11e6-9256-dc41c2eece64.png)

@jreback i tried `read_sas` from 0.18.0, it was more than twice slower than (slightly improved) Jared Hobbs' `sas7bdat` module. A lot of time is spent on decoding cp1251

try with master. its now 4x faster.

@jreback i've tried.) 2x faster on my test table, which is still amazing improvement
![image](https://cloud.githubusercontent.com/assets/53390/14887049/ac78ce4c-0d5c-11e6-90cc-2e03d4e07b0f.png)

cc @kshedden 

ok, seems reasonable.

thanks!
PERF: don't create the skiprows set if using the c-parser
State: closed

```
In [4]: DataFrame(np.random.randn(1000000,1)).to_csv('test.csv',index=False)
```

branch 

```
In [1]: %memit pd.read_csv('test.csv',skiprows=999999)
peak memory: 65.74 MiB, increment: 1.59 MiB

In [2]: %memit pd.read_csv('test.csv',skiprows=999999)
peak memory: 65.89 MiB, increment: 0.22 MiB

In [3]: %memit pd.read_csv('test.csv',skiprows=999999)
peak memory: 65.98 MiB, increment: 0.28 MiB
```

master

```
In [1]: %memit pd.read_csv('test.csv',skiprows=999999)
peak memory: 169.84 MiB, increment: 105.79 MiB

In [2]: %memit pd.read_csv('test.csv',skiprows=999999)
peak memory: 171.27 MiB, increment: 24.11 MiB

In [3]: %memit pd.read_csv('test.csv',skiprows=999999)
peak memory: 173.39 MiB, increment: 24.63 MiB
```


Labels:
name: Performance
description: Memory or execution speed performance
name: IO CSV
description: read_csv, to_csv

Comments:
@gfyoung I believe this is handled internally in the c-parser.

@jreback : Travis and I both agree.  LGTM otherwise.
DOC: more pd.to_datetime examples
State: closed

xref #12996


Labels:
name: Docs
name: Datetime
description: Datetime data dtype

Comments:
@jorisvandenbossche 

also added #12974 fix here.

@jreback Can you please try to not change the commit too much during the merge? This makes reviewing PRs / seeing what has been merged a lot more difficult using github (as it seems the 'Files changed' is quite different from the actual merged commit).

(of course, small fixup changes are not a problem, certainly for PRs of other contributors to not have them endlessly adapt minor things)

@jorisvandenbossche I normally don't. but ended up going on a tangent with some other things (somewhat unrelated).

That's OK! (I just noticed it for this one)
BUG/DOC: restrict possibilities for assembly of dates using a DataFrame
State: closed

xref #12967


Labels:
name: Datetime
description: Datetime data dtype

Comments:
cc @jorisvandenbossche 

@shoyer 

Apart from the doc comment, I would personally even leave out the abbreviations for `minutes` and `seconds`. Then also the possible confusion of `m` as month or minute is avoided (I also never seen 'm' used as abbreviation of minute)

+1 for leaving out any abbreviations. I'm not even sure it's actually a
good thing to accept both plural and singular forms.

On Tue, Apr 26, 2016 at 4:07 PM, Joris Van den Bossche <
notifications@github.com> wrote:

> Apart from the doc comment, I would personally even leave out the
> abbreviations for minutes and seconds. Then also the possible confusion
> of m as month or minute is avoided (I also never seen 'm' used as
> abbreviation of minute)
> 
> —
> You are receiving this because you were mentioned.
> Reply to this email directly or view it on GitHub
> https://github.com/pydata/pandas/pull/12996#issuecomment-214914504
PEP: use triple-double-quotes rather than triple-single-quotes in doc-strings
State: closed

mostly in cython, but in tests as well. This is as per PEP. not to mention that it screws up syntax highliting.


Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
xref #12995 

anyone know of a way to turn on checking for this? (e.g. in flake?)
BUG: agg() function on groupby dataframe changes dtype of datetime64[ns] column to float64 
State: closed

closes #12821 
closes #12941


Labels:
name: Bug
name: Datetime
description: Datetime data dtype
name: Groupby

Comments:
can you rebase / update

Sorry, @jreback.

This month I was far busy than expected, and I'm planning to fix the bug on next Sunday.

## [Current coverage](https://codecov.io/gh/pydata/pandas/pull/12992?src=pr) is 85.30% (diff: 100%)

> Merging [#12992](https://codecov.io/gh/pydata/pandas/pull/12992?src=pr) into [master](https://codecov.io/gh/pydata/pandas/branch/master?src=pr) will decrease coverage by **<.01%**

``` diff
@@             master     #12992   diff @@
==========================================
  Files           139        139          
  Lines         50138      50140     +2   
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
+ Hits          42769      42770     +1   
- Misses         7369       7370     +1   
  Partials          0          0          
```

> Powered by [Codecov](https://codecov.io?src=pr). Last update [3186fef...607a170](https://codecov.io/gh/pydata/pandas/compare/3186fef545b55ca7b4a3c79c800f32b1d586545e...607a170563bdc56a3df3d064325b75b5a446bd9f?src=pr)

can you rebase / update?

does this also address #12941  ?

pls rebase / small update as well. ping on green.

Hi, I have fixed #12941 and rebase again.

ok looks good. pls rebase / squash (and build failed).

rebase again.

can you rebase and i'll review

can you rebase. ping when green.

@jreback, hi, agg function will eat up Name, is it a bug?

not sure can u show s complete example

``` python
In [12]: df = pd.Series(range(5)).to_frame()

In [13]: df
Out[13]:
   0
0  0
1  1
2  2
3  3
4  4

In [14]: df.min()
Out[14]:
0    0
dtype: int64

In [15]: df.loc[0]
Out[15]:
0    0
Name: 0, dtype: int64
```

Giving a slightly more clear example. I suppose this could be done. Though its only for a very limited number of reducers, prob just min/max. For example, anything that does an aggregation like `.sum/.mean` this by-definition obliterates the index. Pls make a new issue and list the functions you think should be changed here.

```
In [19]: df = pd.Series(range(5),index=list('abcde')).to_frame(name='foo')

In [20]: df
Out[20]: 
   foo
a    0
b    1
c    2
d    3
e    4

In [21]: df.loc['a']
Out[21]: 
foo    0
Name: a, dtype: int64

In [22]: df.min()
Out[22]: 
foo    0
dtype: int64

```

@jreback Hi, I'll submit a issue later. And all checks have passed, so do you think that the pr is ok?

@jreback all checks have passed.

lgtm. pls rebase and ping on green.

@jreback Both #12821  and #12941 are fixed in the PR. The two bugs behave similarity, while they are located in different code. And all checks have passed.

thanks!

@ningchi can you have a look at these, see if they are _also_ closed (so maybe just need some validation tests).

xref #8617 (might be a duplicate)
xref #4147

Ok

It seems that #8617 has been fixed, while #4147 behaves differently now.
BUG: Preserve timezone in unaligned assignments
State: closed

- [x] closes #12981 
- [x] tests added / passed
- [x] passes `git diff upstream/master | flake8 --diff`
- [x] whatsnew entry

A fix for #12981.  When doing a slice assign to a DataFrame where the
right-hand-side was timezone-aware datetime Series which required
realignment to perform the assignment, the timezone would be stripped
from the RHS on assignment.


Labels:
name: Bug
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Timezones
description: Timezone data dtype

Comments:
minor changes. ping when green.

## [Current coverage](https://codecov.io/gh/pydata/pandas/pull/12982?src=pr) is **83.85%**

> Merging [#12982](https://codecov.io/gh/pydata/pandas/pull/12982?src=pr) into [master](https://codecov.io/gh/pydata/pandas/branch/master?src=pr) will increase coverage by **+<.01%**

``` diff
@@             master     #12982   diff @@
==========================================
  Files           136        136          
  Lines         49711      49711          
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
+ Hits          41679      41683     +4   
+ Misses         8032       8028     -4   
  Partials          0          0          
```
1. File `...as/core/internals.py` (not in diff) was modified. [more](https://codecov.io/gh/pydata/pandas/commit/94451553513be70af21eb8649a9217e10dac27fd/changes?src=pr#70616E6461732F636F72652F696E7465726E616C732E7079) 
   - Misses `-4` 
   - Partials `0` 
   - Hits `+4`

[![Sunburst](https://codecov.io/gh/pydata/pandas/pull/12982/graphs/sunburst.svg?size=660&src=pr)](https://codecov.io/gh/pydata/pandas/pull/12982?src=pr)

> Powered by [Codecov](https://codecov.io?src=pr). Last updated by 9445155

thanks!