BrandonFors/pandas-github-issues huggingface dataset:
BUG: union_categoricals can't handle NaN
State: closed

- [x] tests added / passed
- [x] passes `git diff upstream/master | flake8 --diff`
- [x] whatsnew not needed

`union_categoricals` doesn't handle `NaN` properly.

**on current master:**

```
from pandas.types.concat import union_categoricals
union_categoricals([pd.Categorical([np.nan, 1]), pd.Categorical([2, np.nan])])
# [1, 1, 2, 2]
# Categories (2, int64): [1, 2]

union_categoricals([pd.Categorical([np.nan]), pd.Categorical([np.nan])])
# IndexError: cannot do a non-empty take from an empty axes.
```


Labels:
name: Bug
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Categorical
description: Categorical Data Type

Comments:
## [Current coverage](https://codecov.io/gh/pydata/pandas/pull/13759?src=pr) is 84.57% (diff: 100%)

> Merging [#13759](https://codecov.io/gh/pydata/pandas/pull/13759?src=pr) into [master](https://codecov.io/gh/pydata/pandas/branch/master?src=pr) will increase coverage by **<.01%**

``` diff
@@             master     #13759   diff @@
==========================================
  Files           141        141          
  Lines         51195      51200     +5   
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
+ Hits          43295      43300     +5   
  Misses         7900       7900          
  Partials          0          0          
```

> Powered by [Codecov](https://codecov.io?src=pr). Last update [9f94e6a...ab99976](https://codecov.io/gh/pydata/pandas/compare/9f94e6a04564e8269b43c5b4e83854d277ea30bd...ab99976f4a29e9291e525a50549f5f37456b92b6?src=pr)

lgtm. 

cc @chris-b1 

Yeah, I had actually fixed this in #13406 too - this PR fixes some other corners I missed.  I used `take_1d` with a fill value, which seems to be a little faster?

``` python
In [13]: cats = [pd.Categorical(list('abcdefghijkl') * 100) for _ in range(100)]

In [14]: union_categoricals_pr = pd.types.concat.union_categoricals # from 13406

In [15]: def union_categoricals_sinhrks(to_union):
 <snip - this pr>

In [16]: %timeit union_categoricals_pr(cats)
100 loops, best of 3: 2.78 ms per loop

In [17]: %timeit union_categoricals_sinhrks(cats)
100 loops, best of 3: 3.45 ms per loop

```

@chris-b1 thx, changed to use `take_1d`.

Looks good to me, go ahead with merging if ready!
BUG: rolling with axis=1
State: closed

- [x] fixes #13753 
- [ ] tests added / passed
- [x] passes `git diff upstream/master | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Bug
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
tests!

On it! Clearly new to TDD but will take a stab.

[contributing guide](http://pandas.pydata.org/pandas-docs/stable/contributing.html)

## [Current coverage](https://codecov.io/gh/pydata/pandas/pull/13757?src=pr) is 84.56% (diff: 100%)

> Merging [#13757](https://codecov.io/gh/pydata/pandas/pull/13757?src=pr) into [master](https://codecov.io/gh/pydata/pandas/branch/master?src=pr) will increase coverage by **<.01%**

``` diff
@@             master     #13757   diff @@
==========================================
  Files           141        141          
  Lines         51195      51196     +1   
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
+ Hits          43295      43296     +1   
  Misses         7900       7900          
  Partials          0          0          
```

> Powered by [Codecov](https://codecov.io?src=pr). Last update [9f94e6a...5bad8f2](https://codecov.io/gh/pydata/pandas/compare/9f94e6a04564e8269b43c5b4e83854d277ea30bd...5bad8f25a62ae8598aa3d11017d4195c37de21a7?src=pr)

this need to address #13503 which is the main issue.

Are you saying that if I resolve the issue in #13503 then it will also automatically address the issue in #13753? It looks like #13503 is related to how `.count()` is implemented for `.rolling`. However, in my `.expanding` example, I am using `.sum()`. It would be helpful if you could point me in the right direction or which class to make changes in. Perhaps, in `Rolling`?

yes it's the same issue and is addressed in the same place

put tests in place first 

pls reopen if you'd like to update
PERF: Improve duplicated perf
State: closed

- [x] closes #10235
- [x] tests added / passed
- [x] passes `git diff upstream/master | flake8 --diff`
- [x] whatsnew entry

`Index/Series.duplicated` now uses dtype-based logic. Also skip algorithm if `Index.is_unique` is `True` (expected to be cached in practical situation).

**asv:**

```
   before     after       ratio
  [bb6b5e54] [16c6da4f]
-   34.41ms     4.69ms      0.14  algorithms.algorithm.time_int_duplicated
-   53.77ms     5.59ms      0.10  algorithms.algorithm.time_float_duplicated
-   60.22ms    40.20μs      0.00  algorithms.algorithm.time_int_unique_duplicated
SOME BENCHMARKS HAVE CHANGED SIGNIFICANTLY.
```

**NOTE:** can create template for htable after #13716.


Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Performance
description: Memory or execution speed performance

Comments:
## [Current coverage](https://codecov.io/gh/pydata/pandas/pull/13751?src=pr) is 84.58% (diff: 100%)

> Merging [#13751](https://codecov.io/gh/pydata/pandas/pull/13751?src=pr) into [master](https://codecov.io/gh/pydata/pandas/branch/master?src=pr) will increase coverage by **<.01%**

``` diff
@@             master     #13751   diff @@
==========================================
  Files           141        141          
  Lines         51233      51258    +25   
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
+ Hits          43331      43356    +25   
  Misses         7902       7902          
  Partials          0          0          
```

> Powered by [Codecov](https://codecov.io?src=pr). Last update [2c047d4...12fb5ac](https://codecov.io/gh/pydata/pandas/compare/2c047d4fbe88e7462ec82314f58d014e7b0ad54e...12fb5ac31cf3f86232be9c6617ead0ce25a33f69?src=pr)

Updated the bench.`algorithms.algorithm.time_int_unique_duplicated` now benchmark under `is_unique` is once cached.
PERF/BUG: improve factorize for datetimetz
State: closed

- [x] tests added / passed
- [x] passes `git diff upstream/master | flake8 --diff`
- [x] whatsnew entry

because `factorize` internally localize datetimetz, it raises when data contains DST boundary.

```
dti = pd.date_range('2016-11-06', freq='H', periods=5, tz='US/Eastern')
dti.factorize()
# AmbiguousTimeError: Cannot infer dst time from Timestamp('2016-11-06 01:00:00'), try using the 'ambiguous' argument
```

Skipped this localization to fix, also it  improves perf.

```
dti = pd.date_range('2011-01-01', freq='H', periods=1000000, tz='Asia/Tokyo')
%timeit dti.factorize()
# on current master
#1 loop, best of 3: 475 ms per loop

# after this PR
#1 loop, best of 3: 262 ms per loop
```

**asv:**

```
   before     after       ratio
  [bb6b5e54] [a2c3370a]
-   22.46ms     9.97ms      0.44  timeseries.datetime_algorithm.time_dti_tz_factorize
SOME BENCHMARKS HAVE CHANGED SIGNIFICANTLY.
```


Labels:
name: Bug
name: Performance
description: Memory or execution speed performance
name: Timezones
description: Timezone data dtype

Comments:
## [Current coverage](https://codecov.io/gh/pydata/pandas/pull/13750?src=pr) is 84.56% (diff: 100%)

> Merging [#13750](https://codecov.io/gh/pydata/pandas/pull/13750?src=pr) into [master](https://codecov.io/gh/pydata/pandas/branch/master?src=pr) will not change coverage

``` diff
@@             master     #13750   diff @@
==========================================
  Files           141        141          
  Lines         51196      51196          
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
  Hits          43296      43296          
  Misses         7900       7900          
  Partials          0          0          
```

> Powered by [Codecov](https://codecov.io?src=pr). Last update [bb6b5e5...a2c3370](https://codecov.io/gh/pydata/pandas/compare/bb6b5e54edaf046389e8cce28e7cd27ee87f5fcc...a2c3370a51db64ec7ed6fc8c186df744501bdb13?src=pr)

lgtm.

I _think_ this may solve some of these Ambiguous time issues: https://github.com/pydata/pandas/search?q=cannot+infer+dst&type=Issues&utf8=%E2%9C%93

maybe

thanks! see my comment above if you have some time.

yeah, i think we can remove most of `tz_localize(None)` logic from core.
PERF: RangeIndex.is_monotonic_inc/dec
State: closed

- [x] tests added / passed
- [x] passes `git diff upstream/master | flake8 --diff`
- [x] whatsnew entry

as the same as `is_unique`, it should be monotonic by definition

```
%timeit -n 1 -r 1 pd.RangeIndex(0, 1000000).is_monotonic_increasing

# current master
#1 loop, best of 1: 17.2 ms per loop

# this PR
#1 loop, best of 1: 45.1 µs per loop
```


Labels:
name: Performance
description: Memory or execution speed performance

Comments:
## [Current coverage](https://codecov.io/gh/pydata/pandas/pull/13749?src=pr) is 84.57% (diff: 100%)

> No coverage report found for **master** at b60e42b.
> 
> Powered by [Codecov](https://codecov.io?src=pr). Last update [b60e42b...8e25563](https://codecov.io/gh/pydata/pandas/compare/b60e42b02040624f319cadd60a99fe50eb62d62e...8e25563bc9061535853c19740684a7d40394cc33?src=pr)
ENH: add is_leap_year property for datetime-like
State: closed

- [x] closes #13727
- [x] tests added / passed
- [x] passes `git diff upstream/master | flake8 --diff`
- [x] whatsnew entry
- found there is `tslib.isleapyear` also, but i don't think it is public (deprecated also).
- `pd.NaT.is_leap_year` results in `False`, as I think users want `bool` array.


Labels:
name: Enhancement
name: Datetime
description: Datetime data dtype
name: Deprecate
description: Functionality to remove in pandas

Comments:
lgtm

add to docs [here](http://pandas.pydata.org/pandas-docs/stable/timeseries.html#time-date-components) as well.

note when looking at them again

```
is_year_start
is_year_end
is_leap_year
```

looks a tiny bit weird.....but makes sense

@jorisvandenbossche @jreback Thx for kind reviews always:) Fixed points.

## [Current coverage](https://codecov.io/gh/pydata/pandas/pull/13739?src=pr) is 85.32% (diff: 100%)

> Merging [#13739](https://codecov.io/gh/pydata/pandas/pull/13739?src=pr) into [master](https://codecov.io/gh/pydata/pandas/branch/master?src=pr) will increase coverage by **<.01%**

``` diff
@@             master     #13739   diff @@
==========================================
  Files           141        141          
  Lines         50679      50686     +7   
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
+ Hits          43240      43247     +7   
  Misses         7439       7439          
  Partials          0          0          
```

> Powered by [Codecov](https://codecov.io?src=pr). Last update [474fd05...5d227ee](https://codecov.io/gh/pydata/pandas/compare/474fd0519b768efd1c7ec6555930873bf9910e00...5d227ee0d9ad39c6e7f0637470e201d7a95052c3?src=pr)

lgtm. merge on pass.

FYI depending on when you rebased, py3 may fail for [this](https://github.com/pydata/pandas/commit/964b7bba7f5878c79130479f75461c58dd0c4b3e)

thanks @sinhrks 
nice PR
TST/BUG: Added more tests for Period(NaT)
State: closed

- [x] follow-up of #12759
- [x] tests added / passed
- [x] passes `git diff upstream/master | flake8 --diff`
- [x] whatsnew entry
- because `Period(NaT)` had separate impl until #12579, there were some bugs related to `pd.isnull`. Now all the issues are fixed, thus added explicit tests and what's new. 
- Also updated what's new to describe `Period(None)` (related to #13723).

**0.18.1:**

```
s = pd.Series([pd.Period('2011-01', freq='M'), pd.Period('NaT', freq='M')])
s.isnull()
#0    False
#1    False
# dtype: bool

s.fillna(pd.Period('2012-01', freq='M'))
#0   2011-01
#1       NaT
# dtype: object

s.dropna()
#0   2011-01
#1       NaT
# dtype: object
```


Labels:
name: Bug
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: Period
description: Period data type

Comments:
Looks good to me!

thanks!
CNL: remove the io.data and io.wb modules 
State: closed

Closes #13724


Labels:
name: Clean

Comments:
@jorisvandenbossche ok lgtm (and fine to have error on both).

whoo hoo!

@jorisvandenbossche : any reason why we just didn't add `pandas-datareader` as a dependency and do a `try-except` block with `import pandas-datareader`?  I ask because in the <a href="https://github.com/pydata/pandas-datareader/blob/master/README.rst">README</a> for `pandas-datareader`, it seems to imply that `pandas` should be dependent on `pandas-datareader` by now.

the dependency is the other way
u can submit a PR to pandas-datareader if you want to change the wording

@jreback : fair enough, but I still don't get why we didn't do `try-except` with `pandas-datareader`, or rather, why we even have these files (`wb.py` and `data.py`) at all in the `pandas` library still.

they raise a more helpful message 

@jreback : Ah, so we're helping people with transitioning from `io.data` to `pandas-datareader`?

of course - 

Hmm, didn't see that in the readme of pandas-datareader (and it's not how it is phrased in the pandas docs). Anyway, I see that your PR over there has already been merged. 
As I said in the issue https://github.com/pydata/pandas/issues/13724, there was some unresolved discussion about this at the time we deprecated it and created pandas-datareader (I think @hayd was in favor of doing it like you say).  

Keeping the file with the custom import error is indeed to have a more helpful error for transitioning to pandas-datareader.
TST/BUG: test unsortable in safe_sort and Categorical (GH13714)
State: closed

- [x] closes #13714
- [x] tests added / passed
- [x] passes `git diff upstream/master | flake8 --diff`
- [ ] whatsnew entry
- Change test for Categorical.from_array (`test_constructor_unsortable`)
- Add test for safe_sort
- Change 2.7 build to latest numpy

---

`safe_sort` always raises on a mixed int-datetime array. This is the expected behaviour. (But, for numpy >= 1.10 & python2, I still have some doubts whether it's not a manifestation of an undefined behaviour.)


Labels:
name: Compat
description: pandas objects compatability with Numpy or Python functions

Comments:
closed by https://github.com/pydata/pandas/commit/622297ccf12e9b013de6c7a1316dd9453176ecb1

thanks!

FYI we have `pd._np_under*` properties which know about numpy versions
TST, COMPAT: Make MMapWrapper tests Windows compatible
State: closed

Partially addresses #13721 by addressing the `MMapWrapper` test failure.  The issue was we didn't have control over the newline character, so the test was re-written to give us that control.


Labels:
name: IO CSV
description: read_csv, to_csv
name: Compat
description: pandas objects compatability with Numpy or Python functions

Comments:
I think you can use `open(..., mode='U')`?

Unfortunately, that seems to butcher what the end-of-line characters are:

``` python
>>> f = open('test.txt', 'w+')
>>> f.write('a\r\n')
3
>>> f.write('a\r\n')
3
>>> f.write('a\r\n')
3
>>> f.close()
>>> f = open('test.txt', mode='U')
>>> f.readlines()   # mmap will recognize the '\r\n' though
['a\n', 'a\n', 'a\n']
```

@jreback : Because this is a `readlines` issue and NOT an issue with `mmap`, as workaround, I am going to call `.strip()` just to remove the newline character and compare actual content.

## [Current coverage](https://codecov.io/gh/pydata/pandas/pull/13732?src=pr) is 84.56% (diff: 100%)

> Merging [#13732](https://codecov.io/gh/pydata/pandas/pull/13732?src=pr) into [master](https://codecov.io/gh/pydata/pandas/branch/master?src=pr) will not change coverage

``` diff
@@             master     #13732   diff @@
==========================================
  Files           141        141          
  Lines         51196      51196          
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
  Hits          43296      43296          
  Misses         7900       7900          
  Partials          0          0          
```

> Powered by [Codecov](https://codecov.io?src=pr). Last update [bb6b5e5...dd1b60b](https://codecov.io/gh/pydata/pandas/compare/bb6b5e54edaf046389e8cce28e7cd27ee87f5fcc...dd1b60b63896d66fb51a7dbb87a82db838c806b3?src=pr)

@jreback : I fixed the test, so Travis now passes.  Ready to merge if there are no other concerns.

lgtm (though @jorisvandenbossche if you'd test on windows to be sure)

oddly though I cannot reproduce this anymore.......

Sorry, I wasn't using my windows the last days
Groupby getitem works with all index types
State: closed

- [x] tests added / passed
- [x] passes `git diff upstream/master | flake8 --diff`
- [x] whatsnew entry

Previously `df.groupby(0)[df.columns]` would fail if all column names
were integers (meaning `df.columns` was an `Int64Index`). This was
because the implementation of `__getitem__` in `SelectionMixin` was
checking for `ABCIndex` when it probably should have checked for
`ABCIndexClass`.


Labels:
name: Bug
name: Groupby

Comments:
## [Current coverage](https://codecov.io/gh/pydata/pandas/pull/13731?src=pr) is 84.56% (diff: 100%)

> Merging [#13731](https://codecov.io/gh/pydata/pandas/pull/13731?src=pr) into [master](https://codecov.io/gh/pydata/pandas/branch/master?src=pr) will increase coverage by **<.01%**

``` diff
@@             master     #13731   diff @@
==========================================
  Files           141        141          
  Lines         51195      51196     +1   
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
+ Hits          43295      43296     +1   
  Misses         7900       7900          
  Partials          0          0          
```

> Powered by [Codecov](https://codecov.io?src=pr). Last update [e357ea1...b901322](https://codecov.io/gh/pydata/pandas/compare/e357ea1105723781032f0ec509236b69325633f4...b901322a24138ee7f5a847269fdf394bbd1628b9?src=pr)

Looks good to me, some minor comments

yep lgtm.

Thanks, fixed.

lgtm

@jcrist Thanks!
BUG: Accept kwargs kind and na_position in Series.sort_index (GH13589)
State: closed

- [x] closes #13589 
- [x] tests added / passed
- [x] whatsnew entry

/cc @jreback 


Labels:
name: API Design
name: Algos
description: Non-arithmetic algos: value_counts, factorize, sorting, isin, clip, shift, diff
name: Compat
description: pandas objects compatability with Numpy or Python functions

Comments:
you always needs tests 

@jreback I'm not sure what to add in tests for this one. Can you please help me here?

you are adding keywords so need tests - look at what Dataframe tests for snd mirror those 

I tried looking into `test_frame.py`. It has no test containing `kind` or `na_position`. Am I missing something or looking at wrong place?

rebase on master you have a really old master 
just recursive grep

> just recursive grep

Thanks for the hint, @jreback. Please check the tests I have added if they are sufficient or not.

@jreback Is this fine?

@sahildua2305 Only a few small comments, for the rest this is already looking fine!

@jorisvandenbossche I've updated the PR. Please check if it's good now.

can you rebase / update?

Yes! I'll do so. 

closing in favor of #14445
BUG: Add check for input array lengths in from_arrays method (GH13599)
State: closed

- [x] closes #13599 
- [ ] tests added / passed
- [x] whatsnew entry

/cc @jreback 


Labels:
name: Error Reporting
description: Incorrect or improved errors from pandas
name: MultiIndex

Comments:
ok this is fine. in general no need to open a new PR, just push to the old one.

@jreback It had conflicts with the base branch. How would you have merged it?

see the contributing docs [here](http://pandas.pydata.org/pandas-docs/stable/contributing.html#contributing-your-changes-to-pandas) and how git works. you simply rebase and fix the conflicts.

Ah! Okay.

Actually I tried to rebase on my branch and ran into issues (it included other commits as well), that's why had to close the old PR.

thanks!

try `git rebase -i origin/master` in the future

Thanks :)
BUG: Fix pd.Timedelta(None) to return NaT.
State: closed

- [ ] closes #xxxx
- [x] tests added / passed
- [x] passes `git diff upstream/master | flake8 --diff`
- [x] whatsnew entry

Before this commit pd.Timedelta raised ValueError
when None is passed.
This behavior is inconsistent because
- pd.Timestamp(None) and pd.Period(None) return NaT
- pd.Timedelta returns NaT if '', 'nat', 'NAT' or np.nan
  is passed

This PR is related to https://github.com/pydata/pandas/pull/13687#issuecomment-233204955


Labels:
name: Timedelta
description: Timedelta data type

Comments:
## [Current coverage](https://codecov.io/gh/pydata/pandas/pull/13723?src=pr) is 84.57% (diff: 100%)

> No coverage report found for **master** at 6efd743.
> 
> Powered by [Codecov](https://codecov.io?src=pr). Last update [6efd743...5e29ad9](https://codecov.io/gh/pydata/pandas/compare/6efd743ccbf2ef6c13ea0c71b7b2e2a022a99455...5e29ad99ac992d48941597cf2afdc75f47052bf2?src=pr)

Actually, regarding no arguments, Timestamp raises in that case:

```
In [2]: pd.Timestamp(None)
Out[2]: NaT

In [3]: pd.Timestamp()
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-3-384f27778dc4> in <module>()
----> 1 pd.Timestamp()

pandas\tslib.pyx in pandas.tslib.Timestamp.__new__ (pandas\tslib.c:9160)()

TypeError: __new__() takes at least 2 positional arguments (1 given)
```

So not sure if we should let it here return NaT.

no `pd.Timestamp()`, `.Timedelta()`, `.Period()` are errors, passing `None` is ok though as better to be explicit.

Rebased and updated.

pls add a whatsnew note.

FYI, PR's always need notes & tests & sometimes doc updates. These are often much more effort, than the actual code change. Esp for tests these need to be done upfront.

Sorry I misunderstood what sinhrks said. I added a whatsnew note.
Thanks for your advice.

thanks, whatsnew [here](https://github.com/pydata/pandas/commit/2c047d4fbe88e7462ec82314f58d014e7b0ad54e)

😍  Thanks!
ENH: Enable automatic writing of dates to Stata files
State: closed

- [x] closes #12259
- [x] tests added / passed
- [x] passes `git diff upstream/master | flake8 --diff`
- [x] whatsnew entry

Automatically select type %tc for datetime[ns] columns
Change ValueErrors to NotImplementedError for unsupported types
Add tests for select exceptions

closes #12259


Labels:
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: IO Stata
description: read_stata, to_stata

Comments:
## [Current coverage](https://codecov.io/gh/pydata/pandas/pull/13710?src=pr) is **84.53%**

> No coverage report found for **master** at 1ce8f8e.
> 
> Powered by [Codecov](https://codecov.io?src=pr). Last updated by [1ce8f8e...73f6565](https://codecov.io/gh/pydata/pandas/compare/1ce8f8e0b8540252dac25497f29d4de66a8bea3f...73f6565d0584f8ffc5785377f5bb334269a247a3?src=pr)

thanks!