BrandonFors/pandas-github-issues huggingface dataset:
BUG: Fix #15344 by backporting ujson usage of PEP 393 API
State: closed

Make use of the PEP 393 API to avoid expanding single byte ascii characters into four byte unicode characters when encoding objects to json.

 - [ ] closes #15344
 - [ ] tests added / passed
 - [ ] passes ``git diff upstream/master | flake8 --diff``
 - [ ] whatsnew entry


Labels:
name: IO JSON
description: read_json, to_json, json_normalize

Comments:
lgtm. can you add a whatsnew note? (Bug fix is fine)
https://travis-ci.org/pandas-dev/pandas/jobs/200143310

we have a c-linter :<
Ouch :-). Hopefully fixed now.
thanks @tobgu 

keep em coming!

I am still puzzled why the encoding of a new UTF8 string, that returns a new reference is affecting existing references. Though this maybe some deep things going on that I don't understand. If you have time I would search / post an issue on CPython to understand the semantics here. Its also possible that this is the reason this function is being deprecated (because of these side effects).

```
PyObject* PyUnicode_EncodeUTF8(const Py_UNICODE *s, Py_ssize_t size, const char *errors)
Return value: New reference.
Encode the Py_UNICODE buffer s of the given size using UTF-8 and return a Python bytes object. Return NULL if an exception was raised by the codec.

Part of the old-style Py_UNICODE API; please migrate to using PyUnicode_AsUTF8String() or PyUnicode_AsUTF8AndSize().
Deprecated since version 3.3, will be removed in version 4.0.
```
I think the culprit in this case is the `PyUnicode_AS_UNICODE` macro that's described as inefficient and something you should migrate away from:
https://www.python.org/dev/peps/pep-0393/#porting-guidelines
https://docs.python.org/3/c-api/unicode.html#c.PyUnicode_AS_UNICODE

It seems like there's actually an allocation for a full strlen*4 byte unicode string here that would be the root cause of the observed behaviour:
https://github.com/python/cpython-mirror/blob/9aa885c9df5b6895f70272b6abc24ef0b1fb8b46/Objects/unicodeobject.c#L4089

Haven't dug into it further than that...

Thanks @jreback and @chris-b1 for helping out with this! It would have been a show stopper for the application that I'm currently working on. Hope to see it released soon! :-)
thanks for looking @tobgu, that is some *nasty* code.
TST: Remaining tseries tests reorg (gh14854)
State: closed

 - [x] closes #14854 
 - [x] tests reorg
 - [x] passes ``git diff upstream/master | flake8 --diff``
 - [x] whatsnew: Moved ```pandas/tseries/tests/*``` to ```pandas/tests/tseries/*```


Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
looks good. can you add a whatsnew note in Other Api changes.
CLN: strip out and form tools/concat.py from tools/merge.py
State: closed

will facilitate some changes in ``tools/merge`` w.r.t. #15321, plus these are independent anyhow.

Labels:
name: Clean

Comments:
even though technically ``pandas.tools.merge`` is not public, I added a FutureWarning for importing concat (similar to what we did for imports from pandas.core.common a while back).
DEPR: remove statsmodels as a dependency
State: closed

remove pd.ols, pd.fama_macbeth from top-level namespace

xref #11898
closes https://github.com/pandas-dev/pandas2/issues/26

previously deprecated in 0.18.0

Labels:
name: Deprecate
description: Functionality to remove in pandas

Comments:
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/15353?src=pr&el=h1) Report
> Merging [#15353](https://codecov.io/gh/pandas-dev/pandas/pull/15353?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/d38d142aaac5ee046aabc32c1e6422f86c37dc41?src=pr&el=desc) will **increase** coverage by `2.38%`.


```diff
@@            Coverage Diff             @@
##           master   #15353      +/-   ##
==========================================
+ Coverage   86.32%   88.71%   +2.38%     
==========================================
  Files         141      133       -8     
  Lines       51163    49466    -1697     
==========================================
- Hits        44167    43884     -283     
+ Misses       6996     5582    -1414
```


| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/15353?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/stats/api.py](https://codecov.io/gh/pandas-dev/pandas/compare/d38d142aaac5ee046aabc32c1e6422f86c37dc41...6622ca05d3dc472ed2fdada888d7c2d8cef7171b?src=pr&el=tree#diff-cGFuZGFzL3N0YXRzL2FwaS5weQ==) | `100% <ø> (ø)` | :white_check_mark: |
| [pandas/util/print_versions.py](https://codecov.io/gh/pandas-dev/pandas/compare/d38d142aaac5ee046aabc32c1e6422f86c37dc41...6622ca05d3dc472ed2fdada888d7c2d8cef7171b?src=pr&el=tree#diff-cGFuZGFzL3V0aWwvcHJpbnRfdmVyc2lvbnMucHk=) | `15.71% <ø> (ø)` | :white_check_mark: |
| [pandas/util/testing.py](https://codecov.io/gh/pandas-dev/pandas/compare/d38d142aaac5ee046aabc32c1e6422f86c37dc41...6622ca05d3dc472ed2fdada888d7c2d8cef7171b?src=pr&el=tree#diff-cGFuZGFzL3V0aWwvdGVzdGluZy5weQ==) | `82.72% <ø> (+0.18%)` | :white_check_mark: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/15353?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/15353?src=pr&el=footer). Last update [d38d142...9563740](https://codecov.io/gh/pandas-dev/pandas/compare/d38d142aaac5ee046aabc32c1e6422f86c37dc41...9563740551d537ca719e8e0554c7b9f2f2f472bb?src=pr&el=footer&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
> Merging #15353 into master will increase coverage by 2.38%

a nice way of increasing coverage :>
TST: Period tests reorg (gh14854)
State: closed

 - [x] Almost completes #14854 
 - [x] tests reorg
 - [x] passes ``git diff upstream/master | flake8 --diff``
 - [x] whatsnew: Reorganization of ```Period``` tests.


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Period
description: Period data type

Comments:
whoosh that ``test_period`` was huge. ok. ping when green.
hahah yeah. As a result, ```scalar/test_period.py``` and ```indexes/period/test_ops.py``` have gotten big as well.

yep
certainly can come back later and split things a bit

but let's wait on that for now
@TrigonaMinima going to merge shortly.
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/15352?src=pr&el=h1) Report
> Merging [#15352](https://codecov.io/gh/pandas-dev/pandas/pull/15352?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/704cdbf830c110001062012b92302ed30d8ae127?src=pr&el=desc) will **increase** coverage by `<.01%`.


```diff
@@            Coverage Diff             @@
##           master   #15352      +/-   ##
==========================================
+ Coverage   86.32%   86.32%   +<.01%     
==========================================
  Files         141      141              
  Lines       51163    51163              
==========================================
+ Hits        44166    44167       +1     
+ Misses       6997     6996       -1
```


| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/15352?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/tseries/index.py](https://codecov.io/gh/pandas-dev/pandas/compare/704cdbf830c110001062012b92302ed30d8ae127...b28eec76ffbfe4556fd2bb7fd81ae9cd021550c8?src=pr&el=tree#diff-cGFuZGFzL3RzZXJpZXMvaW5kZXgucHk=) | `95.42% <ø> (+0.09%)` | :white_check_mark: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/15352?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/15352?src=pr&el=footer). Last update [704cdbf...b28eec7](https://codecov.io/gh/pandas-dev/pandas/compare/704cdbf830c110001062012b92302ed30d8ae127...b28eec76ffbfe4556fd2bb7fd81ae9cd021550c8?el=footer&src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
thanks again @TrigonaMinima !

if you are up for more :>

let's move the remainder of tseries/tests to pandas/tests/tseries

also let's remove tests/timeseries_legacy.py (and some of data can be removed as well).

BUT  should move any useful tests to a proper location. IOW the pickle things you can simply remove (they are really old). but some of the freq tests might be useful.
Sure. I'll try to do it tonight.

I should also thank you. You are helping me alot throughout the way.

Is there something remaining to be done in this PR?
closed via https://github.com/pandas-dev/pandas/commit/9ba10089f1c57d6fff569af39f0e6d37ee0210f5

@TrigonaMinima no this is all good. thanks!
Small documentation fix for MultiIndex.sortlevel
State: closed

MultiIndex.sortlevel returns two values (MultiIndex and indices that sort the MultiIndex). Only the MultiIndex was described in the documentation. 


Labels:
name: Docs
name: MultiIndex

Comments:
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/15345?src=pr&el=h1) Report
> Merging [#15345](https://codecov.io/gh/pandas-dev/pandas/pull/15345?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/153da508a536a3ef203d9cd315c67b5fd3022a51?src=pr&el=desc) will **not impact** coverage.


```diff
@@           Coverage Diff           @@
##           master   #15345   +/-   ##
=======================================
  Coverage   86.32%   86.32%           
=======================================
  Files         141      141           
  Lines       51165    51165           
=======================================
  Hits        44169    44169           
  Misses       6996     6996
```


| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/15345?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/indexes/multi.py](https://codecov.io/gh/pandas-dev/pandas/compare/153da508a536a3ef203d9cd315c67b5fd3022a51...c394448eb3d06c3aad08c2e2a49d0c931605e138?src=pr&el=tree#diff-cGFuZGFzL2luZGV4ZXMvbXVsdGkucHk=) | `96.58% <ø> (ø)` | :white_check_mark: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/15345?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/15345?src=pr&el=footer). Last update [153da50...3a3a11a](https://codecov.io/gh/pandas-dev/pandas/compare/153da508a536a3ef203d9cd315c67b5fd3022a51...3a3a11a5a3d97a968dc6609691e621f7e6cddd53?el=footer&src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
this Returns is basically the same doc-string as ``.reindex``. can you make it like that formatting.
thanks @J535D165
BUG: Multiline Eval broken for local variables after first line
State: closed

Also fixes the code which attempted to ignore any blank lines in the
multiline expression.

 - [x] closes #15342 
 - [x] tests added / passed
 - [x] passes ``git diff upstream/master | flake8 --diff``
 - [x] whatsnew entry


Labels:
name: Bug
name: Numeric Operations
description: Arithmetic, Comparison, and Logical operations

Comments:
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/15343?src=pr&el=h1) Report
> Merging [#15343](https://codecov.io/gh/pandas-dev/pandas/pull/15343?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/d38d142aaac5ee046aabc32c1e6422f86c37dc41?src=pr&el=desc) will **decrease** coverage by `-0.01%`.


```diff
@@            Coverage Diff             @@
##           master   #15343      +/-   ##
==========================================
- Coverage   86.32%   86.32%   -0.01%     
==========================================
  Files         141      141              
  Lines       51163    51162       -1     
==========================================
- Hits        44167    44166       -1     
  Misses       6996     6996
```


| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/15343?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/computation/eval.py](https://codecov.io/gh/pandas-dev/pandas/compare/d38d142aaac5ee046aabc32c1e6422f86c37dc41...fe67ede5abed071980a44600e66397a41577780d?src=pr&el=tree#diff-cGFuZGFzL2NvbXB1dGF0aW9uL2V2YWwucHk=) | `94.56% <100%> (-0.06%)` | :x: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/15343?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/15343?src=pr&el=footer). Last update [d38d142...fe67ede](https://codecov.io/gh/pandas-dev/pandas/compare/d38d142aaac5ee046aabc32c1e6422f86c37dc41...fe67ede5abed071980a44600e66397a41577780d?el=footer&src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
@chris-b1 
Thanks @stephenrauch - I only half remember how the scope level works - is there any way to make this clearer / cleaner?  We're now doing a `+1` in 3 places.

https://github.com/stephenrauch/pandas/blob/c7ac553366d747d33fba861e6f3afcc44f145721/pandas/computation/eval.py#L257

https://github.com/stephenrauch/pandas/blob/c7ac553366d747d33fba861e6f3afcc44f145721/pandas/computation/scope.py#L22

https://github.com/stephenrauch/pandas/blob/c7ac553366d747d33fba861e6f3afcc44f145721/pandas/computation/scope.py#L102

@chris-b1 , so yesterday was the first time I had seen this code base, so I am not familiar with the workings of the scope levels.  I did see all of the +1's, and thought it note-worthy, but didn't investigate any further.
Looked at this again, and I'm fine with your fix.  `level` is ultimately passed to `sys._getframe`, so makes sense that we keep incrementing as we go down the call stack.
thanks @stephenrauch !
ENH: .squeeze accepts axis parameter
State: closed

 - [X] closes https://github.com/pandas-dev/pandas/issues/15339
 - [X] tests added / passed
 - [X] passes ``git diff upstream/master | flake8 --diff``
 - [X] whatsnew entry


Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
@kernc Is there an issue for this? 
Can you give some rationale? Or show an example use case.
Thanks. Now it is: https://github.com/pandas-dev/pandas/issues/15339.
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/15335?src=pr&el=h1) Report
> Merging [#15335](https://codecov.io/gh/pandas-dev/pandas/pull/15335?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/153da508a536a3ef203d9cd315c67b5fd3022a51?src=pr&el=desc) will **increase** coverage by `<.01%`.


```diff
@@            Coverage Diff             @@
##           master   #15335      +/-   ##
==========================================
+ Coverage   86.32%   86.32%   +<.01%     
==========================================
  Files         141      141              
  Lines       51165    51163       -2     
==========================================
- Hits        44169    44168       -1     
+ Misses       6996     6995       -1
```


| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/15335?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/compat/numpy/function.py](https://codecov.io/gh/pandas-dev/pandas/compare/153da508a536a3ef203d9cd315c67b5fd3022a51...44d3c54edfc5927c584aab63b5e059dc8e594cb4?src=pr&el=tree#diff-cGFuZGFzL2NvbXBhdC9udW1weS9mdW5jdGlvbi5weQ==) | `94.19% <ø> (-0.08%)` | :x: |
| [pandas/core/generic.py](https://codecov.io/gh/pandas-dev/pandas/compare/153da508a536a3ef203d9cd315c67b5fd3022a51...44d3c54edfc5927c584aab63b5e059dc8e594cb4?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZ2VuZXJpYy5weQ==) | `96.33% <100%> (ø)` | :white_check_mark: |
| [pandas/core/common.py](https://codecov.io/gh/pandas-dev/pandas/compare/153da508a536a3ef203d9cd315c67b5fd3022a51...44d3c54edfc5927c584aab63b5e059dc8e594cb4?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvY29tbW9uLnB5) | `91.36% <ø> (+0.33%)` | :white_check_mark: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/15335?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/15335?src=pr&el=footer). Last update [153da50...44d3c54](https://codecov.io/gh/pandas-dev/pandas/compare/153da508a536a3ef203d9cd315c67b5fd3022a51...44d3c54edfc5927c584aab63b5e059dc8e594cb4?el=footer&src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
thanks!
Added a 2  comments on the documentation (see above and also the comment on the merged commit: https://github.com/pandas-dev/pandas/commit/87c2c2af6150d6e8fa8cfbc017fbfd52e7f8c5c7#commitcomment-20800784). 
@kernc would you like to do a follow-up PR?
adding "include" parameter dataframe.from_records
State: closed

 -  closes #15319 
 -  simple adding include parameter for from_records


Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
@miahunsicker always add tests please.
@miahunsicker @jreback can we double check that the `columns` parameter doesn't already do this? 
closing the issue as we already have this feature, but as I said, if you'd like to do a documentation PR with example would be fantastic. thanks!
TST/STYLE: remove multiprocess nose flags and slight PEP fixes
State: closed

https://github.com/pandas-dev/pandas/pull/13856#issuecomment-278058522

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Code Style
description: Code style, linting, code_checks

Comments:
I ended up having to autopep8 things to remove the extra lines when I took out the multiprocessing directive. But seems to pass and all be good. Slight PEP is even on non-qa files (some aspects of them).

@jorisvandenbossche @TomAugspurger @bashtage 
This will not conflict with Tom's pytest PR?
I just did a test rebase on top of this, and didn't have to resolve any conflicts. Haven't looked at the new diff yet though.
@jreback The test files are not tested for PEP8 on travis? And would they pass now with those changes?
i am not 100% sure we are testing ALL files (that's why some things were changed)

everything passes now but let me see if i can add the other files
actually I think the slight peps are to files what we ignore flake on. so pretty innocuous.
ENH: allow timestamp subtraction with different timezones (GH15249)
State: closed

 works on #15249 
passes the relevant tests but fails on few. Those include numpy dependency  libgfortran and some environment issues. The output works just fine.
@jreback 

Labels:
name: Datetime
description: Datetime data dtype
name: Timezones
description: Timezone data dtype

Comments:
If we allow this, we should also fix related ``Index`` and ``Block`` ops.
@Vutsuak16 you apparently changed all of the whitespace, so you have a massive diff. Further the most important thing here is to nail down the tests to figure out what is changing.
can you update
closing this as stale, but if you'd like to add more tests / update pls commentt.
ENH: enabled 'on' keyword for groupby(..).resample()
State: closed

 - [x] closes #15021 
 - [x] tests added / passed
 - [x] passes ``git diff upstream/master | flake8 --diff``
 - [x] whatsnew entry


Labels:
name: Bug
name: Resample
description: resample method

Comments:
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/15326?src=pr&el=h1) Report
> Merging [#15326](https://codecov.io/gh/pandas-dev/pandas/pull/15326?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/34cdfa48881118a6327fe0e599fb41467ef6ffcc?src=pr&el=desc) will **increase** coverage by `<.01%`.


```diff
@@            Coverage Diff             @@
##           master   #15326      +/-   ##
==========================================
+ Coverage   86.32%   86.33%   +<.01%     
==========================================
  Files         141      141              
  Lines       51163    51164       +1     
==========================================
+ Hits        44169    44170       +1     
  Misses       6994     6994
```


| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/15326?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/groupby.py](https://codecov.io/gh/pandas-dev/pandas/compare/34cdfa48881118a6327fe0e599fb41467ef6ffcc...9fc3b4fa45112c1eb69197bb4d9c553546ea5a78?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZ3JvdXBieS5weQ==) | `95.15% <100%> (ø)` | :white_check_mark: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/15326?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/15326?src=pr&el=footer). Last update [34cdfa4...9fc3b4f](https://codecov.io/gh/pandas-dev/pandas/compare/34cdfa48881118a6327fe0e599fb41467ef6ffcc...9fc3b4fa45112c1eb69197bb4d9c553546ea5a78?el=footer&src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
thanks @verhalenn 

changed slightly to put in a more logical place. 

we have had this dichotomy between ``.groupby`` and ``.resample`` where the former uses ``.key`` and latter ``.on``. We had a discussion about this to normalize that, but didn't take any action.

@chrisaycock @jorisvandenbossche 

worth deprecating ``key`` in favor of ``on`` in ``pd.Grouper``? or just leave it (I happen to like ``on`` better as we use this in ``.merge`` as well).
> worth deprecating `key` in favor of `on` in `pd.Grouper`?

Makes sense to me. Consistent parameter names makes things easy.
I would personally just leave this. 
The current 'sounds' also OK: "resample on a column", group by a key" (not "group by on")
ok, so split opinion! going to make an issue in any event.