BrandonFors/pandas-github-issues huggingface dataset:
DOC: avoid StorageOptions type alias in docstrings
State: closed

Small follow-up on https://github.com/pandas-dev/pandas/pull/35655, replacing the "StorageOptions" with a plain "dict" in the docstrings ("StorageOptions" is not something known to users, in the type annotations it will expand but not in the docstrings)

(cc @martindurant)

Labels:
name: Docs

Comments:
I thought I was following style :|
> I thought I was following style :|

In any case, it the first PR adding storage_options, it was not done like that ;)
lgtm
REF: use BlockManager.apply for Rolling.count
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Refactor
description: Internal refactoring of code
name: Window
description: rolling, ewma, expanding

Comments:
Thanks @jbrockmendel. Couldn't find a straightforward simplification immediately, so this is a nice cleanup.
BUG: item_cache invalidation in get_numeric_data
State: closed

- [ ] closes #xxxx
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Bug
name: Internals
description: Related to non-user accessible pandas implementation

Comments:
thanks @jbrockmendel 
whatsnew in 1.1.2. should this be backported and milestone changed or release note moved?

in release readiness I now check whatsnew on master and backport in sync

https://github.com/simonjayhawkins/pandas-release/runs/1078142445?check_suite_focus=true
@meeseeksdev backport 1.1.x
REF: ignore_failures in BlockManager.reduce
State: closed

Moving towards collecting all of the ignore_failures code in one place.

The case where we have object dtypes is kept separate in this PR, will be handled in the next pass.

Labels:
name: Refactor
description: Internal refactoring of code
name: Internals
description: Related to non-user accessible pandas implementation

Comments:
> But so, that also means that the behaviour with this PR depends on the presence of (another) object dtype column or not?

I think it means the behavior in _master_ depends on the presence of (another) object dtpye column.  Which is part of the cluster-frack that is the reason we should rip off the bandaid.
Reverted behavior-changing component so that bug can be fixed separately
lgtm. i know you are trying to cleanup this whole area, so good to go. @jorisvandenbossche any comments (I am sure this is going to be simplified over time)
yah the real simplifications dont come until we do the actual bugfixes, but this is a step in the right direction
great. glad finally to get this in.
REGR: DatetimeIndex.intersection incorrectly raising AssertionError
State: closed

- [x] closes #35876
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

@dsaxton this PR's test and whatsnew note reference the underlying issue you identified in #35876, but not the OP issue.  Do you think we should add a test/note for that directly?

Labels:
name: Datetime
description: Datetime data dtype
name: Regression
description: Functionality that used to work in a prior pandas version
name: Index
description: Related to the Index class or subclasses

Comments:
> @dsaxton this PR's test and whatsnew note reference the underlying issue you identified in #35876, but not the OP issue. Do you think we should add a test/note for that directly?

I personally think it's fine to test / note only the root issue since the OP wasn't minimal (makes for cleaner / better tests IMO). Did a quick sanity check and this does fix the groupby issue also.
thanks @jbrockmendel and @dsaxton
COMPAT: Ensure rolling indexers return intp during take operations
State: closed

- [x] closes #35294
- [x] closes #35148
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Tested locally with @TomAugspurger docker container and the tests were working: https://github.com/pandas-dev/pandas/pull/35228#issuecomment-658833239




Labels:
name: Compat
description: pandas objects compatability with Numpy or Python functions
name: Window
description: rolling, ewma, expanding

Comments:
nice!
thanks @mroeschke 
This should backport automatically and fix the failing groupby 32bit tests https://github.com/pandas-dev/pandas/pull/37466#issuecomment-717896083 

after backporting this could also revert #36220
@meeseeksdev backport 1.1.x
BUG: to_dict_of_blocks failing to invalidate item_cache
State: closed

- [ ] closes #xxxx
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Bug
name: Internals
description: Related to non-user accessible pandas implementation

Comments:
doesn't this close an issue?
> doesn't this close an issue?

Not that I know of.  to_dict_of_blocks is only used on one place outside of `pandas._testing`, and that was just made removable in #35861
was thinking of https://github.com/pandas-dev/pandas/issues/35739 (but likely this is comething else)
REF: simplify latex formatting
State: closed

- [x] closes https://github.com/pandas-dev/pandas/issues/35790
- [x] tests added ``pandas/tests/io/formats/test_latex.py``
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`

Refactor ``to_latex`` using polymorphism and the builder design pattern.

Polymorphism
--------------

Previously there was a complicated logic in multiple methods
for either longtable or regular table.
This PR implements various builders:
  - ``RegularTableBuilder``
  - ``LongTableBuilder``
  - ``TabularBuilder``

Selection of the appropriate builder is carried out in ``LatexFormatter``,
depending on the kwargs provided.

Each builder must implement construction of all the table's components:
  - Beginning and end of the environment,
  - Separators (top, middle, bottom),
  - Header,
  - Body.

This allows one eliminate complex logic throughout multiple methods.

Separate Concerns
------------------

Separated concerns: row formatting into string (separate class ``RowStringConverter``)
is carried out independently of the table builder.
Derived from ``RowStringConverter`` are two classes used to iterate over headers or body of the tables:
  - ``RowHeaderIterator``
  - ``RowBodyIterator``

Extract Methods
----------------

Multiple methods were extracted to improve code readability.


Labels:
name: Refactor
description: Internal refactoring of code
name: IO LaTeX
description: to_latex

Comments:
What is not covered by the tests are the abstract methods.

```
    @property
    @abstractmethod
    def env_end(self):
        pass
```
I think that the best thing is to setup coverage in such a way to ignore simple pass in the function.
Alternatively I can simply put a docstring there, which will be considered by the bot as a check.

The methods removed (actually one, ``_write_tabular_begin``) and renamed (``_print_cline`` -> ``_compose_cline``) are internal implementation. Do I need to keep them and mark deprecated? They are not used anywhere else.

Also, do you want me to add docstrings for the internal methods? Or only public ones?
I mean, it is probably not reasonable to add docstrings to ``_escape_symbols`` or ``_convert_to_bold``.
Unless required.
@jreback, is there anything else to be done from my side? It seems that I covered all the points you raised previously.
Hello @ivanovmg! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:







There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-09-06 17:57:25 UTC
Can anybody restart the task Windows py37_np16?
It is unlikely that the failure in ``TestPandasContainer.test_to_s3`` is anyhow related to the present PR.
thanks @ivanovmg for this, very nice!
REF: implement Block.reduce for DataFrame._reduce
State: closed

This lets us avoid reconstructing results from a dict, makes it feasible to use the same block-skipping code for DataFrame._reduce that we use for cython_agg_blocks and apply_blockwise.

Labels:
name: Refactor
description: Internal refactoring of code
name: Internals
description: Related to non-user accessible pandas implementation

Comments:
sure
DOC: Fix code of conduct link
State: closed

closes #35855


Labels:
name: Docs

Comments:
LGTM - does this now work if you build the docs locally? 
> LGTM - does this now work if you build the docs locally?

Yes! Built via ./pandas_web.py pandas. Here is my best attempt at test evidence (note on hover showing corrected link in bottom left)

<img width="1327" alt="Screen Shot 2020-08-22 at 4 18 57 PM" src="https://user-images.githubusercontent.com/69096687/90965117-51af7480-e493-11ea-89bd-87f8186c1eb2.png">

I too built this locally and can confirm it works - thanks @21CSM, and welcome!
BUG: Index.get_slice_bounds does not accept datetime.date or tz naive datetime.datetimes
State: closed

- [x] closes #35690
- [x] closes #34077
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

This method appeared under tested so added some additional tests for numeric and object `Index`


Labels:
name: Bug
name: Datetime
description: Datetime data dtype

Comments:
Just commented on #35690, the upshot of which is that I think the current behavior is correct.
Hello @mroeschke! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:









There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-09-02 05:43:44 UTC
lgtm @mroeschke 

ideally we *also* test via ```.loc[]``` for these behaviors (we likey have some already tested); but as a followup is fine.
See comment [above](https://github.com/pandas-dev/pandas/pull/35848#issuecomment-679397568), this PR should not be merged.
CI: Revert 31323 for deprecation warning from Jedi
State: closed

- [x] closes #31407

ipython/ipython#12102 has been solved, and the new release came out a couple weeks ago, so revert should be fine according to https://github.com/ipython/ipython/issues/12102#issuecomment-670987484


Labels:
name: Dependencies
description: Required and optional dependencies

Comments:
emm, seems with the latest version of ipython (7.17.0) and jedi (0.17.2), the deprecation warning is still there.

I converted this PR to WIP, and will ask in ipython issue again about this.
Is this still active?
emm, it is blocked by an ipython issue. my previous issue was closed ipython/ipython#12102 however, it wasn't being solved in the later version of ipython

there is a new issue  https://github.com/ipython/ipython/issues/12597 now tracking this warning issue, not sure if the fix will be included a newer version.
what is the status here?
@charlesdong1991 can you merge master and ping on green
we have a *new* jedi issue, but this one is moot.
REF: simplify _cython_agg_blocks
State: closed

Orthogonal to #35839, though a rebase will be needed.

cc @TomAugspurger did you already try this in #31616?  If so, we need to identify a test case in which this doesnt work

Labels:
name: Refactor
description: Internal refactoring of code
name: Groupby

Comments:
No, I don't think I tried anything like this.
thanks, yet another reason to do the these ops in internals ....
Fix Series construction from Sparse["datetime64[ns]"]
State: closed

- [x] closes #35762
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

Labels:
name: Bug
name: Datetime
description: Datetime data dtype
name: Sparse
description: Sparse Data Type
name: Constructors
description: Series/DataFrame/Index/pd.array Constructors

Comments:
LGTM
thanks @dsaxton 
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
$ git checkout 1.1.x
$ git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
$ git cherry-pick -m1 d0ca4b347b060bc4a49b2d31a818d5a28d1e665f
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
$ git commit -am 'Backport PR #35838: Fix Series construction from Sparse["datetime64[ns]"]'
```

4. Push to a named branch :

```
git push YOURFORK 1.1.x:auto-backport-of-pr-35838-on-1.1.x
```

5. Create a PR against branch 1.1.x, I would have named this PR:

> "Backport PR #35838 on branch 1.1.x"

And apply the correct labels and milestones.

Congratulation you did some good work ! Hopefully your backport PR will be tested by the continuous integration and merged soon!

If these instruction are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
                
Manual backport PR: https://github.com/pandas-dev/pandas/pull/35915
Update SparseDtype user guide doc
State: closed

Tiny doc nit. The doc says you can pass both dtype and fill_value to SparseDtype then only passes one in the example.

Labels:
name: Docs
name: Sparse
description: Sparse Data Type

Comments:
The user guide already shows an example on the next line of passing a fill value though - perhaps, instead of making both examples pass a fill value, the wording could be changed so that the first example shows construction with just the dtype of the non-sparse values. Something like

```
A :class:`SparseDtype` may be constructed by passing the former

.. ipython:: python
   pd.SparseDtype(np.dtype('datetime64[ns]'))

, in which case the default fill value for a given NumPy dtype is the "missing" value for that dtype. Note that it may be overridden by passing a `fill_value`, for example

.. ipython:: python
   pd.SparseDtype(np.dtype('datetime64[ns]'),
                  fill_value=pd.Timestamp('2017-01-01'))
```
Good point @MarcoGorelli. Actually I changed some other wording as well (it's not entirely true that a missing value indicator will be used for numpy dtypes, e.g., not for integers)

```python
In [1]: import numpy as np                                                                                                                                                                                   

In [2]: import pandas as pd                                                                                                                                                                                  

In [3]: pd.SparseDtype(np.int64)                                                                                                                                                                             
Out[3]: Sparse[int64, 0]
```
Thanks @dsaxton. The CI failure was unrelated

```
Skipping uploading of coverage data.

=================================== ERRORS ====================================
_____ ERROR at setup of TestParquetPyArrow.test_s3_roundtrip_explicit_fs ______
```
MAINT: Manual Backport PR #35825 on branch 1.1.x
State: closed

Manual backport of below

Ref: https://github.com/pandas-dev/pandas/pull/35828#issuecomment-677917854

This is because we dropped 3.6 support on master (https://github.com/pandas-dev/pandas/pull/35214)

cc @simonjayhawkins 


Labels:
name: Admin
description: Administrative tasks related to the pandas project
name: CI
description: Continuous Integration

Comments:
conda compat issues - retrying...  
https://travis-ci.org/github/pandas-dev/pandas/jobs/719780475

Might need a lower version to fix this on 3.6 builds trying 0.13
Thanks @alimcmaster1
Backport PR #35757 on branch 1.1.x (CI: Unpin Pytest + Pytest Asyncio Min Version)
State: closed

Backport PR #35757: CI: Unpin Pytest + Pytest Asyncio Min Version

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: CI
description: Continuous Integration

Comments:
CI should pass after we backport: https://github.com/pandas-dev/pandas/pull/35835
Jsonlines append mode
State: closed

**Description**: Adding support to append mode for `to_json` when 'orient' is `records` and 'lines' is `True`

**Motivation**: [jsonlines](http://jsonlines.org/) is a format that is gaining some space (e.g. [Cloud Storage](https://cloud.google.com/bigquery/docs/loading-data-cloud-storage-json)). In pandas It's achieved using `to_json` with `orient='records'` and `lines=True`. But `to_json` doesn't have the option to use `mode='a'` to append to a file, this PR aims to provide a simple append mode for jsonlines using pandas.

---

- [X] closes #35849
- [X] tests added / passed
- [X] passes `black pandas`
- [X] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

Labels:
name: Enhancement
name: IO JSON
description: read_json, to_json, json_normalize
name: Mothballed
description: Temporarily-closed PR the author plans to return to

Comments:
This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.
> This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.

I'm moving to another country and It's been hard to find time to work on the proposed changes but please don't close this PR, I'll update as soon as I find time to work on It again.
> > This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.
> 
> I'm moving to another country and It's been hard to find time to work on the proposed changes but please don't close this PR, I'll update as soon as I find time to work on It again.

No problem! We'll keep this open
@arw2019 is there a reference on proper way of adding tests on pandas?
> @arw2019 is there a reference on proper way of adding tests on pandas?

Not really. Take a look at similar tests in the `pandas/tests/io/json` for a general idea of the format. Typically the pattern we use is:
- construct result
- construct expected result
- assert the two are equal

 If you push an attempt we'll give you detailed pointers
This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.
Sorry, I'm not having time to work on this and probably will not be able to do for a while. I'll close the PR to give the chance of other people to work on this. Anyone feel free to fork my repo and continue
DOC: Start 1.1.2
State: closed


Labels:
name: Docs

Comments:
```
/home/runner/work/pandas/pandas/doc/source/whatsnew/v1.1.1.rst:70: WARNING: Cannot find contributors for range 'v1.1.0..v1.1.1': Cmd('git') failed due to: exit code(128)
  cmdline: git log --grep=Co-authored --pretty=%b v1.1.0..v1.1.1
  stderr: 'fatal: ambiguous argument 'v1.1.0..v1.1.1': unknown revision or path not in the working tree.
```
> ```
> /home/runner/work/pandas/pandas/doc/source/whatsnew/v1.1.1.rst:70: WARNING: Cannot find contributors for range 'v1.1.0..v1.1.1': Cmd('git') failed due to: exit code(128)
>   cmdline: git log --grep=Co-authored --pretty=%b v1.1.0..v1.1.1
>   stderr: 'fatal: ambiguous argument 'v1.1.0..v1.1.1': unknown revision or path not in the working tree.
> ```

only just pushed the tag. will restart github actions and should be OK
known test failures on travis-37-locale
hmm, marking as ready for review has re-triggered azure build.
@jbrockmendel I think now OK to merge if holding up other PR changes.
thanks @simonjayhawkins
Backport PR #35809 on branch 1.1.x (CI: more xfail failing 32-bit tests)
State: closed

Backport PR #35809: CI: more xfail failing 32-bit tests

Labels:
name: CI
description: Continuous Integration
name: 32bit
description: 32-bit systems

Comments:
all test passing except known pyarrow related fails on travis-36-locale
WEB: Add new Maintainers to Team Page
State: closed

Add new maintainers.

cc:
@bashtage
@charlesdong1991
@Dr-Irv
@dsaxton
@MarcoGorelli
@rhshadrach

In case anyone doesn't want to appear here: https://pandas.pydata.org/about/team.html

Labels:
name: Docs

Comments:
thanks @alimcmaster1
CI: more xfail failing 32-bit tests
State: closed

xref #35294 and https://github.com/pandas-dev/pandas/issues/35489#issuecomment-676550757

Labels:
name: CI
description: Continuous Integration
name: 32bit
description: 32-bit systems

Comments:
/azp run
<samp>
Azure Pipelines successfully started running 1 pipeline(s).<br>

</samp>
@jreback the failures here are the known pyarraow releated failures. ok to merge (and backport)

This should fix some of the nightly build failures (although I'm not sure why strict is not used for the existing 32bit xfails, maybe doesn't fail on XPASS on MacPython?) The other nightly build related failures (MacOS builds) should be fixed by https://github.com/MacPython/pandas-wheels/pull/94

This is not strictly necessary (but I think cleaner) see https://github.com/MacPython/pandas-wheels/pull/93#issuecomment-676600874

Moreless ready to tag release (hoping to update #35801 with today's date at some point today) 
sure go for it
@meeseeksdev backport 1.1.x
DOC: another pass of v1.1.1 release notes
State: closed

https://pandas.pydata.org/docs/dev/whatsnew/v1.1.1.html

Labels:
name: Docs

Comments:
it can hang around for a bit for any other comments (and maybe wait to update the release date) as there are issues with building wheels so release is blocked at the moment.
tests passing except for known fails on travis-37-locale
TST: resample does not yield empty groups (#10603)
State: closed

- [x] closes #10603
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

This adds a test to verify that resample does not yield empty groups.

I'm new to contributing to pandas, and I'm not sure where to add this test.
Let me know if this place is inappropriate.

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: Resample
description: resample method

Comments:
If you merge master CI failures should be resolved
thanks @tkmz-n very nice