BrandonFors/pandas-github-issues huggingface dataset:
ENH: Support multi row inserts in to_sql when using the sqlite fallback
State: closed

Currently we do not support multi row inserts into sqlite databases
when `to_sql` is passed `method="multi"` - despite the documentation
suggesting that this is supported.

Adding support for this is straightforward - it only needs us
to implement a single method on the SQLiteTable class and so
this PR does just that.

- [x] closes #29921
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: IO SQL
description: to_sql, read_sql, read_sql_query

Comments:
@jorisvandenbossche 
@simongibbons does this improve performance?
@WillAyd 

> @simongibbons does this improve performance?

It can do in some cases e.g.

```python
In [1]: import sqlite3                                                                                 

In [2]: import pandas as pd                                                                            

In [3]: db = sqlite3.connect(":memory:")                                                               

In [4]: df = pd.DataFrame({"x": range(10000), "y": range(10000)})                                      

In [5]: %timeit df.to_sql("df", db, if_exists="replace")                                               
15 ms ± 87.2 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

In [6]: %timeit df.to_sql("df", db, if_exists="replace", method="multi", chunksize=100)                
11.6 ms ± 56.2 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
```

tbh I would have been happy with closing the issue with a more informative exception. But given how easy it is to support I went with that approach.
Ping, would be nice to get this in before there is another merge conflict with whatsnew.
@simongibbons Thanks a lot! (and sorry it took so long)
I just upgraded from 0.25.3 to 1.0.3 thinking the `method` keyword would solve my problem, but I still have the same error message when I put the `method` keyword to `multi`:
```
df.to_sql("myTable", con, if_exists='append', method="multi")
Traceback (most recent call last):
  File "<input>", line 1, in <module>
  File "C:\Appl\Python38-32\lib\site-packages\pandas\core\generic.py", line 2653, in to_sql
    sql.to_sql(
  File "C:\Appl\Python38-32\lib\site-packages\pandas\io\sql.py", line 512, in to_sql
    pandas_sql.to_sql(
  File "C:\Appl\Python38-32\lib\site-packages\pandas\io\sql.py", line 1734, in to_sql
    table.insert(chunksize, method)
  File "C:\Appl\Python38-32\lib\site-packages\pandas\io\sql.py", line 755, in insert
    exec_insert(conn, keys, chunk_iter)
  File "C:\Appl\Python38-32\lib\site-packages\pandas\io\sql.py", line 679, in _execute_insert_multi
    conn.execute(self.table.insert(data))
TypeError: insert expected 2 arguments, got 1
```
Instead, I now do it manually and it's a bit faster than trying to play with the `chunksize `parameter of `.to_sql`: 
```
con.executemany('INSERT INTO myTable VALUES (?, ?, ?, ?, ?, ?)', df.to_records().tolist())
con.commit()
```
DOC: Capitalize the 'p' in 'pandas code style guide'
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:

Comments:
I think this is intentionally lowercase
Closing as I think just a matter of convention, but can be reopened if anyone disagrees

Thanks in any case @MomIsBestFriend  - appreciate all your contributions
> I think this is intentionally lowercase

I was the one who wrote this file :smiley:
and It's not on purpose I can guarantee you.
> appreciate all your contributions

Thank you, means a lot!
CI: Disallow bare pytest raise
State: closed

- [x] closes #23922
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: CI
description: Continuous Integration

Comments:
Closing in favor in #30755
Is it possible to do this via regex? similar to the current code check for context manager:
https://github.com/pandas-dev/pandas/blob/master/ci/code_checks.sh#L157
TYP: _config/config.py && core/{apply,construction}.py
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
Thanks @MomIsBestFriend
Fix read_json category dtype
State: closed

- [X] closes #21892
- [ ] tests added / passed
- [X] passes `black pandas`
- [X] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] Add support for "category" dtype in read_json


Labels:
name: IO JSON
description: read_json, to_json, json_normalize

Comments:
Hello @taoufik07! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:





There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-01-06 12:30:52 UTC
Hi @taoufik07  - sorry to chase you up, just wanted to ask whether you're still working on this :)
Hey @MarcoGorelli, sorry I'm kinda busy right now, yes I would love to keep working on this, I'll try finish the tests in the couple days
Thanks - no hurry, just checking it's not gone stale
@taoufik07 closing as stale. ping if you want to continue.
REF: Create test_encoding file for CSV
State: closed

This is 99.99% copy and paste

Labels:
name: Refactor
description: Internal refactoring of code
name: Testing
description: pandas testing functions or related to the test suite
name: IO CSV
description: read_csv, to_csv

Comments:
On board with the concept. What's the bit that isn't copy / paste?
> What's the bit that isn't copy / paste?

* `BytesIO` isn't used in `test_common.py` anymore
* The module documentation in `test_encoding.py` is new

The 99.99% was a very pedantic figure so to speak. 🙂 
thanks @gfyoung
BUG: PeriodArray comparisons inconsistent with Period comparisons
State: closed

The second of two non-cosmetic changes mentioned in #30720.

Labels:
name: Bug
name: Algos
description: Non-arithmetic algos: value_counts, factorize, sorting, isin, clip, shift, diff
name: Period
description: Period data type

Comments:
does this slightly change conversion semantics? e.g. the bug part, if so can you add a whatsnew
thanks, IIRC you mentioned that we were going to deprecate comparing * raw strings* to Periods as well?
> IIRC you mentioned that we were going to deprecate comparing * raw strings* to Periods as well?

no such plans.  with this in im planning to share the DTA/TDA/PA comparisons, then fix some inconsistencies in searchsorted (so that DTI/TDI/PI can dispatch to their EAs)
Make DTA _check_compatible_with less strict by default
State: closed

One of the two non-cosmetic things mentioned in #30720.

There are a bunch of places where DTA or DTI do a compatibility check that for tz_awareness_compat, but not requiring the same tz.  This check is analogous to `PeriodArray._check_compatible_with` and `TimedeltaArray._check_compatible_with`, so this adds a kwarg to _check_compatible_with so that we can use _check_compatible_with in all the relevant places and subsequently de-duplicate a bunch of code.

In addition to the comparisons, this is going to be relevant for searchsorted and insert, where we have slightly different behavior in a bunch of EA/Index subclasses.

Labels:
name: Datetime
description: Datetime data dtype
name: Algos
description: Non-arithmetic algos: value_counts, factorize, sorting, isin, clip, shift, diff

Comments:
updated+green
REF: cosmetic differences between DTA/TDA/PA comparison methods
State: closed

There are 2 remaining non-cosmetic differences between these methods remaining, which will be the subjects of upcoming PRs.  Once those are addressed, we'll be able to de-duplicate the methods completely.

Labels:
name: Refactor
description: Internal refactoring of code
name: Datetime
description: Datetime data dtype
name: Algos
description: Non-arithmetic algos: value_counts, factorize, sorting, isin, clip, shift, diff

Comments:
thanks @jbrockmendel
Consistent json error
State: closed

This will take a few passes to do comprehensively, but trying to clean up error handling in the extension module. Right now failure points are allowed, which can lead to segfaults or surprising behaviour when debugging.

Trying to push closers towards the CPython error handling conventions:

https://docs.python.org/3/c-api/exceptions.html

So basically:

  - Explicitly check for NULL from most C API functions except integer returning functions (where they return -1 and set PyErr) AND
  - Don't set error messages from failing functions, as they should have already set it

Labels:
name: Error Reporting
description: Incorrect or improved errors from pandas
name: IO JSON
description: read_json, to_json, json_normalize

Comments:
@gfyoung as an alternate I think could set any of these to NULL and use a goto to Py_XDECREF on these. I'll give that a look over the next few days
Come back to this one later; clearing queue for now
REF: move sharable methods to ExtensionIndex
State: closed


Labels:
name: Clean

Comments:
Is the goal to provide an index class that can work with arbitrary EAs? If so, I think that ExtensionIndex should only be using methods and attributes defined on ExtensionArray. In particular, I don't think that `_ndarray_values` is part of the interface.
> I don't think that _ndarray_values is part of the interface.

I guess this is ambiguous.  the base EA class does have _ndarray_values, but the docstring says "This method is not part of the pandas interface"
Mmm, that's a bit tricky then. Perhaps this will help us pin down some of the required semantics on `_ndarray_values`.

Longer-term, what are you plans for ExtensionIndex? Will it be a base class that 3rd parties can inherit from and customize?
> Longer-term, what are you plans for ExtensionIndex? Will it be a base class that 3rd parties can inherit from and customize?

That's an option.  Ideally I'd like to keep the customization inside the EAs.  ATM I'm working on smoothing out the small differences between e.g. DatetimeIndex.searchsorted vs DatetimeArray.searchsorted so we can both delegate more and improve internal consistency.
Sounds good. Happy to continue finding common methods and develop on interface out of that.

Can you check the ASVs for our EA-backed indexes on this branch?
It looks like IntervalIndexing.time_getitem_list and time_loc_list are significantly slower, will look into this
Looks like the slowdown was caused by using `IntervalIndex.__new__` instead of `IntervalIndex._shallow_copy`.  ATM there is a problem with DTA/TDA _shallow_copy that prevent us from using that in base class, but ill be pushing a fix for that before long
can you rebase
thanks @jbrockmendel
IntervalArray equality follow-ups 
State: closed

Follow-ups to #30640 based on @jbrockmendel's comments.

Haven't addressed all the comments yet but pushing this up now so there's a record of it.

Changes thus far:
- Created `tests/arithmetic/test_interval.py ` and moved the tests there
- Used `make_wrapped_comparison_op` to add `__eq__` and `__ne__` to `IntervalArray`.

Labels:
name: Interval
description: Interval data type

Comments:
LGTM
Thanks @jschendel 
@jschendel could IntervalIndex also pass through inequalities to IntervalArray?  if so then thats some more code we can share
Yes, although inequalities haven't been implemented yet, even at the scalar `Interval` level.  Still would be good to get the code to share in place though, as the error messages are inconsistent.
Datetime error message (unable to continue that PR)
State: closed

- [ ] closes #10720
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`


Labels:
name: Datetime
description: Datetime data dtype
name: Error Reporting
description: Incorrect or improved errors from pandas
name: Timedelta
description: Timedelta data type

Comments:
Tests passed locally, but failed in Ci.
On some platforms
1) There's extra **%s** flag:
`AssertionError: Pattern "Unknown string format: Month 1, 1999. You can coerce to NaT by passing errors='coerce'" not found in "Unknown string format: %s Month 1, 1999. You can coerce to NaT by passing errors='coerce'"
`
2) or date is missing at all:
`AssertionError: Pattern "Unknown string format: Month 1, 1999. You can coerce to NaT by passing errors='coerce'" not found in "Unknown string format. You can coerce to NaT by passing errors='coerce'"`
Trying to figure it out...
`str(e)` doesn't pass even locally. Output looks like: 
`"('Unknown string format:', 'Month 1, 1999'). You can coerce to NaT by passing errors='coerce'"`
Tried to use list comps. Doesn't help too.
ping @WillAyd 
ping №2: @jbrockmendel
@baevpetr ci is not passing
CI: Fix numpydev build
State: closed

- [x] closes #30709

hmm.. im not sure on this. But don't see any usage of this ret value.

The C extensions now build with this change

Potentially related change on the numpy side:
https://github.com/numpy/numpy/pull/15232

cc. @WillAyd @jreback 

Labels:
name: CI
description: Continuous Integration
name: Dependencies
description: Required and optional dependencies

Comments:
LGTM, thanks for tracking this down
this is fine

that return value is from a really old version
CI: Test case for wrong placed space
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

Test case for:
```python
foo = (
            "bar"
            " baz"
)
```

Labels:
name: CI
description: Continuous Integration
name: Code Style
description: Code style, linting, code_checks

Comments:
Closing in favor of #30755
STY: Spaces over concat strings - batch 1
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
I personally prefer the spaces at the start of the strings. I find it easier to spot missing spaces. not sure what others think.
No strong preference but I think OK. @MomIsBestFriend can you merge master?
Thanks @MomIsBestFriend
CLN: replacing '.format' with f-strings in various files
State: closed

- [x]  contributes to #29547
- [ ] tests added / passed
- [X] passes `black pandas`
- [X] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
Hello @AlfredoGJ! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:

















There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-01-06 16:14:06 UTC
thanks @AlfredoGJ
CLN: Replace fstring in tests/groupby/*.py files 
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [x] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

Ref to #29547

Labels:
name: Code Style
description: Code style, linting, code_checks
name: Clean

Comments:
thanks @Sangarshanan
Fix integer check; also add column with integer name in test case.
State: closed

A small embarrassing mistake is corrected here that was introduced in the recently merged #28215 

Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
Looks like this fixes CI on master - running `my pandas` - currently seeing:
```
pandas/core/generic.py:464: error: Non-overlapping identity check (left operand type: "str", right operand type: "Type[int]")
```
thanks for he quick fix @hwalinga
Implement PeriodIndex.difference without object-dtype cast
State: closed

Analogous to #30666.  After this we'll be able to share some code between the set operations.

The edit in pd._testing is mostly unrelated.  Both are motivated by tracking down the places where object-dtype ndarray is passed to PeriodIndex._shallow_copy.

Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Period
description: Period data type

Comments:
lgtm ping on green.
thanks
CI: Fix pytest junit_family warnings
State: closed

- [x] closes #30433

As per https://docs.pytest.org/en/latest/deprecations.html#junit-family-default-value-change-to-xunit2

This will mean we produce xml output with xsd as per: (opposed to old legacy v1)
https://github.com/jenkinsci/xunit-plugin/blob/xunit-2.3.2/src/main/resources/org/jenkinsci/plugins/xunit/types/model/xsd/junit-10.xsd

Done following check
- [x] Warning no longer present in builds -> checked [here](https://travis-ci.org/pandas-dev/pandas/jobs/632732296?utm_medium=notification&utm_source=github_status)
- [x] CodeCov unaffected since it uses the output xml -> [here](https://codecov.io/gh/pandas-dev/pandas/compare/b29d58d1a3e0ebb1ed3bf3bed1e1cf4848092c77...2a2d21a14a78e028e8e9a9c3bbf815ea76501036)


Labels:
name: CI
description: Continuous Integration

Comments:
lgtm
thanks @alimcmaster1
ERR: Improve error message and doc for invalid labels in cut/qcut
State: closed

- [x] closes #13318
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] add whats new note



Labels:
name: Docs
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Error Reporting
description: Incorrect or improved errors from pandas

Comments:
Hello @ryankarlos! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:







There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-01-07 17:48:37 UTC
Looks like you already made most of these changes in this PR: https://github.com/pandas-dev/pandas/pull/30615/files ?
looks like this pickup up your other changes.
sorry my mistake - just rebased with master now
hmm mypy seems to be giving:

```
Performing static analysis using mypy
pandas/core/generic.py:464: error: Non-overlapping identity check (left operand type: "str", right operand type: "Type[int]")
Found 1 error in 1 file (checked 842 source files)
Performing static analysis using mypy DONE
##[error]Process completed with exit code 1.
```
Need to wait for this to be merged: https://github.com/pandas-dev/pandas/pull/30698
Looks like im getting an unrelated compiler error in one of the builds - can see its throwing errors in other PRs as well 

```
gcc -pthread -shared -B /home/vsts/miniconda3/envs/pandas-dev/compiler_compat -L/home/vsts/miniconda3/envs/pandas-dev/lib -Wl,-rpath=/home/vsts/miniconda3/envs/pandas-dev/lib -Wl,--no-as-needed -Wl,--sysroot=/ build/temp.linux-x86_64-3.7/pandas/io/sas/sas.o -o /home/vsts/work/1/s/pandas/io/sas/_sas.cpython-37m-x86_64-linux-gnu.so
error: command 'gcc' failed with exit status 1
##[error]Bash exited with code '1'.
Finishing: Setup environment and build pandas
```
Yeah the numpy dev build will fail if any warnings are raised.
Due to this flag https://github.com/pandas-dev/pandas/blob/master/ci/azure/posix.yml#L59

Hopefully someone will have a look soon..
thanks @ryankarlos
TYP: NDFrame.(loc|iloc|at|iat)
State: closed

Currently, the NDFrame indexers (.loc/.iloc/.at/.iat) are too complex set up to let mypy understand them. This PR makes their implementations understandable for mypy (and humans also maybe, the old imp. was a bit indirect so took som effort to understand).

One issue is that I  - for the life of me - can't programatically set doc strings of properties and make mypy understand the properties' types at the same time. Python/mypy seem to demand that doc strings be set directly on the property if the type is to be understandable by mypy. E.g. Appender on a property is not supported by mypy. So I've set the doc string on the NDFrame properties, and reference them from the indexer classes instances. This makes the PR seem large, but it's mostly just moving the doc strings.


Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
The failure is unrelated and will be fixed in #30698. I can reun the tests when #30698 is merged.
thanks @topper-123 very nice
CI: Fix IPython Tab Completion test async warning
State: closed

- [x] closes #29070
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`

- New PR since https://github.com/pandas-dev/pandas/pull/29087 went stale


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Output-Formatting
description: __repr__ of pandas objects, to_string

Comments:
thanks @alimcmaster1
CLN: share compatibility-check code
State: closed


Labels:
name: Period
description: Period data type
name: Clean

Comments:
> what is the advantage of having raise_on_incompatible return an exception, just to always raise it?

Among other things, mypy has an easier time telling that we have raised.  This is also the pattern we've moved to in Index
AFAIK you could just type the function as `NoReturn` (though maybe not available until 3.6.4)

No preference on my end between raise and return so lgtm
ok thanks
PLT: Add tests for missing markers
State: closed

closes #14563 
closes #14958 
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Visualization
description: plotting

Comments:
Thanks @charlesdong1991
Fix PeriodIndex.get_indexer with non-PI
State: closed

I'm _pretty_ sure that for non-comparable we should be returning an array of -1s, not raising.  Can you double-check me on that @jreback?


Labels:
name: Period
description: Period data type
name: Index
description: Related to the Index class or subclasses

Comments:
> was this not tested before?

Most of the existing tests pass a PeriodIndex.  There is one existing test that passes a DatetimeIndex to PeriodIndex.get_indexer_nonunique (which currently incorrectly calls asi8 before doing a freq check).
k thanks
TYP: Add mypy as a pre-commit
State: closed

This is quite helpful in developing with typing, especially if you plan to update the mypy version.

- [ ] closes #xxxx
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
thanks @xhochy