BrandonFors/pandas-github-issues huggingface dataset:
BUG: to_csv output formatting for datetimes
State: closed

closes #29711 

…ssue and provides rounding variable to csv formatter to handle datetimes differently. I decided to try and tackle this issue for one of the classes I am in at university. The issue seems to be stemming from the fact that BlockManager is grouping together two differently formatted datetime values in the DataFrame thus making it so the second datetime value is formatted like the first(leading to the trailing zeroes in the example). If you want to format all datetimes the same in the csv you can simply make use of the date_format attribute which the CsvFormatter takes as a variable to format the datetimes however you like. I decided to add a rounding_milliseconds attribute to the csv formatter which would remove  the milliseconds from the end of the datetime if they are all equal to '0'. 

- [x] tests added / passed
- [x] passes `black pandas`


Labels:
name: Output-Formatting
description: __repr__ of pandas objects, to_string

Comments:
Hello @burkbre! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:







* In the file [`pandas/tests/frame/test_to_csv.py`](https://github.com/pandas-dev/pandas/blob/0b8091ca51ab487867eb2ab30d7512ef8819a5eb/pandas/tests/frame/test_to_csv.py):

> [Line 1365:79](https://github.com/pandas-dev/pandas/blob/0b8091ca51ab487867eb2ab30d7512ef8819a5eb/pandas/tests/frame/test_to_csv.py#L1365): [E231](https://duckduckgo.com/?q=pep8%20E231) missing whitespace after ','
> [Line 1370:89](https://github.com/pandas-dev/pandas/blob/0b8091ca51ab487867eb2ab30d7512ef8819a5eb/pandas/tests/frame/test_to_csv.py#L1370): [E501](https://duckduckgo.com/?q=pep8%20E501) line too long (89 > 88 characters)
> [Line 1388:79](https://github.com/pandas-dev/pandas/blob/0b8091ca51ab487867eb2ab30d7512ef8819a5eb/pandas/tests/frame/test_to_csv.py#L1388): [E231](https://duckduckgo.com/?q=pep8%20E231) missing whitespace after ','
> [Line 1399:89](https://github.com/pandas-dev/pandas/blob/0b8091ca51ab487867eb2ab30d7512ef8819a5eb/pandas/tests/frame/test_to_csv.py#L1399): [E501](https://duckduckgo.com/?q=pep8%20E501) line too long (89 > 88 characters)
> [Line 1405:79](https://github.com/pandas-dev/pandas/blob/0b8091ca51ab487867eb2ab30d7512ef8819a5eb/pandas/tests/frame/test_to_csv.py#L1405): [E231](https://duckduckgo.com/?q=pep8%20E231) missing whitespace after ','
> [Line 1416:89](https://github.com/pandas-dev/pandas/blob/0b8091ca51ab487867eb2ab30d7512ef8819a5eb/pandas/tests/frame/test_to_csv.py#L1416): [E501](https://duckduckgo.com/?q=pep8%20E501) line too long (89 > 88 characters)



##### Comment last updated at 2019-12-10 20:42:15 UTC
pls search the issue queue, IIRC there are a number of issues about this
@burkbre is this still active? As mentioned don't want to add new keywords so would need an alternate approach like the dict argument for `date_format`
Thanks @burkbre for the PR but I think has gone stale. I think also needs a different approach but ping if you'd like to continue and can address comments
Simplified JSON string handling
State: closed

Provides a slight boost to performance as well

```sh
       before           after         ratio
     [52f5fdf1]       [325833e0]
     <master>         <faster-json-str>
-         112±1ms        101±0.7ms     0.90  io.json.ToJSON.time_to_json('index', 'df')

SOME BENCHMARKS HAVE CHANGED SIGNIFICANTLY.
PERFORMANCE INCREASED.
```

Labels:
name: IO JSON
description: read_json, to_json, json_normalize

Comments:
sure
BUG+DEPR: undeprecate item, fix dt64/td64 output type
State: closed

- [x] closes #29250
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

Does the un-deprecation need a dedicated discussion, or is there consensus on that?

I'd be happy to split this into separate bug/depr pieces of reviewers prefer.

Labels:
name: Deprecate
description: Functionality to remove in pandas

Comments:
This issue is https://github.com/pandas-dev/pandas/issues/29250
also needs a rebase
LGTM, assuming that https://github.com/pandas-dev/pandas/pull/30175/files/0d2867d5518ac8e5032e7dfca5d0adb5811fa84d#diff-9c718a39eb63c1bfac4fbeacf2906ffd 
has been addressed.
Thanks @jbrockmendel !

(will look into adding a EA test)
Remove Dead JSON Code
State: closed

I think this function is somewhat misleading as it implies that any ndarray type goes through it, but realistically the flag that sends you through to this function is only set for datetime arrays

Labels:
name: IO JSON
description: read_json, to_json, json_normalize
name: Clean

Comments:
> but realistically the flag that sends you through to this function is only set for datetime arrays

Nice.  Would a comment to this effect be worthwhile?
My goal is to do away with that function entirely; just going to need a few more passes I think
thxs
PERF: Improve perf initalizing DataFrame with a range
State: closed

Performance improvement when initalizing a DataFrame with a range:

```python
>>> %timeit pd.DataFrame(range(1_000_000)) 
347 ms ± 4.46 ms per loop  # master
5.98 ms ± 61.9 µs per loop  # this PR
# other initialization types don't seem to be affected by this problem:
>>> %timeit pd.Series(range(1_000_000))
5.75 ms ± 27.6 µs per loop  # master and this PR
>>> %t pd.DataFrame({"a": range(1_000_000)})
11.2 ms ± 119 µs per loop  # master and this PR
```


Labels:
name: Performance
description: Memory or execution speed performance
name: DataFrame
description: DataFrame data structure

Comments:
Can you check we already have a test for this construction?
nice.  does this need/merit an asv?
> 
> 
> Can you check we already have a test for this construction?

It's tested in ``pandas\tests\frame\test_constructors.py`` line 1079 and used in a few more places.
> nice. does this need/merit an asv?

IMO that would not be really important, though I'm not against it either. This pattern is probably mostly used interactively, in ipython and jupyter etc. and little otherwise.

bugs can certainly be addressed in a followup, but can you add an asv ? I think we have a number of construction asv's already
I've added a ASV. I can take on #30173 in a followup for both Series and Dataframes.
lgtm. merge on green.
STY: Underscores for long numbers
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
thanks @MomIsBestFriend 

yeah agree with the others, this is for reading clarify, so want to use it mainly on long static constants.
STY: Underscores for long numbers
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
lgtm if green
See also the comments in https://github.com/pandas-dev/pandas/pull/30169#discussion_r355711538
yeah, I agree with @jorisvandenbossche (underscores only for 7+ digits), though I like having underscores in tests, as it helps when reading the tests.
Hello @MomIsBestFriend! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:

















There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2019-12-10 11:47:05 UTC
Thanks @MomIsBestFriend
DOC: Cleaned doctrings
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Code Style
description: Code style, linting, code_checks
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
Thanks @MomIsBestFriend
CI: test for file leakage
State: closed

Continue troubleshooting followning #30096. 

Locally I'm seeing some failures sqlite files are leaking, but im not seeing the xls files that are causing problems in the CI.

Labels:
name: CI
description: Continuous Integration

Comments:
xref https://bitbucket.org/openpyxl/openpyxl/issues/832
> I think worth adding a whatsnew as well

Sure.  Also worth questioning if we want to just bump the dependency.  I know 3.0.2 has caused some problems; if we're going to change it, might as well do it in one go.
Looks like the sqlite failures are in test_memory_map and test_repr_html_ipython_config.  the test_memory_map one is fixed by running gc.collect at the end of it, but i dont have a clear fix for the ipython one.  Suggestions?


> Can we move this cleanup to a test fixture used for the relevant tests? And just ensure that it's called for the relevant openpyxl version before check_for_file_leaks is run?

@TomAugspurger not clear on what you're asking for.  Are you looking to execute the fixture for fewer tests (to limit overhead)?  or to make the gc call in the relevant tests instead of in the library code?
> or to make the gc call in the relevant tests instead of in the library code?

That. IIUC, the primary motivator here is tracing down file leaks between tests, rather than patching the behavior of old versions of openpyxl.
> the primary motivator here is tracing down file leaks between tests, rather than patching the behavior of old versions of openpyxl.

I like this line of reasoning.
The min version we have listed for openpyxl is 2.4.8, which was released 2017-05-30.  The earliest version that fixed this bug was 2.5.6, released 2018-08-30.  I think implementing Tom's suggestion and bumping the version (separate PR) would be reasonable here
huh, putting the gc.collect in the fixture doesn't seem to get the job done.  putting it at the end of each test seems hit-and-miss.  
> The min version we have listed for openpyxl is 2.4.8, which was released 2017-05-30. The earliest version that fixed this bug was 2.5.6, released 2018-08-30. I think implementing Tom's suggestion and bumping the version (separate PR) would be reasonable here

I would be +1 on just bumping and call it a day :->
thanks @jbrockmendel 

btw not sure if mention openpyxl version in install.rst, if so can you update it
DOC: Cleaned docstrings
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Docs
name: Code Style
description: Code style, linting, code_checks

Comments:
Thanks @MomIsBestFriend
DOC/STY: Cleaned pandas/_libs/{internals,interval}.pyx
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Code Style
description: Code style, linting, code_checks
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
thanks @MomIsBestFriend
STY: some files in pandas/_libs/
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
Thanks @MomIsBestFriend
DOC: Cleaned pandas/_libs/algos.pyx docstrings
State: closed

- [ ] closes #xxxx
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Docs

Comments:
ping @simonjayhawkins 
Thanks @MomIsBestFriend
TST: DataFrameGroupBy.shift Loses TimeZone in datetime64 columns
State: closed

Adding test case for issue GH30134

- [x] closes #30134
- [x] tests added / passed


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Groupby
name: Timezones
description: Timezone data dtype

Comments:
lgtm ping on green
Thanks @tranctan
ENH: Add numba engine for rolling apply
State: closed

- [x] closes #28987
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Enhancement
name: Dependencies
description: Required and optional dependencies
name: Window
description: rolling, ewma, expanding

Comments:
Random question: do you have benchmarks on this anywhere? It would be good to give guidance on when this is helpful (data sizes, type of user-defined function, ratio of time spent in windowing vs. the UDF, etc.).
@TomAugspurger performance testing is lightly described in https://github.com/twosigma/pandas/pull/29, but good point to have a dedicated section in the documentation describing when the numba engine would be advantageous 
Hello @mroeschke! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:



















There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2019-12-26 18:15:58 UTC
@jreback ready for a review.
Huh for some reason I was lucky enough to get https://github.com/pandas-dev/pandas/pull/30370/files#r360319942
Code check failures are related to https://github.com/pandas-dev/pandas/pull/30370/files#r361222880
I can submit a PR to fix this
```
Check for non-standard imports
##[error]pandas/io/stata.py:12:from collections.abc import Iterator
##[error]pandas/io/sas/sas7bdat.py:16:from collections.abc import Iterator
##[error]pandas/io/sas/sas_xport.py:10:from collections.abc import Iterator
##[error]pandas/io/common.py:5:from collections.abc import Iterator
##[error]pandas/io/parsers.py:6:from collections.abc import Iterator
##[error]pandas/io/json/_json.py:2:from collections.abc import Iterator
```

PR here: https://github.com/pandas-dev/pandas/pull/30455
This looks really nice.  way to go
Thanks for the review everyone. Think I addressed all the comments.
Thanks for another round of review. Addressed all the comments.
Looks good. 
thanks @mroeschke very nice!
@mroeschke heads up that im now seeing this in test output locally:

```
pandas/tests/window/test_numba.py::TestApply::test_cache[False-True-False-False]
  /Users/bmendel/Desktop/pd/dtmean/pandas/core/window/numba_.py:75: NumbaPerformanceWarning: 
  The keyword argument 'parallel=True' was specified but no transformation for parallel execution was possible.
  
  To find out why, try turning on parallel diagnostics, see http://numba.pydata.org/numba-doc/latest/user/parallel.html#diagnostics for help.
  
  File "pandas/core/window/numba_.py", line 59:
  
              def impl(window, *_args):
              ^
  
    result[i] = numba_func(window, *args)
```
I'll make a followup to suppress this warning (no baring on what is being test).
Modify hash(pd.NA) to avoid integer hash collisions
State: closed

- [X] closes #30013
- [X] tests added / passed
- [X] passes `black pandas`
- [X] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`

No whatsnew entry since this issue was never user facing.

Labels:
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate

Comments:
ping @jreback
Fix NoneType error when pulling non existent field
State: closed

If normalizing a jsonstruct a field can be absent
due to a schema change.

- [X] closes #30148
- [X] tests added / passed
- [X] passes `black pandas`
- [X] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [X] whatsnew entry


Labels:
name: IO JSON
description: read_json, to_json, json_normalize

Comments:
Can you add test(s)? Typically the first thing we look for in a PR
@WillAyd of course. Test has been added.
Thanks for the test. I don't really agree with the change though - why wouldn't you want this to raise a `TypeError`?
Imagine if I have 3 records:
```
[{'state': 'Florida', 'info': [1, 2,]}]
[{'state': 'Texas', 'info': None}]
[{'state': 'California', 'info': [2,3,]}]
```

I cannot use json_normalize now with `record_path=['info']` as the field for the second record is None. That's perfectly valid for this field, but not acceptable to pandas. Its a real world case btw.

if the key is not there it should raise a `KeyError` not a `TypeError` I think (I can change the test to verify for that).
@WillAyd I have changed it to use the fixture, but I'm not entirely happy with it. The fixture sets it to float('NaN') which is different from 'None'. Now I am testing against an iterable and catching a `TypeError` which changes the gist of it.

What do you think?
@jreback @WillAyd ready for re-review
@jreback ping
Ping @jreback 
Friendly reminder @jreback
can you merge master and will look again
@jreback rebased. Or do you truly want me to merge master into my branch?
@jreback all green again.
thanks @bolkedebruin very nice
REF: use new pytables helper methods to de-duplicate _convert_index
State: closed

The assertions in here are intended to be temporary.  Once reviewers are convinced this is accurate, then the assertions can be removed and a bunch of these blocks can be condensed down to share code.

The new helper func _get_data_and_dtype_name is also going to be used later to pre-empt the need for the state-altering `get_data` method.

Labels:
name: Refactor
description: Internal refactoring of code
name: IO HDF5
description: read_hdf, HDFStore

Comments:
looks good. if you can do follows as indicated above when you have a chance.
REF: pytables remove need for set_atom
State: closed

Note this is doing the same thing as #30142 but in a different place.  In both cases we are taking attrs that are currently pinned on post-constructor and instead passing them to the constructors.

Labels:
name: Refactor
description: Internal refactoring of code
name: IO HDF5
description: read_hdf, HDFStore

Comments:
needs a rebase
Thanks @jbrockmendel
REF: pytables remove IndexCol.infer, move attr setting to constructors
State: closed

In this PR the calls to `get_attr` are moved to right after the constructor calls; in  the next pass they will be removed entirely at which point IndexCol will be non-stateful

Labels:
name: Refactor
description: Internal refactoring of code
name: IO HDF5
description: read_hdf, HDFStore

Comments:
lgtm. ping on green.
ping
Update v1.0.0.rst
State: closed

- [ ] closes #30011 
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:

Comments:
@maheshbapatu for the same issue all commits need to be on the same PR, I see you have #30083 pls add this commit there.
Thanks for helping out @jreback. Now I have added the changes in the #30083 itself.
BUG: incorrect output of first('1M') in case if first index is the last day of the month (#29623)
State: closed

- [ ] closes #29623 
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Frequency
description: DateOffsets

Comments:
Hello @Franklinluo17! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:

* In the file [`pandas/core/generic.py`](https://github.com/pandas-dev/pandas/blob/4fa289dc3394acf5460f57a9094e2e396b3a8c0f/pandas/core/generic.py):

> [Line 8553:1](https://github.com/pandas-dev/pandas/blob/4fa289dc3394acf5460f57a9094e2e396b3a8c0f/pandas/core/generic.py#L8553): [W293](https://duckduckgo.com/?q=pep8%20W293) blank line contains whitespace

* In the file [`pandas/tests/generic/test_generic.py`](https://github.com/pandas-dev/pandas/blob/4fa289dc3394acf5460f57a9094e2e396b3a8c0f/pandas/tests/generic/test_generic.py):

> [Line 915:9](https://github.com/pandas-dev/pandas/blob/4fa289dc3394acf5460f57a9094e2e396b3a8c0f/pandas/tests/generic/test_generic.py#L915): [E704](https://duckduckgo.com/?q=pep8%20E704) multiple statements on one line (def)
> [Line 925:9](https://github.com/pandas-dev/pandas/blob/4fa289dc3394acf5460f57a9094e2e396b3a8c0f/pandas/tests/generic/test_generic.py#L925): [E704](https://duckduckgo.com/?q=pep8%20E704) multiple statements on one line (def)
> [Line 934:9](https://github.com/pandas-dev/pandas/blob/4fa289dc3394acf5460f57a9094e2e396b3a8c0f/pandas/tests/generic/test_generic.py#L934): [E704](https://duckduckgo.com/?q=pep8%20E704) multiple statements on one line (def)



##### Comment last updated at 2019-12-08 03:45:40 UTC
closing as stale, if you want to continue, pls ping.
CLN/TYPE: window aggregation cleanups and typing
State: closed

- xref https://github.com/pandas-dev/pandas/pull/29878#issuecomment-561919909
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`


Labels:
name: Clean
name: Window
description: rolling, ewma, expanding

Comments:
All green.
Removed is_sparse_array from objToJSON
State: closed

@jbrockmendel @jorisvandenbossche 


Labels:
name: IO JSON
description: read_json, to_json, json_normalize
name: Clean

Comments:
i couldve sworn i did that in the other PR.  did i screw up?
Are you thinking #29989 or another PR? If former I don't see this module touched in there; know we discussed it I assumed left to follow up
#29989, yah, i even commented here https://github.com/pandas-dev/pandas/pull/29989#issuecomment-561874530 that i had removed is_sparse_array
mystery notwithstanding, LGTM
thanks @WillAyd 
do we actually tests that .to_json() with EAs? is this supported in any way currently?
CLN: change str.format() to f-string
State: closed

- [x] ref #29547
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
Thanks @AlpAribal !
REF: simplify pytables set_kind
State: closed

Pass more info to the constructors and set less stuff afterwards.

Labels:
name: IO HDF5
description: read_hdf, HDFStore

Comments:
Hello @jbrockmendel! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:



There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2019-12-07 15:40:20 UTC
gentle ping @WillAyd.  _dtype_to_kind turns out to be useful for de-duplicating and de-state-ifying a ton of code
@jbrockmendel i will look soon
thanks @jbrockmendel
DOC: Cleaned docstrings in core/indexers.py
State: closed

- [ ] closes #xxxx
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Docs

Comments:
@MomIsBestFriend can you rebase
rebased @jbrockmendel 
Thank you @jorisvandenbossche 
(Will merge when green)
Thanks @MomIsBestFriend !
CLN: Split test_base
State: closed

part of #23877

- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`

Some notes:
- `test_ops.py` still feels pretty bloated, but I didn't see a good opportunity for further splitting
- `test_conversion.py` is testing things like `to_list` and `to_numpy`. Maybe there's a better name for the file

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Clean

Comments:
Yes, I will definitely do that
thanks @SaturnFromTitan nice split on this, keep em coming!