BrandonFors/pandas-github-issues huggingface dataset:
fix for the issue-11833
State: closed

updated the to_string() function of  DataFrameFormatter(TableFormatter) class ,in format.py file, of the core directory

Closes https://github.com/pydata/pandas/issues/11833


Labels:
name: Output-Formatting
description: __repr__ of pandas objects, to_string

Comments:
needs tests

@jreback   so how should I push the commit?

I believe it works and solves the issue

just push
nothing special 

@jreback  it shows that the commit fails the check, but it solves the issue mentioned??
can you review the commit ?

you need to have it pass all existing and any new tests

@Vutsuak16 The problem is that you _also_ need to change the test that fails. As we are changing the behaviour here (it was explicitly tested that there were blank spaces, which we now want to remove)
DEPR: pandas.stats.var, pandas.stats.plm, pandas.stats.ols, pandas.stats.fama_macbeth
State: closed

closes #6077 

this cleans ups the remainder of the `pandas.stats` modules

we are directing:
- `OLS` -> `http://statsmodels.sourceforge.net/stable/regression.html`
- `PanelOLS/MovingOLS/PooledOLS` to `http://statsmodels.sourceforge.net/stable/mixed_linear.html`
- `VAR` to `http://statsmodels.sourceforge.net/stable/vector_ar.html#var`
- `fama_macbeth` to `http://statsmodels.sourceforge.net/stable/index.html` (as not implemented in statsmodell, but good place to start)


Labels:
name: Deprecate
description: Functionality to remove in pandas

Comments:
cc @jseabold 
cc @josef-pkt

looks reasonable?

looks fine to me (without checking details)

Hopefully the deprecation warnings induce some users to help, at least with missing API and testing.
I can check panel models (missing conveniences for fixed effects handling) and move FamaMacBeth (given the example and test case in the issue) in late winter or spring.

I don't know if there is anything directly to do about expanding and rolling OLS or whether it's handled by just a call to the general pandas methods. (I haven't looked at that in years.)

thanks @josef-pkt 

yeah meant to deprecate these a while ago. 

+1 !
BUG: masking empty DataFrame
State: closed

closes #11859 

A `ValueError` was raised when trying to mask an empty DataFrame. In addition to solve the bug I've also added a simple new unit test in `test_frame.py`. I didn't add a line in `whatsnew` because this was already done in #10126 


Labels:
name: Bug
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
needs a whatsnew note - this is in a different release 

thank you sir!
BUG: Spurious matches in DataFrame.duplicated when keep=False
State: closed

Fixes GH #11864


Labels:
name: Bug
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
lgtm - ping when updated and green

@evanpw can you rebase / update

Added test loop, rebased, travis is green.

merged via https://github.com/pydata/pandas/commit/b431f85a505a4d84e38f4d6f9ed5080775d4f31c

thanks!
Fix passing args in groupby plot (GH11805)
State: closed

Closes https://github.com/pydata/pandas/issues/11805

So probably all groupby plot calls that use kwargs are broken in 0.17.x. 
@shoyer it was only a very small change actually!

Still need to add tests.


Labels:
name: Bug
name: Visualization
description: plotting

Comments:
Good catch! I guess we had very poor coverage for groupby plots previously.

Very poor coverage indeed .. 
Added tests + whatsnew notice.

lgtm
DOC: add examples of database drivers (GH11686)
State: closed

Closes #11686


Labels:
name: Docs

Comments:
:+1: Only minor thing is that you have "you also need a database specific driver." could be "you _may_ also need a database specific driver." depending on the database your using.

True, but I think that is only the case for sqlite? (that it is not necessary) Although you actually need one, only it is included in the standard library. But will include that.

A very, very minor thing :smile: 

Updated!

lgtm
Add examples for pandas.*.sample
State: closed

The following doc files are generated from the sample() function from
pandas/core/generic.py:

pandas.DataFrame.sample.html
pandas.Panel.sample.html
pandas.Panel4D.sample.html
pandas.Series.sample.html

Examples for each have been added to the sample() function docstring.


Labels:
name: Docs

Comments:
This looks great, but I would drop the examples for Panel and Panel4D. These structures (especially Panel4D) are rarely used, so including them adds noise.

Done!

This looks good to me -- anyone else have any comments before I merge?

Yep, thanks!
Gbq upgrade
State: closed

This adds support for service account authorization to Google Big Query API + some informative messages on std out.

GBQ can be used with Python 3. In RTBHouse we are doing this since may. It works perfectly with this pandas upgrade.


Labels:

Comments:
this was done in 0.17.0 released on October 

http://pandas.pydata.org/pandas-docs/stable/whatsnew.html#google-bigquery-enhancements

so pls rebase on master

you can submit other enhancements if you wish - 

I will rebase :) sorry
but I do not see any support for google service account auth in master now.
This is usable when working from remote host Jupyter/iPython web notebooks and one can not authorize to google by web request

iirc we have an issue for the service account
if u can find one pls xref it (and of course impl welcome as well)

ok, I found this 
https://github.com/pydata/pandas/issues/8489
I'll push my patch for that
Doc: added names parameter in read_excel
State: closed

closes #11468

Please Review.


Labels:
name: Docs
name: IO Excel
description: read_excel, to_excel

Comments:
Looks good, thanks!

You changed the explanation compared to the one included in `read_csv` (explicitly passing `header=None` when there is no header). This is indeed the current behaviour! (because for `read_excel` the default for `header` is 0, while for `read_csv` this is 'infer'). 
But, I think we should consider to make this more consistent. Will open a new issue for that.
index is included in memory usage by default
State: closed

...and sys.getsizeof returns correct value. Closes https://github.com/pydata/pandas/issues/11597

Is this the best implementation for a common method across classes?


Labels:
name: Compat
description: pandas objects compatability with Numpy or Python functions

Comments:
@jreback Python 2.6 error with `delta` argument on `assertAlmostEqual` - should I skip the test on 2.6 or find an alternate way of doing this?

xref https://github.com/pydata/pandas/issues/7718

@jreback `assertLess` isn't supported either on 2.6? 
But we're green

@jreback updated & green

@jreback updated & green

@MaximilianR lgtm.

can you run: `git diff master | flake8 --diff` and fix those issues. we are going to be shortly enforcing this, so might as well not add to it :)

ping when green.

Done (phew...)

lgtm. ping when green.

merged via https://github.com/pydata/pandas/commit/60cacabdd37dcfb00920aed453787e77358219d1

thanks!
look for colormap in rcParams['axes.prop_cycle'] (mpl 1.5+) first
State: closed

Fixes #11614


Labels:
name: Visualization
description: plotting

Comments:
Thanks!

Could you add a test and a release note? The test should ensure that the colors of the bar patches are correct.

Added a new test with both a passed colormap and one set through mpl's rcParams. What do I need to do to add a release note?

For the release note add a line in `doc/source/whatsnew/v0.18.0` under the bug fixes heading.

I think it's good to go, though looks like I forgot to pull in the most recent changes to the release notes before adding my change... oops. Let me know if there's anything else that needs doing.

@rcarneva Can squash the commits? (http://pandas-docs.github.io/pandas-docs-travis/contributing.html#combining-commits) For the rest this looks good I think!

Thanks for the help with this, @TomAugspurger and @jorisvandenbossche 

I'm going to merge this now. I think it has the same problem that all of our compatibility code for mpl 1.5 has: what if the user has a prop_cycle where color isn't in the cycler? May need to look into that if it's something matplotlib tries to support.

Thanks @rcarneva!
BUG: force list type for tuples from chunked sql table reads #11522
State: closed

Closes #11522.


Labels:
name: IO SQL
description: to_sql, read_sql, read_sql_query

Comments:
@grahamjeffries Looks good! thanks

This is actually in code that is deprecated, and will be removed in the future. But as it is only a small change, ok to let this in for the case it will not yet be removed in the next version.

Sure thing. Yeah, I wasn't quite sure what the protocol was for testing cases which fail with an exception. I'll update the PR

@grahamjeffries OK! Just a last question, can you add a note in the whatsnew file (doc/source/whatsnew/v0.18.0.txt) in the bug fixes section, and then also squash the commits? (http://pandas.pydata.org/pandas-docs/stable/contributing.html#combining-commits)

Noted and squashed.

Thanks a lot!
fixed conversion to sparse for non-numeric index
State: closed

this fixes  #11633

this fix breaks some sparse tests,
stuck on `test_combine_first` : `ValueError: total size of new array must be unchanged`


Labels:
name: Bug
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: Sparse
description: Sparse Data Type

Comments:
this looks reasonable. I think you may have to adjust some tests, and maybe some more fixing of the dtype propogation.

I fixed one test to its original state (accounting for conversion of integer values to floats with `to_sparse`). Now `./test_sparse.py` runs OK. 

can you rebase when you have a chance.

rebased

need some tests for this.

I added a test. 
One existing test fails now.

The current fix still does not completely solve the problem of sparse pivot #11633 (it uses sparse as input, but produces a dense output). One will need to look into `pivot_table` itself to see if one can make it spit out a sparse table.

@DSLituiev that's ok, let's get this bug fix in, then can work on that (which is a bit more complicated). _you_ can always start constructing a fix based on THIS branch if you would like.

@DSLituiev can you update?

@DSLituiev can you update, lots of Sparse changes recently.

I'll have a look

On Sat, Apr 9, 2016 at 8:05 AM, Jeff Reback notifications@github.com
wrote:

> @DSLituiev https://github.com/DSLituiev can you update, lots of Sparse
> changes recently.
> 
> —
> You are receiving this because you were mentioned.
> Reply to this email directly or view it on GitHub
> https://github.com/pydata/pandas/pull/11856#issuecomment-207804337

can you rebase/update

can you rebase / update, we've had lots of sparse changes.

As it is important for #667, I've prepared #13201.

superseded by #13021
Change pandas to never let objects share `_data`, create views instead
State: closed

Closes Issue #11814 

Several pandas operations create objects that share `_data` objects. In particular, all of the following result in `df_new`s and `df` sharing the same `_data` (i.e. `df_new._data is df._data` returns `True`). 

```
df = DataFrame({'col1':[0,1,2], 'col2':[2,3,4]})
df_new = DataFrame(df)
df_new1 = df.loc[:,:]
df_new2 = df.iloc[:,:]
```

Needed for PR #11500 


Labels:
name: Internals
description: Related to non-user accessible pandas implementation

Comments:
xref #11970 

this idea will be incorporated there.

If you want to update before then (as this might take a while), feel free to reopen.
CLN: frequency.get_offset always return copy
State: closed

Closes #11192. 

Fix 2 points which should not affect to normal users (thus no release note descriptions).
- get_offset (#11192): It returns cached offset, overwriting it may leads to unexpected results. `pandas` public methods doesn't use them as it is. Thus it should not affect to almost all users, except who explicitly uses `frequencies.get_offset`
- get_offset_name: This looks leagacy impl which raises `ValueError` for `BDay` with `n != 1` which is valid input. I've marked this as deprecated, but it can be removed immediately as non-public.


Labels:
name: Frequency
description: DateOffsets
name: Clean

Comments:
let's leave the deprecation in, and note in the whatsnew. I marked it on the DEPR issue.

@jreback Thanks, added the release note.

thank you sir!
ERR: between_time now checks for argument types
State: closed

Closes #11818


Labels:
name: Datetime
description: Datetime data dtype
name: Error Reporting
description: Incorrect or improved errors from pandas

Comments:
do you think passing non-time like strings are also an error?

e.g.
`2015-01-01 09:00` would be valid (previously and after this one). We could restrict to `Timedelta` parsables I think (not exactly the same thing but would work).

Unfortunately, I don't know how to check that.  How it currently works is the dateutil parser is used to parse the time string but a datetime.datetime is returned regardless if it's a time string or date string.

```
>>> from dateutil.parser import parse
>>> parse("2015-12-13 10:10")
datetime.datetime(2015, 12, 13, 10, 10)
>>> parse("10:10")
datetime.datetime(2015, 12, 13, 10, 10)
>>> 
```

I see what your idea is but I'm just not sure that the `Timedelta` and dateutil parser are completely the same.

```
>>> Timedelta("10:10")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "pandas/tslib.pyx", line 2277, in pandas.tslib.Timedelta.__new__ (pandas/tslib.c:41085)
    value = np.timedelta64(parse_timedelta_string(value, False))
  File "pandas/tslib.pyx", line 2862, in pandas.tslib.parse_timedelta_string (pandas/tslib.c:49601)
    raise ValueError("expected hh:mm:ss format")
ValueError: expected hh:mm:ss format
```

the timedelta parser is a bit strick for this, can prob make a mini one by doing something like:

```
try:
    return time.strptime(s, '%H:%M:%S')
except:
    try:
        return time.strptime(s, '%H:%M')
    except:
        # rise here
```

Seems like we should also do a milliseconds parse for safe measure.

```
>>> parse("10:10:10.123")
datetime.datetime(2015, 12, 13, 10, 10, 10, 123000)
```

don't use dateutil.parse we want a time only

My only comment was that it's currently supported so we need that additionally.

easiest might be to loop thru a list of allowed formats until u match (or exhaust the list and raise)

using strptime is strict (which is good)

I'm confused as to why this fails on 2.6.  I created a 2.6 environment and the test works fine there.

```
Python 2.6.9 |Continuum Analytics, Inc.| (unknown, Aug 21 2014, 18:28:52) 
...
In [1]: from datetime import datetime

In [2]: datetime.strptime("2:00am", "%I:%M%p").time()
Out[2]: datetime.time(2, 0)
```

Ah, it's because that test uses the am/pm from it_IT locale.  Skipped if there is a locale present.

Hopefully we are all good here.

small comments.

pls run: `git diff master | flake8 --diff` and fix the PEP8 issues (this is going to become standard shortly).

this looks really good. I think we _could_ expose this via `pd.to_time`. Though I we do need better time support. Pls create an issue so we can consider this in the future.

ping when green.

@jreback Green.  I feel more comfortable leaving the ability to add another format given that we do not know how much people were relying on dateutil's parser.  It's two lines of code and innocuous.

merged via https://github.com/pydata/pandas/commit/9c71dbf7e5149ca37e454d33f04fb81dc282a54c

thanks.
CLN: Use `is` operator for comparing with `None` (Pep8)
State: closed


Labels:
name: Clean

Comments:
I've reverted the changes to that file.
Btw, is it kosher to add comments like PyCharm's "# noinspection PyComparisonWithNone" to specifically try to avoid newbies like me from thinking there's an error there? 

@proinsias yes having a comment is ok to prevent linters from finding this is fine

ping when updated

I've updated the pull request to avoid modifying the test code, and to add #noinspection comments in the test code.

thanks!
CLN: Use sentinel values instead of mutable default arguments
State: closed

From [Quantified Code](https://www.quantifiedcode.com/):

Mutable list or dictionary used as default argument to a method or function. Python creates a single persistent object and uses it for every subsequent call in which the argument is left empty. This can cause problems if the program was expecting the function to return a new list or dictionary after every call. Use a sentinel value to denote an empty list or dictionary.


Labels:
name: Clean

Comments:
ok, but really need to limit scope to tested code, and need to be really careful changing the tests

Should be all set now. 

this is fine. ping when green.

All checks have passed.

thanks!
DOC: Add example to pandas.DataFrame.applymap
State: closed


Labels:
name: Docs

Comments:
Examples are great - how would you feel about having one that distinguishes `applymap`? `df.apply(abs)` gives the same result there

Further, `abs` is actually not a really good example as a functions for which you need `applymap`, as `abs(df)` will also work due to delegation to `__abs__`

I've updated this to use a lambda function that operates element-wise.

:+1: 

Thanks!
BUG: read_excel fails when empty sheets exist and sheetname=None #11711
State: closed

Fixes issue #11711.

Existing code prematurely returns an empty dataframe when an empty sheet in the source excel file is encountered.
Fix is to store an empty dataframe in the output dict and continue to the next sheet.


Labels:
name: Bug
name: IO Excel
description: read_excel, to_excel
name: Compat
description: pandas objects compatability with Numpy or Python functions

Comments:
does this cover both cases in #11711 (IIRC blank sheet in middle was ok, at end was buggy)

but was there another case?

This should cover cases where blank sheets are anywhere in the excel file, including in the middle, as stated in the #11711 [comment](https://github.com/pydata/pandas/issues/11711#issuecomment-160151977). 

ok lgtm. ping when green.

build done & green

thank you!
ENH: add 'asm8' to NaT
State: closed

I was suprised to find that `NaT.asm8` does not exist though there is a valid nat form for `numpy.datetime64`.

This also adds tests for the asm8 behavior for a random sample of times and `NaT`


Labels:
name: Enhancement
name: Datetime
description: Datetime data dtype
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate

Comments:
why is this needed? 

ahh sorry I see u moved to _Timestamp

lgtm, ping when green.

Thank you!

One question I have about the `NaT` class heirarchy is the purpose for the `_NaT` class. It is only sublclassed by `NatType` and seems to add a few operations.

the `_NaT` is a sub-class of `_Timestamp` class which is a `cdef` class so its purely a c-extension type. While `NaTType` and `Timestamp` are available to the python space (and the c-space).

The entire reason for this is that `datetime` is untimately sub-classes in c-land as its a python extension type.

@wesm original comment.

```
# This is PITA. Because we inherit from datetime, which has very specific
# construction requirements, we need to do object instantiation in python
# (see Timestamp class above). This will serve as a C extension type that
# shadows the python class, where we do any heavy lifting.
```

I see, it appears cython doesn't give us a way to override the `tp_new` directly so we cannot create the overridden `__new___` with the correct signature. Thanks for explaining this.

We are green!

thank you sir!
ENH: GH11128 add weekday_name to DatetimeIndex and .dt
State: closed

closes #11128 

some examples:

```
In [5]: s2 = s.to_frame('value').assign(weekday=s.index.weekday_name)

In [6]: s2
Out[6]:
            value    weekday
2015-09-01      0    tuesday
2015-09-02      1  wednesday
2015-09-03      2   thursday
2015-09-04      3     friday
2015-09-05      4   saturday
2015-09-06      5     sunday
2015-09-07      6     monday
2015-09-08      7    tuesday
2015-09-09      8  wednesday
2015-09-10      9   thursday
```

and

```
In [7]: s3 = pd.Series(pd.date_range('2015-09-01', '2015-09-10'))

In [8]: s3.dt.weekday_name
Out[8]:
0      tuesday
1    wednesday
2     thursday
3       friday
4     saturday
5       sunday
6       monday
7      tuesday
8    wednesday
9     thursday
dtype: object
```

Please review my code :smile: 


Labels:
name: Enhancement
name: Datetime
description: Datetime data dtype

Comments:
The first letters should be capitalized?

pls also add this to `Timestamp`.

Sorry I did not comment on the original issue before you put effort in this, but I wonder a bit if this is worth a new method (we already have many many methods, so adding a new needs a strong case I think). 
Wouldn't be some good docs be sufficient? Eg specifically add in the docstring that if you want to get the names instead of the numbers, how you can do that most easily?

I actually thing this is pretty reasonable. Its an adjust method for weekday that is useful. No harm IMHO at trivially expanding the API.

@jorisvandenbossche that's ok, this is my first time doing PR on pandas and I learn a lot about the code and contributing process, etc. What I think about it is a 'nice to have' feature and it has Difficulty Novice label that I think this is a perfect issue for me to learn more about the underlying structure of pandas. IMHO I have not found a strong usage on this feature. But, @jreback may have another perspective about it.

What I think about the usage of it is on the REPL like IPython or just python to check thing that what we are intended to do on a date are doing exactly what we want. For example, in #11123 if we can see the day name seamlessly without interpreting the number into day name in our mind. Perhaps it could help our mind a little bit :joy: 

@kawochen Hmm yeah, I'm just thinking that the capitalized form may be useful for serialization into csv, excel, etc. Should I change to capitalized form @jreback? I need some advice to choose between lower case or capitalized :pray:.

prob should use what `calendar` gives back for the names

and, this actually could/should be implemented similarly to the `_field_accessors` (e.g. directly in cython). 

@bahrunnur can you rebase / update

I still like this. @bahrunnur can you rebase / update

Oh hi @jreback !
I'm sorry for the slow response. I've been on vacation since the first day of this year. Now, I have rebased my code.

can you update

can you rebase/update

closed in favor of #12803
ENH: Using built-in round on a series #11763
State: closed

closes #11763 

This contains changes to support `round()` (works in Python 3 only) for DataFrame, Series and Panel. Also includes `Panel.round()`.  Note on code changes - moved code from test_format.py to test_frame.py because it made more logical sense to put the testing code for `round()` into test_frame.py


Labels:
name: Enhancement
name: Numeric Operations
description: Arithmetic, Comparison, and Logical operations

Comments:
small docs changes. pls squash. ping when green.

@jreback It's squashed now.  Not sure what to do next to force the Travis CI thing to happen since it says there are conflicts

you need to rebase on master as well 

@jreback I think this is good to go.

couple of trivial changes. ping when pushed (as its already green)

@jreback pushed those changes (and squashed them as well)

@jreback last build failed due to a URL issue.  Not sure how to restart the run

@Dr-Irv no worries. thanks for this!

yeh travis fails once in a while on the build step.