BrandonFors/pandas-github-issues huggingface dataset:
DOC: added exp weighting clarifications from #8861
State: closed

I integrated the comments in #8861 from @seth-p into the exponential weighted moment functions docs.


Labels:
name: Docs

Comments:
Thanks! Looks good from a technical point. 
@seth-p Can you do a review?

It looks good, though would be even better to include some of the substance in @immerrr's https://github.com/pydata/pandas/issues/8861#issuecomment-63778421, specifically that for the `adjust=False` case to make sense, you need to assume that "the first element is not an ordinary value, but rather a prior knowledge of the whole infinite-ish series".

Good point, I'll put some of that content in.

@matthewgilbert pls review the doc string `pd.ewma` as well for consistency with this (you can also put in a link to the actual docs to not make it overly long)

@jreback Bit confused with your suggestion to link to the docs string, was there a specific place you had in mind for this?

put a link in the doc-string to the more detailed explanations 

Incorporated link to the docstring and added some explanation of describing assumptions on first element when `adjust=False`, squashed this into one commit

@matthewgilbert, I'm afraid I don't think you incorporated @immerrr's point in https://github.com/pydata/pandas/issues/8861#issuecomment-63778421 correctly. It is when `adjust=False` that one is implicitly assuming that `x[0]` is a moving average encapsulating the data from all "previous" observation; whereas for `adjust=True` one is implicitly assuming that `x[0]` is an observation just like every other `x[i], i>0`. (In all of these, `0` should really be replaced by the index of the first non-`NaN` value in `x[]`, but I'm glossing over that subtlety.)

@seth-p Thanks, yes this was a typo, sorry about that. Just to be clear, changing

"When `adjust=True` we have..." to "When `adjust=False` we have..." should fix this, or is there something else I'm missing?

@matthewgilbert, I think that's fine, though also I think `:math:`\alpha x_t + (1 - \alpha) y_{t-1}`` should be `:math:`y_t = \alpha x_t + (1 - \alpha) y_{t-1}``.

All comments have been incorporated. I squashed this all into one commit so unsure if as a result this does not send notifications and thought I would just comment here to be sure.

@matthewgilbert yes a push does not ping anyone.

This looks good.

@seth-p any final comments?

Looks good.

@matthewgilbert Thanks a lot! 
In some time, this should be built in the dev docs: http://pandas-docs.github.io/pandas-docs-travis/, so you can always check there if everything looks fine
added a note on index creation for pandas dataframe stored in hdf5 file
State: closed

Based on my discussion with Jeff

http://stackoverflow.com/questions/25714549/indexing-and-data-columns-in-pandas-pytables/25715005?noredirect=1#comment53932720_25715005


Labels:
name: Docs
name: IO HDF5
description: read_hdf, HDFStore

Comments:
@jreback This isn't useful enough to go in the main hdf5 docs?

yeh,, why don't you move this into: http://pandas.pydata.org/pandas-docs/stable/io.html#indexing

@ksxh can you move as described above

merged via https://github.com/pydata/pandas/commit/74f0111e0aa049077fc808e07b0300ca143a517f

thanks!
ENH: Add Index.fillna
State: closed

Closes #10089. 
- `value` can only accept scalar.
- `MultiIndex.fillna` raises `NotImplementedError` because `isnull` is not defined for MI.
- Moved `hasnans` and related properties to base `Index`.


Labels:
name: Enhancement
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate

Comments:
Thanks, did the suggested changes. 

A test fails in `Series.fillna` which uses `DatetimeIndex.putmask`. Looks #11329 fixes the same part, I'm leaving it now.

Rebased, and now green.

Thanks, simplified the logic based on the comments.

Thanks, updated based on the comments.
- `_convert_for_op` converts value for scalar op
- `assert_can_do_op` check whether the value is acceptable for scalar op 
  - timezone check is split to different method. Because it is used in list-likes such as `equals`.
- Some changes related to output detailed error messages / keep current messages 

@sinhrks looks really good!

just a minor comment / question

Reabsed, and now green.

lgtm

merge when ready

thanks!

might want to add s short example in the Index docs (iirc where we go over Index set ops)

Thanks. Added brief doc to `indexing.rst`. Merging within few days if there is no further comments.

CC: @jorisvandenbossche 

@sinhrks Looks good! One more remark, doc-wise: I see there is a `downcast` keyword, but there is no example given of its usage in docstring or rst docs. That would be nice to include to clarify its usage I think.

@sinhrks thanks!

always gr8 PR's from you!

I don't see an entry in `API.rst` can you add?

@jreback Ah, let me add it with #11537.
BUG: Bug in DataFrame.replace with a datetime64[ns, tz] and a non-compat to_replace #11326
State: closed

closes #11326
xref #11153 


Labels:
name: Bug
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: Dtype Conversions
description: Unexpected or buggy dtype conversions

Comments:
@sinhrks I ended up rewriting replace/fillna quite a bit to make much more generic. as lots of duplicated code in #11153 

This simplifies quite a bit.

lmk if you have comments.
BUG: Fix to_dict() problem when using datetime DataFrame #11247
State: closed

closes #11247

This is my first time contributing through PR so forgive me if I make any silly mistake.
The issue can be referred to in GH11247
I followed @jreback 's suggestion to box things up in `to_dict()` but I am not sure if this is what he means because I do not know exactly when `_maybe_datetime()` should be used. Initially I tried to tweak `internals.py` but it fails.

Let me know how I can improve from here. Hope I can really learn from my first PR.


Labels:
name: Bug
name: Output-Formatting
description: __repr__ of pandas objects, to_string
name: Dtype Conversions
description: Unexpected or buggy dtype conversions

Comments:
see guidelines here: http://pandas.pydata.org/pandas-docs/stable/contributing.html

dont' turn on PEP changes for your editor.

I think I forgot to fetch to update my local master before my first commit to this PR. Somehow I messed up is it true?

add tests! (ideally for each one of the orients if we dont' have already).

which test case shall I add in? I am new to this :)

> On 15 Oct 2015, at 2:50 AM, Jeff Reback notifications@github.com wrote:
> 
> add tests! (ideally for each one of the orients if we dont' have already).
> 
> —
> Reply to this email directly or view it on GitHub https://github.com/pydata/pandas/pull/11327#issuecomment-148154675.

@jreback I have added tests:)

pls add a whatsnew note for 0.17.1 in bug fixes

Just asking what I should do if I merge updates locally with my branch and need to merge the 2 commits into one..

—
Sent from Mailbox

On Fri, Oct 16, 2015 at 6:25 AM, Jeff Reback notifications@github.com
wrote:

> ## pls add a whatsnew note for 0.17.1 in bug fixes
> 
> Reply to this email directly or view it on GitHub:
> https://github.com/pydata/pandas/pull/11327#issuecomment-148540340

merged via https://github.com/pydata/pandas/commit/89b4e5b7337606f3e1e40a3d13be863a8be6ff44

thanks!
ENH: itertuples() returns namedtuples (closes #11269)
State: closed

closes #11269  

This will make itertuples return namedtuples. I'm not sure about tests, here. Since `namedtuple` is a drop-in replacement for ordinary tuples (once they are created) I naively expect things to work.


Labels:
name: Enhancement
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Compat
description: pandas objects compatability with Numpy or Python functions

Comments:
tests!

take a frame of multiple dtypes, call itertuples on list, then assert that the first tuple generated has the correct fields, named correctly (and handles rename properly).

need a test for python26 when renaming fails (but the tuple works), and where it fails completely.

these should go with other tests of itertuples.

need some tests

Added tests for `namedtuple`s and the `name` argument. Squashed.

Sorry for the noise, but I'm hitting `SyntaxError: more than 255 arguments` with a large number of columns. I've modified things to just return regular tuples via `zip()` on any exception, as things were before.

The commit with updated tests was for some reason not pushed, trying again.

Ok, thanks for your comments. I was quite happy to have a regular tuple fallback set up.

ok minor comment.

need a whatsnew note. put in the API section

ok, ping when green.

Python 2 turned out to happily accept an almost infinite number of arguments, so i just set a hard limit of 255 fields.

@jreback all done if you approve the last modification.

It's just that calling the method with a large number of fields on Python 2 was really slow.

@mjoud i c. ok then.

@jreback ok fixed

Thanks, better. I'm not exactly showing off here...

@mjoud thanks!

nice PR.
DEPR: deprecate `engine` keyword from to_csv #11274
State: closed

closes #11274 


Labels:
name: IO CSV
description: read_csv, to_csv
name: Deprecate
description: Functionality to remove in pandas

Comments:
merged via https://github.com/pydata/pandas/commit/eda1924bd313476436742c653e08307ce232f36f

thanks!
BUG: multi-index excel header fails if all numeric
State: closed

If the multi-line headers come from Excel, and the header was not a string, the line all(['Unnamed' in c[n] for c in columns]): will fail because c[n] is an int and not iterable.  So either force the headers to be strings (the proposed change) or come up with some other test when looking for 'Unnamed'


Labels:
name: IO CSV
description: read_csv, to_csv
name: IO Excel
description: read_excel, to_excel

Comments:
you need a test where this is actually failing 

For some reason, I can't submit the simple XLSX file that illustrates the problem, or a CSV representation as a file.  Here is the CSV version:
Id,A,A,B,C
is,1,2,1,3
1,13,23,33,43
2,24,44,64,84
3,35,65,95,125
4,46,86,126,166
5,57,107,157,207
6,68,128,188,248

Read this into excel, and save as a file  "testmulti.xlsx"

Then this fails:
import pandas as pd
df=pd.read_excel('testmulti.xlsx',header=[0,1], index_col=0, parse_cols=4)

so add that as an example in `test_excel.py`. It should fail w/o your fix and pass afterwards

I  had to create a new pull request with all the files that illustrate the bug and that the bug is fixed. https://github.com/pydata/pandas/pull/11328

replaced by #11328
BUG: GH10355 groupby std() no longer sqrts grouping cols
State: closed

closes #10355 

Previously grouping columns were square rooted when as_index=False
New method closely follows the format of var() method.


Labels:
name: Bug
name: Groupby

Comments:
see comments on #11085 w.r.t. performance of this

My solution is slightly different from #11085 (I keep the distinction from ddof == 1 and ddof != 1)
I believe my solution is essentially the same as the sketch one proposed on #10355 where you calculate the solution and then take the square root.

Isn't this the same performance-wise as the current method which just square roots the answer from var() (but also includes unnecessary columns)
I'm not an expert on performance so could you clarify this for me?

the problem is that when you go to a python soln it is evaluated for every group rather than using the cython function. The soln needs to simply take the `.var` (with the correct ddof), then `np.sqrt` on the result (excluding the selection columns).

can you update according to comments

@henrystokeley can you update according to comments

replaced by #11507 

https://github.com/pandas-dev/pandas/pull/25315
PERF: Checking monotonic-ness before sorting on an index #11080
State: closed

closes #11080 

Worked for this during PyCon JP 2015 pandas sprint /w @sinhrks 

I found `test_frame.py:TestDataFrame.test_sort_index` cover this change.


Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Performance
description: Memory or execution speed performance

Comments:
pls add some benchmarks that I indicated in the issue (or point to ones that have this covered)

Added a `frame_sort_index` test in `frame_methods.py`
-   30.96ms     4.64ms      0.15  frame_methods.frame_sort_index.time_frame_sort_index

@scari I think you did something wrong with rebasing (as there are a lot of other commits here). Can yo do

```
git fetch upstream
git rebase upstream/master
```

and then push again.

@jorisvandenbossche  Thanks for your kind guidance! I just push again. 

thanks @scari
TST: Add hex color strings test
State: closed

It's already resolved. I added hex string color test.

closes #10299


Labels:
name: Visualization
description: plotting

Comments:
Thanks! This looks to be fixed by #9894. 

@TomAugspurger Please merge if it looks OK.

```
df = pd.DataFrame(np.random.randn(5, 5))
custom_colors = ['#FF0000', '#0000FF', '#FFFF00', '#000000', '#FFFFFF']
df.plot(color=custom_colors)
```

![2015-10-12 17 00 39](https://cloud.githubusercontent.com/assets/1696302/10423585/d4ed2e32-7102-11e5-9b0d-29901d8f4805.png)

@hironow pls add a whats new note (bug fxies)

Sorry, unexpectedly closed.
I tried to 'git rebase' (and replace this PR's repo) to resolve conflicts, but I deleted this by mistake. 
So, what should I do about this PR?

Does Github show a button for you to restore the branch?

@TomAugspurger Thanks for your comment. I restore this branch.
I've created a branch of duplicate name in my own forked repo.

Thanks for updates. Can you squash to the single commit? You can do it by:
1. Open your local pandas repository
2. `git checkout master` to switch to master
3. `git remote add upstream https://github.com/pydata/pandas.git` to register original url
4. `git pull upsream master` to update your master
5. `git checkout pyconjp-2015-sprint`
6. `git rebase -i master` to apply the master's update to PR branch

Pls feel free to ask me if you have any question.

@sinhrks Thanks for your comment and list of git commands.
I probably was able to squash this branch.

@hironow Thanks!
Fix mistake in Pytables querying with numpy scalar value.  Fixes #11283
State: closed

Fixes #11283

First PR to this project.  The fix is minuscule, but let me know if I did anything awry.


Labels:
name: Bug
name: IO HDF5
description: read_hdf, HDFStore

Comments:
Could you please add a note to the bug fixes section of the release notes?
https://github.com/pydata/pandas/blob/master/doc/source/whatsnew/v0.17.1.txt

Also, be sure to include `Fixes #XXXX` in the body of your comment in pull requests -- that links the issues and will automatically close it when the PR is merged (I just edited your message to add that in).

lgtm. pls squash. ping on green.

@jreback Are you saying to do `rhs = np.array(rhs)` instead of `rhs = [rhs]`?  As far as I can tell that line is only to make the iteration work right in the line right after the call to `conform` ([here](https://github.com/pydata/pandas/blob/c2aa6a2333483d0e95661c34de360dbcdc8d1bba/pandas/computation/pytables.py#L249)).  I don't think it needs to be a numpy array and I'm a bit reluctant to make that change without knowing why.  It doesn't appear to be causing any problems.

The fix is working fine now.

thanks
PERF: Release GIL on some datetime ops
State: closed

This is a WIP, but far enough along I thought I'd share and see if the approach was reasonable.  

This releases the GIL on most vectorized field accessors (e.g. `dt.year`) and conversion to and from `Period`.  May be places it could be done - obviously would be nice for parsing, but I'm not sure that's possible.


Labels:
name: Datetime
description: Datetime data dtype
name: Performance
description: Memory or execution speed performance

Comments:
ohh nice!

can u share some timings?

Here are some timings - getting a pretty nice speedup.  In single-threaded case things are looking about flat.

``` python
In [1]: from pandas.util.testing import test_parallel
In [2]: dti = pd.date_range('1900-1-1', periods=100000)

In [3]: def f():
   ...:     for i in range(4):
   ...:         dti.year
In [4]: @test_parallel(4)
   ...: def g():
   ...:     dti.year

In [8]: %timeit f()
10 loops, best of 3: 25.8 ms per loop

In [9]: %timeit g()
100 loops, best of 3: 7.71 ms per loop
```

@chris-b1 loooks good. can you add a whatsnew note (perf) and squash.

@jreback - updated

thanks!

@chris-b1 can you add these (clean then make again to see them)

```
warning: pandas/src/period.pyx:144:24: Use boundscheck(False) for faster access
warning: pandas/src/period.pyx:145:23: Use boundscheck(False) for faster access
warning: pandas/src/period.pyx:147:55: Use boundscheck(False) for faster access
warning: pandas/src/period.pyx:148:19: Use boundscheck(False) for faster access
warning: pandas/src/period.pyx:169:24: Use boundscheck(False) for faster access
warning: pandas/src/period.pyx:170:19: Use boundscheck(False) for faster access
warning: pandas/src/period.pyx:172:15: Use boundscheck(False) for faster access
warning: pandas/src/period.pyx:172:53: Use boundscheck(False) for faster access
building 'pandas._period' extension
```
DOC: some hacks to get rid of warnings
State: closed

This PR includes some hacks to numpydoc to get rid of a bunch of warnings (split of from https://github.com/pydata/pandas/pull/11069 as I want these merged for 0.17.0). There is some more explanation in the commit messages.

Closes #6100

The disadvantage of this is that this causes a deviation from numpydoc upstream, and makes it more difficult to update to that one.


Labels:
name: Docs

Comments:
squash / merge when ready

I would like to not squash the HACKs, as it are separate changes with separate explanations in the commit messages, and so it is more easy to see what we exactly changes and more easy to revert them separately if things change in numpydoc.

sure no prob
COMPAT/PERF: lib.ismember_int64 on older numpies/cython not comparing correctly #11232
State: closed

closes #11232 

PERF: use np.in1d on larger isin sizes


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Performance
description: Memory or execution speed performance
name: Compat
description: pandas objects compatability with Numpy or Python functions

Comments:
...and `series.isin(list_of_strings)` throws an error since `len(series) > 1000000`...
BUG: datetime64 series reduces to nan when empty instead of nat
State: closed

I ran into some strange behavior with a series of dtype datetime64[ns] where I called max and got back a `nan`. I think the correct behavior here is to return `nat`. I looked through test_nanops but I am not sure where the correct place to put the test for this is.

The new behavior is:

``` python
In [1]: pd.Series(dtype='datetime64[ns]').max()
Out[1]: NaT
```

where the old behavior was:

``` python
In [1]: pd.Series(dtype='datetime64[ns]').max()
Out[1]: nan
```


Labels:
name: Bug
name: Datetime
description: Datetime data dtype
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: Numeric Operations
description: Arithmetic, Comparison, and Logical operations
name: Compat
description: pandas objects compatability with Numpy or Python functions

Comments:
can I add a test in test_nanops and/or test_series

also want to test for timedelta64/datetime64[ns, tz] too

Sure, I will update this tonight when I get home

pls also add a note in the 0.17.1 whatsnew bug section

ping @jreback 

question for you. otherwise lgtm, pls squash to a single commit (I know different from other projects, just convention here)

Also I removed the workaround branch so hopefully the tests pass, otherwise we can put that branch back.

ok looks good. I'll be merging things on friday after releasing 0.17.0

Thank you very much

@llllllllll no thank you!

@jreback just rebased against master, good to merge?

we have lots of prs that are worked on at the same time
so when someone adds a note in whatsnew if they are all at the end
merging one requires everyone to rebase
but if the notes are inserted in a big list they are resolved by git
so we can merge many things w/o conflicts 

Ah, that's a cool trick

minor comment, ping when green

tests are passing

thank you sir!
DOC: clarify requirements for read_stata's args
State: closed

read_stata(buf) needs a buf which implements both read() and seek(). read() is not enough. For example, the object returned from zipfile.ZipFile.open('filename') only provides read() and not seek() and cannot be used.


Labels:
name: Compat
description: pandas objects compatability with Numpy or Python functions
name: IO Stata
description: read_stata, to_stata

Comments:
that's great. can you also update the code to check for the `.read` and `.seek` attributes and raise if they don't exist (so you get a nice error message if you try this). Pls add a test as well (you can use the `ZipFile` example)

Actually, I might be wrong. It looks like StataReader turns everything
into a BytesIO object (which is seekable). Not sure why I got an error
in that case.

On 10/05, Jeff Reback wrote:

> that's great. can you also update the code to check for the `.read` and `.seek` attributes and raise if they don't exist (so you get a nice error message if you try this). Pls add a test as well (you can use the `ZipFile` example)
> 
> ---
> 
> Reply to this email directly or view it on GitHub:
> https://github.com/pydata/pandas/pull/11243#issuecomment-145687781

thanks, pls reopen if you find the cause & some problem in read_state
test regarding fix about non propagating along axis on pct_change
State: closed


Labels:

Comments:
pls move to #11150 

you can cherry-pick to your other PR

```
git checkout the_other_pr_branch
git cherry-pick the_hash_of_the_commit_for_the_tests
```

```
git log this_branch
```

will show you the commits (and hashes) for this branch
BUG: HDFStore.append with encoded string itemsize
State: closed

Closes https://github.com/pydata/pandas/issues/11234

Failure came when the maximum length of the unencoded string
was smaller than the maximum encoded length.

Need to run a perf check still. We end up having to call `_convert_string_array` twice, once before we know the min_itemsize, and a second time just before appending once we do know the min itemsize.


Labels:
name: Bug
name: Unicode
description: Unicode strings
name: IO HDF5
description: read_hdf, HDFStore

Comments:
So initially I had to call `_convert_string_array` twice, the second time just to change the length. But it _seems_ like we can just `astype` the converted string to the new length, which is quite a bit faster.

```
In [1]: from pandas.io.pytables import _convert_string_array

In [2]: a = np.array(10000 * ['ë́'])

In [3]: c =  _convert_string_array(a, 'utf-8')

In [4]: %timeit _convert_string_array(a, 'utf-8')
100 loops, best of 3: 5.37 ms per loop

In [5]: %timeit c.astype('|S6')
10000 loops, best of 3: 91.4 µs per loop
```

All the tests passed, but I've never worked with numpy's fixed length strings. Hopefully I'm not breaking any rules here.

merged via https://github.com/pydata/pandas/commit/26db1729f2947d95d9a2e51878ba0f297fe24165

thanks @TomAugspurger
BUG: to_excel duplicate columns
State: closed

closes #11007 
closes #10970 (data in wrong order)
closes #10982 (columns lost).    

Using the approach suggested [here](https://github.com/pydata/pandas/issues/11007#issuecomment-141019207)

All three occurred when using `to_excel` with duplicate columns in the `DataFrame`


Labels:
name: Bug
name: IO Excel
description: read_excel, to_excel

Comments:
@jreback - updated for youf comments.  `to_csv` uses `loc` as well so I did not change that to `reindex` but did add a couple tests (raises if no correct columns specified, otherwise like `reindex`).

pls squash otherwise lgtm.

Nice!

can you rebase/squash

@jreback this is squashed now

thanks!