BrandonFors/pandas-github-issues huggingface dataset:
DOC: make io.rst utf8 only
State: closed

https://github.com/pydata/pandas/issues/5142

@JanSchulz , can you test whether this solves the problem for you?


Labels:

Comments:
Maybe this can solve the windows building issue (I will also test), but aside: do we want this in the docs? Because the example by itself does work, it's only the building that does not work (as far as I understand).

I think we want users/contributors to be able to build the docs, yeah. Even if they're on windows/diff locale
Do you feel the workaround clutter detracts much from the example? 

It took a lot of effort to get pandas to play nice wth unicode and one lesson learned is not
to mix encodings. The docs should be utf8-clean IMO.

I tried it, and it does not solve the issue. And in retrospect, that is maybe also logical: the problem in windows is in the building of the rst with unicode to html, and it is the output generated by the code example which causes this. With your changes, the output of the code example still contains special characters (which is also the point of the code example), and so causes the build on windows to stop.

I think @JanSchulz had another approach as a kind of hack: something along the lines of https://github.com/pydata/pandas/issues/5142#issuecomment-26281482. I also vaguely remember that the issue was fixed when using ipython's version of the ipython directive, but I should check that.

Then I misunderstood the issue. There is a definite difference:

```
s1='word,length\nTr\xc3\xa4umen,7\nGr\xc3\xbc\xc3\x9fe,5'
s2=s1.decode('utf8').encode('latin-1')

s1.decode('utf8')
Out[33]: u'word,length\nTr\xe4umen,7\nGr\xfc\xdfe,5'

s2.decode('utf8')
---------------------------------------------------------------------------
UnicodeDecodeError                        Traceback (most recent call last)
<ipython-input-34-7c1601a98c33> in <module>()
----> 1 s2.decode('utf8')

/usr/lib64/python2.7/encodings/utf_8.pyc in decode(input, errors)
     14 
     15 def decode(input, errors='strict'):
---> 16     return codecs.utf_8_decode(input, errors, True)
     17 
     18 class IncrementalEncoder(codecs.IncrementalEncoder):

UnicodeDecodeError: 'utf8' codec can't decode byte 0xe4 in position 14: invalid continuation byte

```

and in the case of python's "encoding utf8" premable that makes all the difference.
I expected sphinx to accept utf8 input, if it doesn't that seems like a bug to me.

Thanks for testing.

btw, there was some `decode` action in our hacked version of ipython_directive, #5925 may actually solve
the problem by sheer coincidence.

See also here https://github.com/pydata/pandas/issues/5142#issuecomment-26761295. There was indeed a `.decode('utf8')` in our version of the ipython directive for some other reason, but that broke the building on windows.

I will try out the other PR with your rebase.

I'm seriously skimming past all the important bits today :), sorry.
BUG: dt+BDay(n) wrong if n>5, n%5==0 and dt not on offset (GH5890)
State: closed

closes #5890


Labels:

Comments:
looks good...pls do a rebase on master and can merge it....

Hi,
When I do "git rebase upstream/master" I get "Current branch bday-addition is up to date."
Not sure if I need to do something more/else?

sorry

```
git fetch upstrem
git rebase upstream/master
```

see here:

https://github.com/pydata/pandas/wiki/Using-Git

Yeah this is what I had done before and git is saying the branch is up to date, hence I'm thinking I'm done on my side. Is that correct or do I need to do something more? Sorry for the confusion

 it can't be automatically merged prob a conflict on release.rst

did you push it after?

`git push origin mybranchname -f`?

you can also try this

`git branch mybranchname -set-upstream origin/master`

it might not be looking at right tracking branch (iow you are rebaseing to your own branch)

Ah I see thanks. I think it worked this time (I've had to solve the conflicts in release.rst)

thanks!
DOC: remove usage of code-block directive in docstrings + fix some warnings
State: closed

Some docs things:
- remove remaining use of `code-block`, closes #3430 (@y-p, I think that were the last ones, so now it can be really closed)
- fix some headers to avoid useless warnings during doc build (if the line under the header is not as long as the header itself, sphinx complains about it although it renders fine)


Labels:

Comments:
"markup code snippets as such" somehow became "remove usage of code-block in docstrings"
somewhere along the way, but as long as it's consistent across the package it's fine by me.

merge away.

@y-p yeah .. it's indeed maybe not exactly the same as the issue (but it is also more difficult to search the code snippets that are not highlighted as code), but I don't mean remove code blocks, but the literal use of `code-block` and replace with eg `::` as the `code-block` is redundant.
Fix bug where use of .ix[tuple(...)]=x fails to correctly check out of b...
State: closed

...ounds index for columns. This is a regression as it used to work in pandas 0.12.0. The code incorrectly checks the column index against the len() of the dataframe to see if it is in bounds not against the number of columns. Have added 2 tests with non-square dataframes to test that the fix works.


Labels:

Comments:
gr8 pls add a release notes entry refing this PR as I don't think this has an issue associated?

perfect...just need to rebase (conflict in release notes)...

`git rebase -i upstream/master`

fixup and force push...then can merge

thanks!
TST: Testing bug in reading json/msgpack from a non-filepath on windows under py3 (GH5874)
State: closed

closes #5874


Labels:

Comments:
@y-p I can't test this directly on py3 on windows...but should work

can you give a run?

I have a PR almost ready for changing the signature on this and read_msgpack to accept
only url/file_path or file-like/StringIO, what do you think?
Will have to be in 0.14 since it's a breaking change though.

I'm not keen on being the windows tester around here. It looks ok, merge and see if it's green?

ok....will merge if its green see if it fixes....then go ahead an break the API if needed..sure
BUG: Don't return the same thing twice.
State: closed

Surely this is a typo.

I noticed this because I was going add more robust parsing to datetools.parse [1], and I was surprised to find this function there. Two suggestions / questions. 1) Why not just roll this into the parse function like statsmodels instead of having it fall down on things like "1989Q1"? Extra function seems unneeded to me.

2) Why return the "resolution"? The parse function doesn't do this, and I'd expect them to have the same return signature. Optionally, the parse function should also try to infer a frequency.

[1] https://github.com/statsmodels/statsmodels/blob/master/statsmodels/tsa/base/datetools.py#L117


Labels:

Comments:
FYI, this is used mostly for interpreting times AFAIK

I think @wesm wanted to allow replaceable date parsers at one point, so was written generally..

datetools.parse is not hit really all that much except in case of an exceptional string
DOC: Add %in% operator into compare w r (GH3850)
State: closed

Doesn't close #3850 but at least the `%in%` operator is now in the comparison with R docs. I've lumped it with the `match` function since thats the page you see the `%in%` operator in the R docs http://finzi.psych.upenn.edu/R/library/base/html/match.html.


Labels:

Comments:
@chappers thanks again!
BLD: fixup setup.py, initialize pipe variable to None
State: closed

recent changes (fe9c0ae0) broke setup.py for some users:

https://github.com/pydata/pandas/issues/5867#issuecomment-31759183


Labels:

Comments:
ironically their is not an easy way to do this if you are a user. iow, you use git to clone the dir, then uninstall git????
or copy a cloned dir to another machine. weird.

Actually, I completely botched this... there used to be a defensive try/except which swallowed this.
users that get a straight tarball are affected by this, and I'm afraid that means 0.13.0 is borked
as well for users who don't have git installed.

Windows users usually install binaries and others that use tarball will almost always have git
installed so not a complete clusterf**k.

never ever change working code. In fact, never write code. it always ends badly.

and naturally, the windows CI box has git installed... so this slipped through. _sigh_

didnt' you change this after 0.13? 7c6e5f662684dbfefc5f2d4900df74c955ae95c7

oh yeah, fooled by commit log ordered by author date rather then commit date.
I remember now, I waited till 0.13.0 to merge. Release paranoia wins out again.

thanks for the report @ruidc
rework Series.sort is_view check on underlying ndarray before inplace sort (GH5856)
State: closed

closes #5856
related #5853

DataFrame._ixs will properly record a cache change (similar to _get_item_cache)


Labels:

Comments:
This was a cache issue (and not related to is_copy)

`df.iloc[:,0]` takes a different path as `df[0]` and caches differently in how it gets the result

(as the former uses `iget` its data and the latter `get`)

@y-p @jseabold ok by you?

I don't understand what the "cache" has to do with the fact the the series values are a view
on a larger ndarray. What exactly is being cached?

in so far as @jseabold  can now do `if s._is_cached`, yeah looks fine.

this series was created as a slice from a frame

when selecting a column of a frame the frame caches a reference to the created series

the series holds a weak reference back to the parent 

if the series is modified it is reflected in the frame (and in the cached series) which may not be the same object but represent the same data

the reason for this is that when u get a series from a frame it MAY be a view (and usually but not always is)

this system if refs and a back weak ref allow for view propagation even when numpy doesn't provide a view

Thanks, that's a good overview.

What you describe sounds like reference bookkeeping to me, I still don't understand why it's 
termed a cache, but that's not a big deal.

What does "view propagation" mean?

yep like ref counting

view propagation means a change in the series will reflect in the DataFrame thru the numpy view mechanism 

but their are times when the series is changed that requires it to be done in a non-trivial way that numpy cannot propogate thru a view (eg a dtype change in the series)

this cache ref then allows the series to be updated in the parent data frame

effectively it allows one to break a view yet still have it work
COMPAT/BUG: Ignore numexpr < 2.0
State: closed

in both computation.expressions and computationr.engine. Issues with API
compatibility.

Fixes #5857, #5855, #5845.


Labels:

Comments:
@jtratner can you fixup....?

@jtratner ?

doing it now - thanks for the second ping.

I'll merge this once I get it to pass - still a few kinks to work out

u prob need 2 change a build maybe 2.6 

wow - never seen one minor change break so many tests!

something weird going on that requires going deeper, going to circle back tomorrow

@jtratner ?

I'm working on it. There's a few weird test failures that I'm not grokking
right now. (sorry I have way less free time than I used to)

closing.. @cpcloud putting up a new PR

okay thanks @cpcloud 

np!
BUG: tail raises on empty DataFrame
State: closed

closes #5846

calling tail on an empty frame threw an exception.   In addition, df.tail(0) returns self without indexing.  


Labels:

Comments:
had and tail should have very similar checking code for the edge cases

added a check for empty frame for head as well.

@MichaelWS fyi....I edited the title / top of the PR.

putting the `closes issue number` sets up a link the original issue, then automatically closes it when its merged.

Thanks.  

don't change the test

can u show me what is returned kn both current and your case for head(0)?

Currently it returns:
In [3]: pd.DataFrame([1,2,3]).head(0)
Out[3]: 
Empty DataFrame
Columns: [0]
Index: []

[0 rows x 1 columns]

I could easily keep that functionality. I am not sure why someone would call df.head(0) for a non-empty frame.

My case would be:

In [3]: pd.DataFrame([1,2,3]).head(0)
Out[3]: 
   0
0  1
1  2
2  3

[3 rows x 1 columns]

yours is my right bug neither is the original

I think it should return self

iloc[0] on an empty frame is wrong too I think - can I make an issue for that one?

yes, please make an issue on that.  
I agree with you. I think self makes more sense.  

can you rebase this?

sure. this is done.

The only real functionality change is df.head(0) returns self

hmm...ok why don't you put an additional note in the API section (leave your existing one their). I don't think its a big deal but should document it (reference this issue or original again)

I put a note in  the docs.

thanks for the PR!
BLD: fix cythonized msgpack extension in setup.py GH5831
State: closed

closes #5831

@yarikoptic, can you confirm this fixes the problem?


Labels:

Comments:
looks completely logical and seems to help! ;)  I will do proper rebuild across to see if no side-effects (although once again -- logical, so I am not expecting any ;) )

no word from @yarikoptic, tested well for me. 

I'm going to pull the trigger on this one. What could possibly happen??

as I said -- it was so logical ;-)  I think this aspect indeed was resolved for me as well

no worries.
DOC: minor fix in extract docstring
State: closed


Labels:

Comments:
@jorisvandenbossche pls review and merge!

@danielballan minor issue here: http://pandas.pydata.org/pandas-docs/dev/basics.html#vectorized-string-methods (scroll down a bit), I think just need `.str` accessor on the examples....

True. Fixed.

OK, and the rest is looking good. So I can merge my first PR
Thanks!
DOC: change doc refs to 0.13.1
State: closed


Labels:

Comments:
@jtratner @y-p  ok?

don't need to change setup? (its set at 0.13.0 until next release, right?)

Changes look ok to me.
Don't know about setup.py, that's up to you and @jtratner.

will just leave the version nums as is for now (and they make sense now that 0.13.1 is (maybe) the target)..

if we decide to switch to 0.14 then it won't matter anyhow as it will be reset for 0.14
DOC: add 'pandas ecosystem' section to docs
State: closed

cc @jseabold, @glamp, @mwaskom, @kjordahl, @wrobstory

Feel free to suggest a better blurb if you'd like.

Replaces the `related libraries` section, which I feel is outdated.

closes #5804


Labels:

Comments:
:+1:

nice!

maybe add a small addendum on create the next release setup

e.g. adding section in release notes / whatsnew / setting the new release nums?

sure, just add it to the wiki page.
DOC: Add example to extract docstring, and re-explain change to match.
State: closed

@jreback, I added examples per your request in #5099. Also, I rewrote the explanation of how `str.match` is changing, which I left confusing and wordy before.


Labels:

Comments:
Nice examples in the docstring! I added some comments related to numpy's docstring standard.

And a suggestion, while you are at it, maybe you could also add a 'See also' section to refer to `match` and `contain`?

looks good....

lmk when you are ready for merging...

Thanks for the help, @jorisvandenbossche . I read up on the numpy documentation standard. Ready to merge.

@danielballan thanks...the py2.6 failure was bogus...

@danielballan I added some more comments. Do you want to handle it a new PR (otherwise I can also do it)?

Ah, I thought you wanted output only for the first example, for brevity. I should have asked for clarification. See #5838 .

@danielballan yeah, maybe that wasn't very clear. But I think the examples are much more valuable to users if you can see in a glimpse the possible outputs.
BUG: Fix DatetimeIndex.insert() with strings.
State: closed

Falls back to object Index instead. (previously wasn't checking for them), but _only_ strings are allowed.

Fixes #5818.


Labels:

Comments:
This fails tests because of restrictions on partial setting from #4940. I'm not actually convinced that the tests are correct (or at the very least the issue needs to be handled elsewhere).  I'll circle back to this tomorrow.

This PR makes `DatetimeIndex` match `Int64Index`:

``` python
In [2]: ind = pd.Index(range(10))

In [3]: ind.insert(3, '5')
Out[3]: Index([0, 1, 2, u'5', 3, 4, 5, 6, 7, 8, 9], dtype='object')
```

I'm not clear on why partial setting that has incompatible dtype labels is actually wrong, especially because you can currently do it with non-DatetimeIndex:

``` python
In [4]: df = pd.DataFrame(range(10))

In [5]: df
Out[5]:
   0
0  0
1  1
2  2
3  3
4  4
5  5
6  6
7  7
8  8
9  9

[10 rows x 1 columns]

In [6]: df.loc['apple', :] = 2

In [7]: df
Out[7]:
       0
0      0
1      1
2      2
3      3
4      4
5      5
6      6
7      7
8      8
9      9
apple  2

[11 rows x 1 columns]
```

You could end up with something similar by concatenating DataFrames with different Index dtypes or by doing a join, so not sure it really makes sense to block partial setting via loc/ix.

https://github.com/jtratner/pandas/pull/7

Just to summarize @jreback 's resolution for those who don't want to follow the link:
1. You can't concatenate floats/ints to DatetimeIndex, because internally DI is represented by int64, so it's too likely you'd get something wrong.
2. You are allowed to concatenate strings.
3. Concatenating something like the string `'2001-01-01'` should probably also result in object Index

@jtratner just need a release note....then go ahead and merge..thxs

@jtratner this looked fine...thought it was merged...

can you rebase and merge it?

@jtratner ?

@jtratner ?
CLN: Make io/data urls easier to monkey-patch
State: closed

This could be useful to make pandas more resilient to basic url changes
like that which happened with yahoo finance.  That said, clearly
wholesale API changes won't be helped by this.

What do you all think, worth it to make this relatively trivial change?


Labels:

Comments:
What about factoring out the function that generates the URL it self so that could be monkey patched. That way pandas can be resistant to more than just changes to the base URL.

Certainly something to consider, but most of the functions are basically
just params --> transform --> pass url to common function anyway.

This looks like a good incremental change to me. merging. Don't think something more 
elaborate is called for.
DOC: add way to document DatetimeIndex field attributes
State: closed

Ping @rockg. The docstring is now listed in the autosummary table in api.rst.

Related to issue #5813 (but does not closes it, this only documents the DatetimeIndex field in the api.rst, not more general the Timestamp values).

I added the attribute docstring within the `_field_accessor` function. Is this a good approach? 
You can also document attributes with a docstring line beneath the definition (see Sphinx docs: http://sphinx-doc.org/ext/autodoc.html#directive-autoattribute). The problem with this is that, for the moment, this works for the sphinx autodoc (so the generated pages), but not for the autosummary (there is an open PR for this: https://bitbucket.org/birkenfeld/sphinx/pull-request/142/make-autosummary-work-with-module-class/diff).


Labels:

Comments:
@jorisvandenbossche I would say this closes #5813 the issue of wether certain datetime index methods should exist on Series as well is addressed by another issue 

(actually it's pretty easy to fix just need a common mixin for series/DatetimeIndex)

the issue though is that these should only apply if the Series has an appropriate dtype (so need some error checking)

@jorisvandenbossche I take that back

#5813 is about Timestamp - so fine to leave open

same approach is needed though - a mixin should work

Very good @jorisvandenbossche.  I didn't see that it was Timestamp either.  It's not clear where that method comes from (it has to be on the _Timestamp class, but where is that?).  Nevertheless, thank you for adding to the DatetimeIndex docs!

The method comes directly from datetime.datetime if I am not mistaken (as Timestamp subclasses datetime). E.g. if you type `pd.Timestamp.weekday?`, you will see the correct docstring from datetime. 
So reagarding issue #5813, the thing left could be to a) add some explanation to one of the tutorials, and/or b) add Timestamp to the api.rst (but this causes problems, see #5218).

@jorisvandenbossche looks fine...merge?

@jreback For me it's OK to merge. I suppose the way I add the docstring is rather straightforward.
BUG: Series replace values using timestamps in a dict GH5797
State: closed

This fixes issue #5797.


Labels:

Comments:
Sorry about the extra pull requests.  I made a few mistakes in git.

@MichaelWS, please include context information in the PR title not just an issue number.

This is an update to fix  a minor bug.  Sorry, jreback and I had a long
conversation about this and I had a git issue in my last PR. Values are
cast to numpy  dtypes so we need to compare the replace using the
datetime64.
On Dec 31, 2013 9:47 PM, "y-p" notifications@github.com wrote:

> @MichaelWS https://github.com/MichaelWS, please include context
> information in the PR title not just an issue number.
> 
> —
> Reply to this email directly or view it on GitHubhttps://github.com/pydata/pandas/pull/5806#issuecomment-31417698
> .

what would be needed to include this pr

good
just need to change release notes to 0.14 section for bug fixes (it's near the top) (you may have to rebase to see the updated notes)

so I need to simply:
git fetch upstream
git rebase upstream/master

Last time, my repo got off track of the github repo.  so I fetched upstream and then rebased. I was not sure if I should simply git push -f with my local.  
otherwise, I get this error: 
hint: Updates were rejected because the tip of your current branch is behind
hint: its remote counterpart. Merge the remote changes 

yes u need to force push
which rewrites the local repo

Thanks, this is done.    This was my confusion last time.  I should be much better going forward with it.  This is fixed

merged!

thanks....see they DO get easier!

thanks.  Much easier now.
update to allow for replace using timestamps with a test. fixed issue 5797
State: closed


Labels:

Comments:
the release notes need to go in the 0.13 section
(you might need to rebase on master to see it)
if you didn't already 

just for clarification, I should be able to fetch upstream and rebase then delete any commit I do not want

yep

git is actually pretty forgiving 

another way to approach this (I know u fixed but just FYI)

you can create another branch off of master then cherry pick the commits that u want to keep 

git branches are cheap so encouraged to use them

( I am not familiar with mercurial but some old svn did not encourage branching)

it's nice to figure it out yourself - future PRs become easier

Thanks I agree with you on figuring it out. (But a lot of time this stuff
gets really frustrating.)

I broke everything this time by deleting four spaces.   I will have to read
a few tutorials on git to try to get a better grasp of what I am doing wrong

On Mon, Dec 30, 2013 at 9:23 PM, jreback notifications@github.com wrote:

> yep
> 
> git is actually pretty forgiving
> 
> another way to approach this (I know u fixed but just FYI)
> 
> you can create another branch off of master then cherry pick the commits
> that u want to keep
> 
> git branches are cheap so encouraged to use them
> 
> ( I am not familiar with mercurial but some old svn did not encourage
> branching)
> 
> it's nice to figure it out yourself - future PRs become easier
> 
> —
> Reply to this email directly or view it on GitHubhttps://github.com/pydata/pandas/pull/5805#issuecomment-31380932
> .