BrandonFors/pandas-github-issues huggingface dataset:
be more careful with half-opened date_range
State: closed

closes #11804 

This changes the definition of 'left' and 'right'. If the generated_dates are strictly within (start, end), then changing the value of closed has no impact. Before the code would always remove the leftmost date or rightmost date independently of the value of start and end.


Labels:
name: Bug
name: Datetime
description: Datetime data dtype
name: Frequency
description: DateOffsets

Comments:
need some tests for this. pls use all of the examples that you provided on gitter.

lgtm. can you add a whatsnew note (bug fixes), and squash. ping when green.

@jreback done. I wasn't sure if there is a specific order for the whatsnew notes, let me know if I didn't do it right. Thanks.

@thrasibule thanks! no specific order in whatsnew, best to put it in the middle somewhere as that tends to not cause merge conflicts (and the reason I leave space for it)

Thanks for merging!
Fix import-time DeprecationWarning on Python 3.5
State: closed

Fixes the following:

```
$ python3.5 -Wall
Python 3.5.0 (default, Sep 18 2015, 09:31:01) 
[GCC 4.4.7 20120313 (Red Hat 4.4.7-11)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pandas.tslib
pandas/__init__.py:7: DeprecationWarning: bad escape \s
  from pandas import hashtable, tslib, lib
```

In Python 3.5 re.sub replacement patterns containing unrecognized character escapes are deprecated.  This raw string format is needed to substitute `\s` literally.


Labels:
name: Clean

Comments:
any idea why this doesn't show the same warning? (this is not in cython)

```
(py3.5)bash-3.2$ python -Wall
Python 3.5.0 |Continuum Analytics, Inc.| (default, Oct 20 2015, 14:39:26) 
[GCC 4.2.1 (Apple Inc. build 5577)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from re import compile as re_compile
>>> re_compile('\s+')
re.compile('\\s+')
```

@jreback This doesn't affect `re.compile()` since to it `\s` is a recognized escape sequence in the context of a regular expression.  In a replacement pattern to `re.sub`, however, it is not a meaningful escape sequence (in the regular expression language), so it needs to be passed literally `\\s` in order for it to replace with the literal `\s`.

thanks!
BUG: GH11786 Thread safety issue with read_csv 
State: closed

closes #11786 

Fixed an issue with thread safety when calling read_csv with a StringIO object.

The issue was caused by a misplaced PyGilSate_Ensure()


Labels:
name: Bug
name: IO CSV
description: read_csv, to_csv

Comments:
@jdeschenes gr8! can you add in the example from the issue as a smoke test. (e.g. just have it run), then read in with a single trhead and compare.

and pls add a release note when you are satisified.

Alright, I did this quickly as I don't have time to work on this right now. How long until next release?

@jdeschenes oh have a while.....when you have a chance...thanks!

@jdeschenes if you can have a look at this again would be great.

ping! @jreback

Thank you both for keeping up on this.

@jdeschenes IIRC this issue is repro with actual files. Is that not the case? is it only `StringIO/BytesIO`. are they not thread-safe?

@jdeschenes can you respond to my comments.

Hi @jreback,

the issue is solely reproducible with StringIO. The root cause of this bug is in function buffer_rd_bytes in
pandas/src/parser/io.c. This function is _only_ used when a StringIO/BytesIO is passed to the read_csv function.

The function was calling Py_XDECREF before ensuring that the thread had the GIL. This behavior could not be seen before since the GIL was always locked throughout the read_csv function call.

I am not aware of any issues when reading from disk and this pull request will not fix any problem related to this.

I think that the release notes should be kept as is.

Let me know what you think.

ok, can you add a test that validates the issue that reading from a disk with multiple threads is ok (so we don't regress).

pls run `git diff master | flake8 --diff` as much PEP checking has been one on these files.

FWIW using `BytesIO` has actual use cases in distributed computing, it isn't just a test case.  

Many parallel storage systems won't give you access to the hard disk but will instead deliver a bunch of bytes.  In this case the best way I've found to use `pd.read_csv` is to hand it a BytesIO object.

@mrocklin oh of course. just covering the bases. I suspect people have tried multi-threading to read files as well :)

It would be very interesting to see if there is any benefit in using a ThreadPool for reading from a BytesIO. We are spending a lot of time into the GIL, thanks to the buffer_rd_bytes function. It should probably be benchmarked.

I have a suspicion that it doesn't help at all(It might be even a net loss).

I added the test for the file read. I didn't do it for the BytesIO. The code would effectively look a lot like what I did up top... Grabbing a list of BytesIO and processing them in a ThreadPool. I can take a look at this a bit later, if that is required.

@jdeschenes thanks!

certainly would take addtl benchmarks / fixes! 

Hey all - any estimate of when this will be go out in a production release? Encountering this bug very very frequently with 0.17.1, and would like to get back up to a newer version of pandas again soon

Thanks

planning on a RC in about 2 weeks, so release should be roughly mid-feb or so
DOC: Add context to pd.to_timedelta examples
State: closed

Update the reStructuredText tutorial on timedeltas to include
context, similar to edits suggested for #11785.

Also, add a space after various commas (PEP8), and periods/colons
at the end of sentences for consistency.


Labels:
name: Docs

Comments:
I've made these changes, and updated the commit.

@proinsias thanks!
ENH: allow saving of meta-data via CArrays to support wide tables
State: closed

supersedes #10243 
closes #6245

Here is the code as it stands now.  Appending MultiIndex axes now works.  The one test that still fails is  test_append_frame_column_oriented: writing the DateTimeIndex as columns fails because the index is converted to an array of timestamps so np.array creates an object array.  If it could be passed down unmolested np.array() would do the right thing.  I don't yet have a good idea how to fix that.

I added a test for reading the old format, let me know if you think it's sufficient coverage.  I've been testing the reading more broadly against my own data.  


Labels:
name: Enhancement
name: IO HDF5
description: read_hdf, HDFStore

Comments:
so, seeing that we are now generating legacy files for hdf5, I think its time to expand [this](https://github.com/pydata/pandas/blob/master/pandas/io/tests/generate_legacy_storage_files.py) generator to do it; you can see how we do this for msgpack/pickle. need to expand this to write several types of files for hdf5 (e.g. Series,DataFrame,Panel), with varying datatypes (you can prob use a lot of what is there now).

Then we just import this to compare, makes creation of legacy files real easy.

@MJuddBooth can you update

Yes, I re-factored a bit before the holidays and all the tests looked good but year-end work stopped me before I could finish and push.  I can get back to it today or tomorrow.

closing, but pls reopen if you'd like to finish this up. Its a nice change.

@MJuddBooth Any chance you would be able to finish this up? I know it has been two years...
Oh my.  I need to upgrade my fork to 0.23 so I will try to revisit this week.
Yay :champagne: ! Thanks a lot! I am happy to help by the way if there is something delegable.
@MJuddBooth I'm quite interested in this feature being rolled out and it seems you're already pretty far into the implementation. Do you have any updates on it? Please let me know if I can help in any way!
Yeah, seems it's dead again @MJuddBooth ?
DOC: Add examples for pandas.tseries.timedeltas
State: closed

Copied examples from the reStructuredText [tutorial](http://pandas.pydata.org/pandas-docs/stable/timedeltas.html) on timedeltas.


Labels:
name: Docs

Comments:
@proinsias Thanks a lot for working on examples in docstrings!

Besides the inline comments, another general: maybe it would be nice to have some text between the different examples to somehow put them a bit into context, eg:

```
Parsing a single string to a Timedelta:

>>> ...
>>> ...

Parsing a list or array of strings:

>>> ...

Converting numbers by specifying the `unit` keyword argument:

>>> ...

```

Thanks for the advice! – I've made these changes and re-pushed the branch. 

thanks!
BUG: to_numeric should raise if input is more than one dimension #11776
State: closed

closes #11776 


Labels:
name: API Design
name: Error Reporting
description: Incorrect or improved errors from pandas

Comments:
pls add a whats new note (you can do in bug fixes)

@jreback it's updated 

@jreback @jorisvandenbossche updated again 

travis is green now 

thanks!
BUG: [GH 11738] Fix for end_time when multiple of a frequency is requested
State: closed

closes #11738 
1. Added a new test with the multiple of a frequency.
2. Rearranged some tests to improve readability of the code.
3. Added bug fix to whatsnew
4. Added a comment in the test

Didn't follow the contributing guidelines carefully in PR #11771 and hence this new PR. 


Labels:
name: Bug
name: Period
description: Period data type

Comments:
merged via https://github.com/pydata/pandas/commit/4fbf5e51f920dd1019360495e7eecd94016a7a9d

thanks!
BUG/API: QuarterBegin n=0 #11406
State: closed

closes #11406 - makes `QuarterBegin` for `n=0` match Month/Year.

xref #11427 - once this is merged the failures there should pass - there is also a little more documentation about general offset semantics in that PR.


Labels:
name: Datetime
description: Datetime data dtype
name: Frequency
description: DateOffsets

Comments:
cc @cancan101 
cc @seth-p

look ok to you?

Looks fine to me,  though I don't recall ever commenting on offsets.

On Dec 7, 2015, at 1:24 PM, Jeff Reback <notifications@github.com<mailto:notifications@github.com>> wrote:

cc @cancan101https://github.com/cancan101
cc @seth-phttps://github.com/seth-p

look ok to you?

## 

Reply to this email directly or view it on GitHubhttps://github.com/pydata/pandas/pull/11777#issuecomment-162493690.

From #8435 does this now return something sensible?

```
pd.Timestamp('11/2/2012', tz='US/Eastern') + pd.tseries.offsets.QuarterBegin()
Out[29]: Timestamp('2012-12-01 00:00:00-0500', tz='US/Eastern')
pd.Timestamp('11/2/2012', tz='US/Eastern') + pd.tseries.offsets.QuarterEnd()
Out[30]: Timestamp('2012-12-31 00:00:00-0500', tz='US/Eastern')
```

cc @MichaelWS

Looks sensible to me

@rockg I believe that @chris-b1 next PR, #11370 wiill address that (and is dependent on this PR)

merged via https://github.com/pydata/pandas/commit/56fd4e34a1d8ec8e82bcaa37ba0237ec94431834

thanks!
BUG: Parsing offset strings with non-zero minutes was incorrect
State: closed

Closes #11708

Minutes needed to be added to the hour offset rather than subtracted.


Labels:
name: Bug
name: Timezones
description: Timezone data dtype

Comments:
this does not close #11481 , correct?

Correct, that's a lot uglier.

Tests added for to_datetime and Timestamp constructor and are green.

thank you sir!
BUG: Fixed DataFrame info() for duplicated columns, GH11761
State: closed

See #11761.


Labels:
name: Bug
name: Output-Formatting
description: __repr__ of pandas objects, to_string

Comments:
pls add a whatsnew note (bug fix) as well

merged via https://github.com/pydata/pandas/commit/43edd83b31660958179c26a205a51f10df236594

thanks!
BUG: [GH 11738] Fix for end_time when multiple of a frequency is requested
State: closed

closes #11738 
1. Added a new test with the multiple of a frequency. 
2. Rearranged some tests to improve readability of the code.


Labels:
name: Bug
name: Period
description: Period data type

Comments:
looks good. can you add a whatsnew note (bug fix), ping when green

@jreback Sorry, didn't follow contributing guidelines carefully in this PR. Please see PR #11779.

ok, np. yeah, just keeping the original PR is much better / easier to follow the flow
MNT: simplify AutoDateFormatter
State: closed

- import constants from matplolib.dates
- do not re-implement AutoDateFormatter functionality that
  is already upstream


Labels:
name: Visualization
description: plotting
name: Clean

Comments:
can you show an example of where this was doing the wrong thing? (e.g. a test for this)

There is nothing wrong, it just was an almost copy-paste job from mpl to pandas.  This change should have 0 functional change.

ok, seems like we were just overriding mpl code. Since everything passed, this either was never tested (probably), or works ok :)
WIP: display_format for style
State: closed

Unfinished at the moment. Just wanted to push something up since there are other PRs that could build on this.

This moves the logic of "what value should I display for this cell?" from the Template to the `Styler` class itself. The two biggest benefits are
1. User extensibility: Users provide formatting functions (via an API that's TBD) which are called upon the data values when we `translate`.
2. Maintainability: Instead of complex logic in the template, we have (slightly) less complex logic in the class itself.


Labels:
name: Output-Formatting
description: __repr__ of pandas objects, to_string
name: IO HTML
description: read_html, to_html, Styler.apply, Styler.applymap

Comments:
Closing in favor of #11667
Consolidate list-like checking to is_list_like
State: closed

Closes https://github.com/pydata/pandas/issues/11740

Let me know what tests would be helpful for these types of 'cleaning' PRs, and whether a "what's new" is needed.


Labels:
name: Clean

Comments:
@kawochen green. 
In the end, I'm not sure it's much simpler, although it is a bit more encapsulated & explicit
Let me know your thoughts

@MaximilianR update?

Yup, on it. Aiming for this w/e.

closing, but pls reopen if working on it

OK, apologies for letting this drop.

np again. your contributions are great! just trying to move things forward.
ENH : GH11729 added index parameter in series to_string
State: closed

Fixes #11729 
Please Review .


Labels:
name: Output-Formatting
description: __repr__ of pandas objects, to_string
name: Compat
description: pandas objects compatability with Numpy or Python functions

Comments:
pls add a whatsnew note (enhancements is ok)

@varun-kr can you update

@jreback  Updated code . Please review .

It seems that there is some problem with the build . 
ERROR: test_get_all_data (pandas.io.tests.test_data.TestYahooOptions) is failing for many builds.

just push again; the option tests occasionally fail

@jreback  Done. 

@varun-kr thanks!

can you open an issue to re-examine whether passing `index=False` should add those leading spaces (I think not).
Improve(?) explanation of SettingWithCopy warning
State: closed

After playing with R a bunch, I started feeling like the explanation of `SettingWithCopy` wasn't getting to the core of the matter, which is actually an essential consequence of python slice assignment semantics. Here's how python handles chained assignment:

``` python
df['foo']['bar'] = quux
# becomes
df.__getitem__('foo').__setitem__('bar', quux)
```

whereas in R, it's this:

``` R
df["foo"]["bar"] <- quux
# becomes
df["foo"] <- `[<-`(df["foo"], "bar", quux)
# becomes
df <- `[<-`(df, "foo", `[<-`(`[`(df, "foo"), "bar", quux))
```

That last is a lot of line noise, though the R method names ``[`` and ``[<-`` are more concise than `__getitem__` and `__setitem__`! But imagine that you could call `__setitem__` with a kwarg `inplace=False` that would cause it to return a modified copy instead of modifying the original object. Then the R version would translate to this in python:

``` python
df = df.__setitem__('foo',
                    df.__getitem__('foo')
                        .__setitem__('bar', quux, inplace=False),
                    inplace=False)
```

This is incredibly awkward, but it has the advantage of making `SettingWithCopy` unnecessary&mdash; _everything_ is a copy, and yet things get set nonetheless.

So this commit is an attempt to explain this without requiring the reader to know R.


Labels:
name: Docs

Comments:
thanks!

but you are making a wild assumption here, namely that users actually **read** documentation! (lmao)

note https://github.com/pydata/pandas/pull/11500 (hopefully going to happen for 0.18.0)

Well, it'll be nice if this is obsolete by the time it hits a release, but it was fun to write anyway. :) And users **do** read documentation. Sometimes. If you go on StackOverflow and direct them to it.

Squashed. Just noticed I used "You might/may be wondering" to start two consecutive paragraphs. Oh well.

ok, feel free to edit again :)

There. Plus some emphasis. :)

ok thanks! (and pls look at the built docs and see if anything needs tweaking)......usually at least 1 hr before travis builds..
Fix to BUG GH11698. added default value of mask
State: closed

closes #11698

Please review.


Labels:
name: Bug
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: Internals
description: Related to non-user accessible pandas implementation

Comments:
minor change, ping when green.

modified test case. 
This branch has some conflicts . Please check.

u need to fix the conflict 

@jreback  There is a conflict in whatsnew file. When I am adding the bug fix message and trying to merge with the latest code , it always shows me conflicts. I merged my whatsnew file with the latest code but there is still a conflict. Please help.

lots of help on this: http://githowto.com/resolving_conflicts

you basically edit the conflicted file, picking the correct parts out. then add the files, rebase, and push

merged via https://github.com/pydata/pandas/commit/8ec8487a4a8362ac94630388f217f273b3ad1525

thanks!

Thanks
ENH: Add support for s3 in read_excel #11447
State: closed

closes #11447 

Updated excel.py to use pandas.io.common.get_filepath_or_buffer (handles s3 urls) instead of urlopen, only when _is_s3_url returns true.    This was the minimally invasive change.


Labels:
name: IO Data
description: IO issues that don't fit into a more specific label
name: IO Excel
description: read_excel, to_excel

Comments:
tests pls

pls add a test for this. you can post an `.xls` file and I will put it up on the pandas s3 bucket.

pls add a whatsnew note as well (enhancements)

Will do. What's the path to the s3 bucket?
Will use [test1.xls](https://raw.githubusercontent.com/pydata/pandas/master/pandas/io/tests/data/test1.xls) and [test1.xlsx](https://raw.githubusercontent.com/pydata/pandas/master/pandas/io/tests/data/test1.xlsx) for testing if you can put them in the bucket for me.

ok I uploaded those 2 files to the standard bucket `https://s3.amazonaws.com/pandas-test/test1.xls` and `.xlsx`. lmk if you cannot access them.

Can you go ahead and add [test1.xlsm](https://raw.githubusercontent.com/pydata/pandas/master/pandas/io/tests/data/test1.xlsm) to the s3 bucket as well?  Looks like the existing excel tests use all three extensions and I want to use the same base class.   

ok, should be good to go.

ok, PR updated with test case and whatsnew note.

perfect. needs a rebase (prob a doc conflict), then ping when green.

also can you extend the doc-string for `read_excel` (similar to how `read_csv` is done) for the s3

Looks like the doc for `read_excel` already mentions s3 in the same way as `read_csv`.  Does this work?
...`Valid URL schemes include http, ftp, s3, and file.`...

Hmm, also looks like I [broke the build.](https://travis-ci.org/pydata/pandas/jobs/94190434) The unit test is failing when trying to import boto to handle the s3 url: `ImportError: No module named 'boto'` 

Is there a config I need to update for the CI process to install boto?

Ahh, I see I can just implement a `nose.SkipTest`.  Will add a `_skip_if_no_boto()`

yep - builds have different deps / versions of things to give code a workout

Good stuff! Ok, I think everything is good to go now.  Thanks for walking me through my first contribution.

@parsleyt thanks for the contribution!
Truncated float support for SAS Xport files
State: closed

SAS xport files allow floats to be stored with fewer than 8 bytes, with implicit zeros in the high order bytes.  Currently we only handle full width (8 byte) floats.  This PR adds support for truncated floats.


Labels:
name: IO SAS
description: SAS: read_sas

Comments:
seems reasonable. can you add a whats note in enhancements.

merged via https://github.com/pydata/pandas/commit/547750aa5ba5b4a1b5d0cde05cc21e588b30cc27

thanks!
BUG: work around for np.bincount with minlength=0
State: closed

closes https://github.com/pydata/pandas/issues/11699

xref https://github.com/pydata/pandas/issues/11189#issuecomment-152613636


Labels:
name: Bug
name: Groupby

Comments:
can you add a note in whatsnew.

I would also file an upstream bug report.

@behzadnouri thank you sir!
BUG: GH11697 where memory allocation failed in rolling_median
State: closed

closes #11697 
thanks @ajspeck


Labels:
name: Bug
name: Numeric Operations
description: Arithmetic, Comparison, and Logical operations

Comments:
any easy test?

On Linux probably no.  But on windows `RAND_MAX` can be quite small (32767), so you are very likely to hit that problem if you do rolling_median on a decently sized array a few times, which is why this is observed only on windows in the wild so far. 

merged, ty.

note that the reason this was not 'rebuilding' is that the deps were not correct, https://github.com/pydata/pandas/commit/3c4a34e5870d4d1e96e51b364a2eb0ffe40d0afe
BUG GH11693 Support NaT series concatenation
State: closed

Please review .

Closes #11693


Labels:
name: Datetime
description: Datetime data dtype
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Timezones
description: Timezone data dtype

Comments:
couple comments, pls add a whatsnew note

can you update according to comments

apologies for the delay. Will fix over the weekend.

can you update

superseded by #12195
BUG/API: Index creation with different tz coerces DatetimeIndex
State: closed

Closes #11488.

Based on the type check added in #11588, the change affects to fewer functions than I originally thought.


Labels:
name: API Design
name: Timezones
description: Timezone data dtype

Comments:
Thanks, moved to `inferrence.pyx`. Also fixed and added tests for `NaT`.

lgtm. comment above.

Thanks, I've fixed the tz comparison logic based on your comment below.

> do an equality check on tz first then only check get_timezone if it's False

merged via https://github.com/pydata/pandas/commit/b80b5c7443499905eda2e3295ed8973db12a9853

thanks!