BrandonFors/pandas-github-issues huggingface dataset:
ENH: optional ':' separator in ISO8601 strings
State: closed

- [X] closes #10041
- [X] tests added / passed
- [X] passes `git diff upstream/master | flake8 --diff`
- [x] whatsnew entry

Includes some refactoring in the ISO8601 parser. If more tests are desired, please let me know where to put them. Same for the whatsnew entry. This is effectively my first pull request for Pandas.

cc @jreback 


Labels:
name: Bug
name: Datetime
description: Datetime data dtype

Comments:
cc @chris-b1 can you have a look

Looks good as far as I can tell.  The partial time formats are a bit out there, but unambiguous, so I don't see any harm.

``` python
In [24]: pd.Timestamp('2015/01/01 1234')
Out[24]: Timestamp('2015-01-01 12:34:00')
```

can you rebase and push again

thanks @thejohnfreeman !

Thanks for the help!

I normally test on windows before merging....but alas

```
.................................................S.......S.........................................................................................
...................................................
======================================================================
ERROR: test_parsers_iso8601 (pandas.tseries.tests.test_tslib.TestDatetimeParsingWrappers)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Users\conda\Documents\pandas3.5\pandas\tseries\tests\test_tslib.py", line 736, in test_parsers_iso8601
    raise Exception(date_str)
Exception: 2005-0101

======================================================================
FAIL: test_nanosecond_string_parsing (pandas.tseries.tests.test_tslib.TestTimestampNsOperations)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Users\conda\Documents\pandas3.5\pandas\tseries\tests\test_tslib.py", line 902, in test_nanosecond_string_parsing
    self.assertEqual(ts.value, expected_value)
AssertionError: 1367392545123456000 != 1367392545123456789

----------------------------------------------------------------------
Ran 9204 tests in 304.769s

FAILED (SKIP=174, errors=1, failures=1)
```

so if you have the ability to debug on windows, pls have a look :<

Just at a glance, the errors make it seem like my changes to the C code weren't built. I didn't have any platform-specific preprocessor guards in the C, so I would expect it to behave the same on each platform. What are the sources of differences between Windows and Linux behavior? Are the numpy and Python datetime types different on each platform? Are there platform-specific tests?

hmm, I just tested this again on windows and its ok. Why don't you rebase and push again. 

ok, maybe something funny happened in the build.

thanks again!
BUG: indexing operation changes dtype, #10503
State: closed

- [ ] closes #10503 
- [ ] tests added / passed
- [ ] passes `git diff upstream/master | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Bug
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Dtype Conversions
description: Unexpected or buggy dtype conversions

Comments:
thanks @kemingts 

impressive! haven't had anyone dig into indexing code this deeply before!

keep 'em coming!
Add a free-form summary function to describe a dataframe (in the vein of R)
State: closed

First cut. Please let me know your thoughts and thanks again to @jreback for teaching me.

Looks like this 

```
                A_cat            B_str                C_bool           D_num  \
0            count=24         count=24              count=24        count=24   
1       cardinality=2    cardinality=4              mean=0.5         mean=12   
2  value:foo=16 (67%)  value:d=6 (25%)             std=0.511        std=7.07   
3   value:bar=8 (33%)  value:b=6 (25%)         min:p47=False         min=0.5   
4                      value:c=6 (25%)           median=True        p25=6.25   
5                      value:a=6 (25%)          p53:max=True       median=12   
6                                              cardinality=2        p75=17.8   
7                                        value:True=12 (50%)        max=23.5   
8                                       value:False=12 (50%)  cardinality=24   

          E_regular  
0          count=24  
1         mean=4.17  
2          std=3.14  
3         min:p65=2  
4         p70:p82=8  
5         p87:max=9  
6     cardinality=3  
7  value:2=16 (67%)  
8                    
```

for input

```
df = pd.DataFrame({'A_cat': pd.Categorical(['foo', 'foo', 'bar'] * 8),
                        'B_str': ['a', 'b', 'c', 'd'] * 6,
                        'C_bool': [True] * 12 + [False] * 12,
                        'D_num': np.arange(24.) + .5,
                        'E_regular': [9,8,2,2,2,2]*4,
                        })
```
- [ ] closes #12460


Labels:
name: Output-Formatting
description: __repr__ of pandas objects, to_string

Comments:
@vii this is awesome! I would love to use pandas more if they have some of the functions that R has. Can you make the formatting something like this.

```
       E_regular
count       :      24 
mean        :    4.17  
std         :    3.14  
min-p65     :       2  
p70-p82     :       8  
p87-max     :       9  
cardinality :       3  
value-2     : 16(67%) 
```

closing. I think this is a nice idea. but would need a bit more flexiblity & examples to be accepted.
DOC: expanding comparison with R section
State: closed

- [ ] closes #9815
- [ ] tests added / passed
- [ ] passes `git diff upstream/master | flake8 --diff`
- [ ] whatsnew entry

This is the beginning of a quick reference section.  It's incomplete,
just did a rough translation of
http://nbviewer.jupyter.org/urls/gist.githubusercontent.com/TomAugspurger/6e052140eaa5fdb6e8c0/raw/811585624e843f3f80b9b6fe89e18119d7d2d73c/dplyr_pandas.ipynb
into tables.  Should try to get some R experts to comment, and it would
be nice to have the pandas versions link to docs for the functions being
used, but I'm terrible at reStructuredText and gave up for the moment.


Labels:
name: Docs

Comments:
Thanks for this. There's a slightly updated version [here](https://gist.github.com/TomAugspurger/6e052140eaa5fdb6e8c0), but I can't really remember what changed.

reST can be a bit of a pain. For linking to methods, you can use e.g. 

```
:meth:`~pandas.DataFrame.sample`.
```

Yeah, I tried :meth: once and didn't like it.

But seriously, I couldn't get it to format well, or deal with the example
arguments. I'll play with it some more eventually.
On Fri, Feb 26, 2016 at 16:40 Tom Augspurger notifications@github.com
wrote:

> Thanks for this. There's a slightly updated version here
> https://gist.github.com/TomAugspurger/6e052140eaa5fdb6e8c0, but I can't
> really remember what changed.
> 
> reST can be a bit of a pain. For linking to methods, you can use e.g.
> 
> :meth:`~pandas.DataFrame.sample`.
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/pydata/pandas/pull/12472#issuecomment-189492001.
> 
> ## 
> 
> ## 
> 
> Cheers,
> Leif

can you update

@jreback thanks for the poke

@leifwalsh thanks.

I merged this in. Pls have a look at the built docs (prob take a few hours). http://pandas-docs.github.io/pandas-docs-travis/comparison_with_r.html

not really sure if there is a way to make the 3 tables (sorting, transforming, grouping) be the same width .....
CLN: consolidate Series.quantile and DataFrame.quantile
State: closed

- closes #10207
- Slightly modified test in tests/series/tests_analytics.py:575 to standardize behavior across Series and DataFrame
- passes `git diff upstream/master | flake8 --diff`


Labels:
name: Clean

Comments:
Collabed with @sergeibernstein on this, commenting here for alerts on any issues.

can you rebase on master, and some PEP errors. thxs.
MAINT: Nicely inform users if they're missing hard dependencies. 
State: closed

Closes #12176
- [✓] tests pass
  Manually tested with condas environments w/ all and no dependencies.
- [✓] passes `git diff upstream/master | flake8 --diff`
- [✗] whatsnew entry


Labels:
name: Error Reporting
description: Incorrect or improved errors from pandas

Comments:
thanks @mfarrugi
ENH: add missing methods to .dt for Period, resolves #8848
State: closed

- [x] closes #8848 
- [x] tests added / passed
- [x] passes `git diff upstream/master | flake8 --diff`
- [x] whatsnew entry

Not 100% complete, but putting this in as a place to house comments:
- I am not sure where to put the whatsnew entry - 0.18.0 or 0.18.1?
- Also not sure if you'd like tests on the actual usage in the .dt accessor, there are actually no tests anywhere in pandas (as far as I can tell) on period_series.dt.\* (as opposed to PeriodIndex). I just added a couple PeriodIndex tests.


Labels:
name: API Design
name: Period
description: Period data type

Comments:
the tests on `.dt.` usage are all in `tests/series/test_datetime_values.py`, so pls add there

0.18.0 is good.

Ok, I think a proper first pass on this is done now, let me know if there's more you would like.

thank you sir!
BUG: fix df concat containing mix of localized and non-localized Timestamps
State: closed

BUG: fix issue with concat of localized timestamps
TST: test concat of dataframes with non-None timezone columns


Labels:
name: Bug
name: Timezones
description: Timezone data dtype

Comments:
Created #12467 and added it in test comment

xref to #12396 which is closed by #12403 

ping!

Hey Jeff, I've looked into the other failures a little but don't fully understand when/why the expected dims change.  It looks like merge (the tests that fail) always wants a 1d array and is failing because we are using atleast_2d on one of our inputs to make our (2-d, 1-d) input combo compatible.  Perhaps we should instead be making the 2-d into a 1-d?  Is this just playing wack-a-mole?

a bit of a hack, but assure 2d-ness only when we are axis=1 concat.

```
diff --git a/pandas/tseries/common.py b/pandas/tseries/common.py
index f32e9b9..11a5fdc 100644
--- a/pandas/tseries/common.py
+++ b/pandas/tseries/common.py
@@ -260,8 +260,7 @@ def _concat_compat(to_concat, axis=0, typs=None):
         # if dtype is of datetimetz or timezone
         if x.dtype.kind == _NS_DTYPE.kind:
             if getattr(x, 'tz', None) is not None:
-                x = x.asobject
-                x = np.atleast_2d(x)
+                x = x.asobject.values
             else:
                 shape = x.shape
                 x = tslib.ints_to_pydatetime(x.view(np.int64).ravel())
@@ -272,6 +271,8 @@ def _concat_compat(to_concat, axis=0, typs=None):
             x = tslib.ints_to_pytimedelta(x.view(np.int64).ravel())
             x = x.reshape(shape)

+        if axis == 1:
+            x = np.atleast_2d(x)
         return x

     if typs is None:
```

That passes the tests that were failing. Unfortunately, can't push from work.  Will make that change this evening.

@tsdlovell np.

travis is failing on OSX and py3.5 at the same points

> ## ERROR: test_range_slice_day (pandas.tseries.tests.test_period.TestPeriodIndex)
> 
> Traceback (most recent call last):
>   File "/Users/travis/build/pydata/pandas/pandas/tseries/tests/test_period.py", line 2619, in test_range_slice_day
>     idx[v:]
>   File "/Users/travis/build/pydata/pandas/pandas/tseries/base.py", line 176, in **getitem**
>     result = getitem(key)
> 
> # IndexError: failed to coerce slice entry of type str to integer
> 
> ## ERROR: test_range_slice_seconds (pandas.tseries.tests.test_period.TestPeriodIndex)
> 
> Traceback (most recent call last):
>   File "/Users/travis/build/pydata/pandas/pandas/tseries/tests/test_period.py", line 2676, in test_range_slice_seconds
>     idx[v:]
>   File "/Users/travis/build/pydata/pandas/pandas/tseries/base.py", line 176, in **getitem**
>     result = getitem(key)
> IndexError: failed to coerce slice entry of type str to integer

Was working in the env I created a bit ago, but fails in an env I just created, so maybe something to do with change in dependencies.   Will look into it.

numpy 1.11 was just updated fully in conda
I am about to push the fixes for this

rebase and should be good to go

@tsdlovell thank you sir!
ENH: Improve error message for empty data constructor. #8020
State: closed

- [X] closes #8020
- [X] tests added / passed
- [X] passes `git diff upstream/master | flake8 --diff`
- [X] whatsnew entry


Labels:
name: Error Reporting
description: Incorrect or improved errors from pandas

Comments:
can you add a whatsnew entry (API Changes), mentioning that this error message is changed

Yup, added a line to the API changes.

thanks @jylin
ENH: replace uses of np.isscalar with pd.lib.isscalar
State: closed

Closes #8708

Added note that pd.lib.isscalar() returns True for None, which is different from np.isscalar().


Labels:
name: Internals
description: Related to non-user accessible pandas implementation
name: Compat
description: pandas objects compatability with Numpy or Python functions

Comments:
I don't see any tests for `lib.isscalar` itself. Can you add some in `test_common.py` (and of course include `None` for this)

Tests for `lib.isscalar` are already in `test_lib.py`.

https://github.com/pydata/pandas/blob/master/pandas/tests/test_lib.py#L189

Do you want these moved to `test_common.py`?

@chrisaycock thanks!

FYI, build was failing because of some flake warnings.
describe() outputs bool as categorical
State: closed

- [ ] closes #6625
- [ ] tests passed so far
- [ ] passes `git diff upstream/master | flake8 --diff`


Labels:
name: Enhancement
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: Numeric Operations
description: Arithmetic, Comparison, and Logical operations

Comments:
pls add a whatsnew entry. put in API changes.

thanks @vii
Update what's new page with relevant issues for df.query() with in-pl…
State: closed

- [x] closes #10486
- [N/A] tests added / passed
- [x] passes `git diff upstream/master | flake8 --diff`
- [x] whatsnew entry

…ace operator

The relevant issues added are :
BUG: query with invalid dtypes should fallback to python engine #10486
BUG: query modifies the frame when you compare with `=` #8664


Labels:
name: Docs

Comments:
this is actually editing an old whatsnew but ok.
DOC: Add examples and notes to empty documentation
State: closed

- [x] closes #12393 
- [x] passes `git diff upstream/master | flake8 --diff`

Added notes and examples to provide clarity around the issue discussed in #12393. This is my first contribution to pandas, so let me know if I need to fix anything.


Labels:
name: Docs

Comments:
Thanks! 

Further comment about the examples, I would start with an example of an actual empty frame (eg a DataFrame with only index), and only then show the example with the NaNs

Thanks to both of you. Your suggestions make sense to me. I'll make those tweaks and I thought of one additional in line with yours:
- make comments actual sentences
- start with empty df in example instead of NaN df
- show output of empty df
- also show output of NaN df

That sounds good!

Looks like travis hit a 404 from continuum's repo? I'm not seeing a 404 on my end.

`Error: HTTPError: 404 Client Error: Not Found for url: https://repo.continuum.io/pkgs/free/linux-64/setuptools-20.1.1-py27_0.tar.bz2: https://repo.continuum.io/pkgs/free/linux-64/setuptools-20.1.1-py27_0.tar.bz2`

Their servers were having issues last night. I've restarted the build.

thanks!
DOC: improve documentation for building extensions on windows
State: closed


Labels:
name: Build
description: Library building on various platforms
name: Docs

Comments:
cc @jorisvandenbossche 
cc @Dr-Irv 

@jorisvandenbossche can you give a try with 3.5 when you have a chance, then update this as to steps that are needed. I don't think you actually need to do any pathing with VS2012. It should _just_ work.

+1 for this.  I am trying to do similar documentation for `numpy` and `scipy` actually, so it's great to see all the libraries coming together at once to improve building for Windows!

Maybe have instructions for different versions of Visual Studio??  In my case, I need VS2012 for another project (and have a valid license), so I don't want to install VS2015 so that things don't get messed up on my laptop.

@Dr-Irv once these are installed (and they can all be installed separately), they are independent. You shouldn't need any addtl after these installations. On a VM I did exactly that.

@jreback My colleagues have had issues in other projects (.NET, C++, etc.) where having multiple versions caused issues.  So I followed their advice.  You have to be really careful with the PATH. 

@Dr-Irv that's a very bad idea.  You should have all compiled software in a single process (e.g. Python, pandas) running with the same MSVC runtime.  Bad things can happen if you mix MSVC versions (and thus, runtimes).  Longer explanation here: http://siomsystems.com/mixing-visual-studio-versions/

The versions suggested above adhere to community standards.  If you build your package with VS 2012, your packages are questionably compatible.  If you want to build your package with VS2012, you really should rebuild your whole Python stack with VS 2012.

@msarahan Thanks for the pointer. 

Is there an alternative to installing the full Visual Studio 2015 for python 3.5 (as that takes 4 Gb) 
@gfyoung in your PR for numpy/scipy (https://github.com/scipy/scipy.org/pull/146), you point to the url https://support.microsoft.com/en-gb/kb/2977003 for Visual C++. These are downloads for "Visual C++ Redistributable Packages", which is much smaller to install, but are these enough to build a library? As I understand they are for running an application that is already built? 

no u have to install everything 

There is https://www.microsoft.com/en-us/download/details.aspx?id=48159 which claims to include the latest compiler from VS 2015, without VS itself.  I haven't tried it, and can't speak for its functionality or content.

@gfyoung Is it possible that you then link to the wrong page in the numpy/scipy PR?

@msarahan Thanks. That still needs the full .NET framework to be installed (which is also 2 GB), but still it's a bit smaller

@jorisvandenbossche : So it seems.  If you could put a comment in my PR to change the link(s), that would be great!  Thanks!

so are the doc links right?
(i am going to fix the markdown links -> rst links) shortly

http://pandas-docs.github.io/pandas-docs-travis/contributing.html#creating-a-windows-development-environment
Notes about unique vs cat.categories #12278
State: closed

- [x] closes #12278
- [ ] tests added / passed
- [x] passes `git diff upstream/master | flake8 --diff`
- [x] adding notes about .unique() and .cat.categories (as suggested in [#12278 (comment)](https://github.com/pydata/pandas/issues/12278#issuecomment-182445522))


Labels:
name: Docs
name: Categorical
description: Categorical Data Type

Comments:
thanks @Dorozhko-Anton
DOC: xray -> xarray
State: closed

@shoyer just came across this in the docs. These should be the only (remaining) mentions of xray


Labels:
name: Docs

Comments:
thanks!

Indeed, thanks!
On Mon, Feb 22, 2016 at 8:45 AM Jeff Reback notifications@github.com
wrote:

> Closed #12419 https://github.com/pydata/pandas/pull/12419 via cb3b63f
> https://github.com/pydata/pandas/commit/cb3b63f46c6f63645ae3741617812ed3e0312973
> .
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/pydata/pandas/pull/12419#event-559941190.
ENH decimal parameter to to_latex and to_html, #12031
State: closed

- [x] closes #12031
- [x] tests added / passed
- [x] passes `git diff upstream/master | flake8 --diff`
- [x] whatsnew entry

I just pass on the `decimal` parameter. All the machinery was already in place.


Labels:
name: Output-Formatting
description: __repr__ of pandas objects, to_string
name: API Design
name: IO CSV
description: read_csv, to_csv
name: Compat
description: pandas objects compatability with Numpy or Python functions
name: IO LaTeX
description: to_latex

Comments:
@jreback All set!

thanks @nbonnotte
MAINT: More informative TypeError for IndexEngine.get_loc
State: closed

Title is self-explanatory.  Closes #12218.


Labels:
name: Error Reporting
description: Incorrect or improved errors from pandas

Comments:
Are either of the following necessary for this PR:

1) What's New (if yes, which section?)
2) Unit test (if yes, where?)

I don't believe this ever becomes an external exception 

But the issue that I'm closing showed an example where the Exception was thrown

so this is right

so add a test that asserts the contents of the error message

it's s type error because the indexer is not the correct type to select the level 

Added a test, and the test passes on Travis.  Should be good to merge if there is nothing else.

Not really sure why Python 3.4 is failing, as the failing test case has no relation to what I changed.  I'm going to rebase onto `master` so as to trigger Travis again.

Tests are passing, so this should be good to merge if there is nothing else.

can you add a whatsnew note (you can put in bug fixes is fine). ping when pushed.

018.0 is ok

Sorry, as soon as I posted the question, I saw you had already added it to the `0.18.0` milestone, which made the answer somewhat obvious. :smile: 

Could you cancel some of my duplicate, older builds?  I see no reason for them to run given all of the updating I have done.  Thanks!

done, do you know any way to configure that on Travis? (e.g. it builds pull-requests, but I want force pushes to cancel anything but the last one). 

Unfortunately not.  See <a href="https://github.com/travis-ci/travis-ci/issues/964">here</a>.

actually I _think_ this will work as the ref commit is now gone. (if you force push on the same branch) 

I'm not entirely sure I follow what you mean since I do force pushes all time with my PR's in `pandas` and haven't seen that happen.

what I mean is that if you force push, then when Travis tries to build it is fails (but is grey so doesn't count) and doesn't build the previous commit (but instead builds your new one)

On a separate note, is there a `pandas-svn` mailing list (similar to `numpy` and `scipy`) that GitHub can send notifications to to let us know when merges / commits have been made to the `master` branch?  I find such notifications to be quite useful so that PR branches don't get too far behind the `master` branch.

Regarding Travis: that should work I suppose, but I haven't seen that work consistently in practice, at least not in this library.  IDK if there's a way to guarantee Travis can't find the old ref.

@gfyoung where is this list defined?

yes easy to add to Travis

Oh, I meant : does such a mailing list exist?

@gfyoung could create one, what I meant was does `numpy` define one somewhere?

@jorisvandenbossche @TomAugspurger would this be useful?

@jreback : If you visit this link <a href="http://www.scipy.org/scipylib/mailing-lists.html">here</a>, you'll see it there.  Every time PR's get merged into `master` or commits are made, everyone on that list gets notified.

@gfyoung can you make a separate issue about this. (and include that pointer)

thanks!
BUG: date_range breaks with tz-aware start/end dates and closed intervals #12409
State: closed

- [X] closes #12409 
- [X] tests added / passed
- [X] passes `git diff upstream/master | flake8 --diff`
- [ ] whatsnew entry - not added since this fixes a pre-release bug

do comparison for closed intervals on UTC basis prior to tz conversion
added tests for date_range covering case of tz-aware start/end dates with closed ranges


Labels:
name: Regression
description: Functionality that used to work in a prior pandas version
name: Timezones
description: Timezone data dtype

Comments:
right, but can find where it was changed and add this issue number 

put this issue number in the whatsnew next to #11804 

lgtm. squash; ping when ready.

good to go

can you add the test from the issue as well. I am not convinced your tests actually replicate the issue.

added more tests covering original issue

ok, looks good. ping when green.

good to go

thanks!