BrandonFors/pandas-github-issues huggingface dataset:
BUG: consistent datetime display format with < ms #10170
State: closed

closes #10170

```
In [3]: Series(date_range('20130101 09:00:00',periods=5,freq='D'))
Out[3]: 
0   2013-01-01 09:00:00
1   2013-01-02 09:00:00
2   2013-01-03 09:00:00
3   2013-01-04 09:00:00
4   2013-01-05 09:00:00
dtype: datetime64[ns]

In [4]: Series(date_range('20130101 09:00:00',periods=5,freq='s'))
Out[4]: 
0   2013-01-01 09:00:00
1   2013-01-01 09:00:01
2   2013-01-01 09:00:02
3   2013-01-01 09:00:03
4   2013-01-01 09:00:04
dtype: datetime64[ns]

In [2]: s = Series(date_range('20130101 09:00:00',periods=5,freq='ms'))

In [3]: s.iloc[1] = np.nan

In [4]: s
Out[4]: 
0   2013-01-01 09:00:00.000
1                       NaT
2   2013-01-01 09:00:00.002
3   2013-01-01 09:00:00.003
4   2013-01-01 09:00:00.004
dtype: datetime64[ns]

In [6]: Series(date_range('20130101 09:00:00',periods=5,freq='us'))
Out[6]: 
0   2013-01-01 09:00:00.000000
1   2013-01-01 09:00:00.000001
2   2013-01-01 09:00:00.000002
3   2013-01-01 09:00:00.000003
4   2013-01-01 09:00:00.000004
dtype: datetime64[ns]

In [7]: Series(date_range('20130101 09:00:00',periods=5,freq='N'))
Out[7]: 
0   2013-01-01 09:00:00.000000000
1   2013-01-01 09:00:00.000000001
2   2013-01-01 09:00:00.000000002
3   2013-01-01 09:00:00.000000003
4   2013-01-01 09:00:00.000000004
dtype: datetime64[ns]
```


Labels:
name: Bug
name: Output-Formatting
description: __repr__ of pandas objects, to_string

Comments:
Thanks!

Does it make sense to also have the 'milliseconds' resolution? (the third example above, so up to 3 decimals after the comma) instead of at once microsecond

@jorisvandenbossche fixed up for 'ms'. updated the top.

Nice!

@jorisvandenbossche look ok?

yes!
DOC: clarify purpose of DataFrame.from_csv (GH4191)
State: closed

Closes #9556

xref #9568, #4916, #4191 

However, while writing this up, I started to doubt a bit if this is necessary. 
`DataFrame.from_csv` is implemented as a round-trip method together with `to_csv`. If you use a plain `df.to_csv(path)`, you cannnot read it in as `pd.read_csv(path)` to get exactly the same. You at least need `pd.read_csv(path, index_col=True)` and a `parse_dates` keyword if you have datetimes. 

Secondly, there is also the question of `Series.from_csv`. It would be logical to deprecate this as well, but for this you don't directly have an alternative (but `pd.read_csv(path, index_col=0)[0]` will work). There is also for example this SO answer of Wes: http://stackoverflow.com/questions/13557559/how-to-write-read-pandas-series-to-from-csv (and it is used in the Python for Data Analysis book).


Labels:
name: IO CSV
description: read_csv, to_csv
name: Deprecate
description: Functionality to remove in pandas

Comments:
`pd.read_csv(path, index_col=0, squeeze=True)` should work for `Series.from_csv`.

I am also not strong on actually deprecating this particular method (as you say its the inverse of `to_csv`).

let's push this off / close. Not sure this is necessary.

let's just update the docs a tiny bit to note that `DataFrame.to_csv <-> DataFrame.from_csv` (and put it on `pd.read_csv` somewhere? I don't think we can actually deprecate this

Yes, it was on my to do list to update this PR with only the docs part (and not the actual deprecation). 
I think we can still encourage users in the docstring to use `read_csv`, but note that `from_csv` is useful if you want an exact opposite of `to_csv`

which is actually the part from the docstring that I now removed ... :-)

@jreback updated it. Thoughts?

I still have to do the same for `Series.from_csv`

lgtm. DISCOURAGED should be an official term!

lgtm

updated with similar text for Series.from_csv
BUG: Index.name is lost during timedelta ops
State: closed

Closes #9926.

CC @hsperr I've changed `common._maybe_match_names` to work with not-named objects, and hopefully can use in #9965 also.


Labels:
name: Bug
name: Timedelta
description: Timedelta data type

Comments:
while refactoring my code from #9965 to use this I found that it might be nice to have the same method for multiindex with the 'names' attribute instead, what do you think?

It is nice the same logic can be used. please change it accordingly on your pr, and I'll rebase.

thanks @sinhrks
BUG: Index.union cannot handle array-likes
State: closed

Closes #10149.
- Added explicit tests for array-likes in set-ops, `intersection`, `union`, `difference` and `sym_diff`.
  - `TimedeltaIndex` previously did additional input check which is inconsistent with others. Made it work as the same manner as others.
  -  `MultiIndex` set-ops should only accept list-likes of tuples. Implement the logic and add explicit test.
- Changed to understandable error message for `str` which has `__iter__` in py3.

```
# current error msg
idx.intersection('aaa')
# TypeError: Index(...) must be called with a collection of some kind, 'aaa' was passed
```

NOTE: This DOESN'T care the `name` attribute checks, which is being worked in #9965.


Labels:
name: Bug

Comments:
rebase / address comments when you have a chance

Rebased. I've added tests to `Base`, but it cannot work in all Index currently. I'll prepare separate issue about `CategoricalIndex` and `TimedeltaIndex` set ops behavior. 

What I'd like to confirm here is `MultiIndex` should accept list-likes of something? Currently, it raises error not understandable for users as below.

```
>>> midx = pd.MultiIndex.from_tuples([(1, 2), (3, 4)])
>>> midx
MultiIndex(levels=[[1, 3], [2, 4]],
           labels=[[0, 1], [0, 1]])
>>> midx.intersection([(1, 2)])
AttributeError: 'list' object has no attribute 'names'
```

Let me update `TimedeltaIndex` behavior here (because simple than I initially thought).  On current master, `DatetimeIndex.union` performs additional check and it causes an error. Removed this check in and made it consistent with others in this PR.

```
idx1 = pd.timedelta_range('1 day', freq='D', periods=5)
idx2 = pd.timedelta_range('3 day', freq='D', periods=5)

idx1.intersection(idx2.asobject.tolist())
# TimedeltaIndex(['3 days', '4 days', '5 days'], dtype='timedelta64[ns]', freq=None)

idx1.union(idx2.asobject.tolist())
# AttributeError: 'list' object has no attribute 'inferred_type'

idx1.difference(idx2.asobject.tolist())
# TimedeltaIndex(['1 days', '2 days'], dtype='timedelta64[ns]', freq=None)

idx1.sym_diff(idx2.asobject.tolist())
# TimedeltaIndex(['1 days', '2 days', '6 days', '7 days'], dtype='timedelta64[ns]', freq=None)
```

Rebased, and updated top descriptions about `TimedeltaIndex` and `MultiIndex` behaviors.

thanks @sinhrks
ENH: groupby.apply for Categorical should preserve categories (closes…
State: closed

closes #10138


Labels:
name: Enhancement
name: Groupby
name: Categorical
description: Categorical Data Type

Comments:
Does this change anything for groupby.apply in the transform case, rather than the aggregate case? It would be nice to add a test for transform-like apply with categoricals if we don't have one already.

Oh good point ... the following actually doesn't reindex:

```
In [12]: grouped.apply(lambda x : 'foo')
Out[12]:
missing  dense
a        a        foo
         b        foo
         c        foo
dtype: object
```

I'll dig into the code path for this

@mortada I actually think not reindexing is the right behavior here, for transforming grouped operators. The result index should be identical to the input index.

@mortada I think @shoyer is correct; transform returns the original index

As cool thanks guys, indeed I see that in the docs about transform. 

I'll add a test case for this behavior 

@jreback @shoyer it's updated 

OK, looks good to me. Can you add a release note?

ah absolutely. I'm not quite sure what section this should go, I just added it in "other enhancements" 

I'd probably call this a bug fix?

yeah I was actually thinking the same thing, but the original issue was created as an enhancement

@jreback moved this to `v0.16.2`, ready for review 

@mortada gr8 thanks!
BUG: declare and use correctly self.unique_check
State: closed

I am not sure I understood the last comment in [the previous pull request](https://github.com/pydata/pandas/pull/9526), but anyway this is a rebased version.

In the previous PR I proposed to simply make the `__get__` of `is_unique` directly rely on `self.initialized`, but I am no more so sure it is a good idea, since the current status is slightly redundant but clearer (and coherent with the use of "is_monotonic").


Labels:
name: Clean

Comments:
ok, what exactly does this fix? 

First, `self.unique_check` (which follows the same logic of `self.monotonic_check`) is now initialized.

Then, when `is_unique` is accessed, `self.unique_check` [is checked](https://github.com/pydata/pandas/blob/master/pandas/index.pyx#L272). But,`self.unique_check` [is (was) not set when the index is not unique](https://github.com/pydata/pandas/blob/master/pandas/index.pyx#L272). So on a non-unique index, every access to `is_unique` results in a call to `_do_unique_check`, which calls then `_ensure_mapping_populated`, which finally does nothing (in calls following the first) because `self.initialized` is now 1. So the performance loss is minimal, but still it doesn't make any sense.

is  `unique_check` really a duplicate of `initialized`? if so, just remove it. check that everything passes and perf is unchanged.

See https://github.com/pydata/pandas/pull/10229

@toobaz just for future reference, you don't have to open a new PR every time you update. Just update your branch and push again will do!

@jorisvandenbossche  : since I was indeed trying to understand this... by "update" do you mean "push a new commit" or "rebase and push -f"? Because I tried the latter and it didn't seem to work (but I may have been mistaken).

As long as you stay on the same branch, it does not matter. Adding a new commit, rebasing, etc all work and the PR will be updated with that if you push (force) again.

But I see that the new PR uses another branch name. So you only need to stay on the same branch to update the PR

I changed branch name precisely because I didn't think using the same would work! Thanks for the tip.
(WIP) PERF: improve .str perf for all-string values (about 2x-)
State: closed

Related to #10081. Make a short path using numpy's string funcs when all the target values are strings.  Otherwise, use current path.

Followings are current comparison results:

```
import pandas as pd
import numpy as np
import string
import random

np.random.seed(1)
s = [''.join([random.choice(string.ascii_letters + string.digits) for i in range(3)]) for i in range(1000000)]

# s_str uses short path
s_str = pd.Series(s)

# set object 
s[-1] = 1

# s_obj uses current path
s_obj = pd.Series(s)
```

```
%timeit s_str.str.lower()
#1 loops, best of 3: 696 ms per loop
%timeit s_obj.str.lower()
#1 loops, best of 3: 1.46 s per loop

%timeit s_str.str.split('a')
#1 loops, best of 3: 1.55 s per loop
%timeit s_obj.str.split('a')
#1 loops, best of 3: 3.52 s per loop
```

The logic has an overhead to check whether target values are all-string using `lib.is_string_array`. But this should be speed-up in most cases because it takes relatively shorter time than string ops, and (I believe) values should be all-string in most cases. 

```
%timeit pd.lib.is_string_array(s_str.values)
#10 loops, best of 3: 21.9 ms per loop
```

If it looks OK, I'll work on all the funcs which is supported by numpy.
- [ ] Add all numpy funcs
  - http://docs.scipy.org/doc/numpy/reference/routines.char.html
- [ ] Add tests to check numpy results for all-string values are identical as current results.
- [ ] Add vbench (both for all-string and object-included)


Labels:
name: Performance
description: Memory or execution speed performance
name: Strings
description: String extension data type and string data

Comments:
Ah noticed above comparison is not fair, preparing valid ones...

I misunderstood the differnce of str and object is caused by numpy logic. As numpy looks to use similar logic as pandas thus cannot expect such a performance gain... Allow me to close this.

Where did the initial speed-up come from then?

The above difference exists on current master depending on dtypes handled. I've misunderstand it is caused be the logic i've changed, but not.
BUG: Strings with exponent but no decimal point parsed as integers in python csv engine
State: closed

Fixes GH #9565. I also made the handling of out-of-range integers consistent with the C engine: parse as a string rather than a double.


Labels:
name: Bug
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: IO CSV
description: read_csv, to_csv

Comments:
pls give this a perf run on the `-r csv` and see if anything is changed

@evanpw looks good. (FYI if you want to add a confirming case for the out-of-range int64 you have above would be good - not sure that is really tested atm).

and pls run perf and report if anything amiss.

Integers near the boundary of int64 values were getting parsed as strings because the boundary comparison was performed using doubles. That's fixed in this last commit, and some tests added. Checked performance: no measurable difference with master. Rebased and moved whatsnew entry to 0.16.2.

fyi, you may have to rebase again as I just fixed a travis build error about 10 min ago.

couple of issues w.r.t. windows. I think some of the libraries are named differently. So just revert that part of the code. ping after travis is green.

cython libc imports reverted, and tests are green

merged via https://github.com/pydata/pandas/commit/38876df03ada8e962fad06ac64cb343562cb976c

thanks!
BUG: get_group fails when multi-grouping with a categorical
State: closed

Example:

``` .python
>>> df = pd.DataFrame({'a' : pd.Categorical('xyxy'), 'b' : 1, 'c' : 2})
>>> df.groupby(['a', 'b']).get_group(('x', 1))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/evanpw/Workspace/pandas/pandas/core/groupby.py", line 601, in get_group
    inds = self._get_index(name)
  File "/home/evanpw/Workspace/pandas/pandas/core/groupby.py", line 429, in _get_index
    sample = next(iter(self.indices))
  File "/home/evanpw/Workspace/pandas/pandas/core/groupby.py", line 414, in indices
    return self.grouper.indices
  File "pandas/src/properties.pyx", line 34, in pandas.lib.cache_readonly.__get__ (pandas/lib.c:41912)
  File "/home/evanpw/Workspace/pandas/pandas/core/groupby.py", line 1305, in indices
    return _get_indices_dict(label_list, keys)
  File "/home/evanpw/Workspace/pandas/pandas/core/groupby.py", line 3762, in _get_indices_dict
    return lib.indices_fast(sorter, group_index, keys, sorted_labels)
  File "pandas/lib.pyx", line 1385, in pandas.lib.indices_fast (pandas/lib.c:23843)
TypeError: Cannot convert Categorical to numpy.ndarray
```

The problem is that `Grouping.group_index` is a CategoricalIndex, so calling `get_values()` gives you a `Categorical`, which needs one more application of `get_values()` to get an `ndarray`


Labels:
name: Bug
name: Groupby
name: Categorical
description: Categorical Data Type

Comments:
can you put the self-contained example in the top of the PR?

can you update according to comments

Done, and moved to 0.16.2 whatsnew. Does this need an entry in the API changes section now, or is this change too minor?

This seems a bug fix to me, so the whatsnew entry is fine!

yep, pls squash. ping when green.

Tests are green

@evanpw thanks. Soon, waiting on travis to finish up its builds of a bunch of stuff.

@evanpw thanks!
BUG: fix Series.plot label setting
State: closed

Closes https://github.com/pydata/pandas/issues/10119, a regression in 0.16.1

Basically `Series.plot(label='foo')` isn't actually setting the label.

I want to look at one more thing. With this fix you'd need

``` python
s.plot(label='foo', legend=True
```

to get the legend to actually show up. I can't remember what the old behavior was. At least for seaborn, setting a label implies that the legend should be drawn (e.g. `sns.kdeplot(s, label='foo')`)


Labels:
name: Visualization
description: plotting
name: Regression
description: Functionality that used to work in a prior pandas version

Comments:
> I can't remember what the old behavior was. 

Because of the default value of `plot_series`, no legend should appear otherwise specified in prior versions.

https://github.com/pydata/pandas/blob/master/pandas/tools/plotting.py#L2492

That's what I thought. Ok, I suppose we'll keep it like that for now. So `ts.plot(label='series')` will attach the legend information, but not actually draw the legend. 

I think this is ready then. Will merge once Travis is green.

Thanks @TomAugspurger !

@TomAugspurger Thanks, Tom. 

I'm trying to tie back to my commit - #9574 . I don't have that .pop() in _compute_plot_data() within my commit. Was that added somewhere else? In fact I deleted a .pop() line in my commit.

@schmohlio This specific line that @TomAugspurger changed in this PR was in any case not introduced by your PR (https://github.com/pydata/pandas/blame/95dc0506f1a9237b0a34a1d26eba1e1a8d357758/pandas/tools/plotting.py#L1001), so certainly not necessary a regression by your PR! (and anyway, who introduced it does not really matter, as it is reviewed by multiple people and we should improve the test suite to catch such things) 
But it was a case we could have thought of to test as well .. (easy to say afterwards of course), we only tested dataframes and not series I think.

@jorisvandenbossche both thoughts makes a lot of sense. Just wanted to see if I could help at any point. Thanks again for you and your team's great work.
BUG: Filter/transform fail in some cases when multi-grouping with a datetime-like key
State: closed

Fixes GH #10114, and a related problem in transform which was already present in 0.16.0


Labels:
name: Bug
name: Groupby
name: Dtype Conversions
description: Unexpected or buggy dtype conversions

Comments:
can you rebase

can you rebase and ping, will have a look

Rebased

thanks! keep-em coming!

Thanks!
parsing latitude and longitude in wb.get_countries
State: closed

World Bank provides latitude and longitude as strings (and an empty string as a missing value). It's inconvenient so, I parse it to floats. Additionally, I added some description to the `wb.get_countries` function.


Labels:

Comments:
Now I see that this module is getting deprecated. I am moving to https://github.com/pydata/pandas-datareader/.

Indeed, thanks!
BUG: drop_duplicates drops name(s).
State: closed

Closes #10115.


Labels:
name: Bug
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: MultiIndex

Comments:
@seth-p can you rebase and update?

Afraid I do not have time to work on this. Could someone else handle it?

@seth-p OK, will do.
ENH: support .strftime for datetimelikes (closes #10086)
State: closed

closes #10086 


Labels:
name: Enhancement
name: Datetime
description: Datetime data dtype
name: Output-Formatting
description: __repr__ of pandas objects, to_string
name: Strings
description: String extension data type and string data

Comments:
this works for `DatetimeIndex`, something broken for `PeriodIndex`. And should not work for `TimedeltaIndex` (as I don't think the `strftime` codes are meaningful). Further want to actually define `.strftime` on the index itself as well (maybe `def strftime(self, format):`).

needs tests for the same

that makes sense

I've added to both `DatetimeIndex` and `PeriodIndex`. Also added a few tests. 

pls add strftime to API.rst 
also I think a mini section in time series.rst would be ok too

@jreback it's updated 

@jreback I added a mini section to `basics.rst` instead, but otherwise this is ready for review 

can you update for comments. also related to #10442. I want _this_ method to be the 'primary' date-string formatting (though `.astype(str)` will be supported by calling what this calls, namely `.format`), but this should be the goto method

copy your example that you added to basics.rst and put in 0.17.0 (in a separate section, maybe datetime-string formatting or somesuch)

@jreback updated according to your comments except for one thing: I couldn't actually find an invalid formatting string that would cause `strftime` to raise an exception 

We also have a `tslib.format_array_from_datetime` function. Can you check if this is faster? (also uses `strftime`, but the loop is cythonized)

@jorisvandenbossche sure I'll check  

@jorisvandenbossche this calls `.format(date_format=...)` which ultimately calls the cython `tslib.format_array_from_datetime`, so this is just syntatic sugar.

@jreback reordered the `lambdas` and the `if/else` as you suggested, also added more unit tests

ok, lgtm. ping when green.

sure will do 

@jorisvandenbossche @jreback updated to return `ndarray` instead of `list` 

@jorisvandenbossche @jreback it's green now, please take a look 

@mortada couple of comments. ping when green.

@jreback updated, please take a look 

thanks!
BUG: categorical doesn't handle display.width of None in Python 3 (GH10087)
State: closed

closes #10087

I think this is fairly straightforward, however I am not sure if the prior code comparing `display.width` to 0 was correct... is `display.width` ever supposed to be set to 0? It looks suspiciously similar to a snippet from `Series.__unicode__()`, but that is looking at `display.max_rows`. Maybe it was brought over incorrectly?

If it should just be a simple comparison to None, not 0 as it was before, just let me know and I'll amend this.


Labels:
name: Bug
name: Output-Formatting
description: __repr__ of pandas objects, to_string

Comments:
I think my git skills have gone rusty. I am trying to amend my commit, but it doesn't like that I have already pushed it - it won't accept my new (amended) push, saying I am behind the remote counterpart. But the remote just has the changes before the amended commit. Any hints?

I just forced the push. Not sure if there was any better way.

At any rate, I incorporated most of your comment. I think 0 is a valid value, as it's checked against in code below. Thanks for your feedback!

@rosnfeld Yes, you need to force if you amend a commit.

Ok, good to know - thanks.

@jreback Travis is happy if you're happy (with this change).

@rosnfeld thanks! hope this exercised those git skills! maybe need another PR to work on that :)