BrandonFors/pandas-github-issues huggingface dataset:
API: multi-line, not inplace eval
State: closed

closes #9297
closes #8664

This is just a proof of concept at this point - the idea is to allow multiple assignments within an eval block, and also include an `inplace` argument (currently default True to match old behavior, although maybe that should be changed) for chaining.

```
In [15]: df = pd.DataFrame({'a': np.linspace(0, 100), 'b': range(50)})

In [16]: df.eval("""
    ...: c = a * 2
    ...: d = c / 2.0
    ...: e = a + b + c""")

In [17]: df.head()
Out[17]: 
        a  b          c         d          e
0  0.000000  0   0.000000  0.000000   0.000000
1  2.040816  1   4.081633  2.040816   7.122449
2  4.081633  2   8.163265  4.081633  14.244898
3  6.122449  3  12.244898  6.122449  21.367347
4  8.163265  4  16.326531  8.163265  28.489796

In [18]: df.eval("""
    ...: f = e * 2
    ...: g = f * 1.5""", inplace=False).head()
Out[18]: 
        a  b          c         d          e          f          g
0  0.000000  0   0.000000  0.000000   0.000000   0.000000   0.000000
1  2.040816  1   4.081633  2.040816   7.122449  14.244898  21.367347
2  4.081633  2   8.163265  4.081633  14.244898  28.489796  42.734694
3  6.122449  3  12.244898  6.122449  21.367347  42.734694  64.102041
4  8.163265  4  16.326531  8.163265  28.489796  56.979592  85.469388
```


Labels:
name: API Design

Comments:
@chris-b1 this looks reasonable. I think that we _should_ change the `inplace` to `False` by default. That obviously is a breaking change. So what I think we could do is this:

make `inplace=None` the default
detect if an assignment is being made. if `inplace` is `None`, then show a deprecation warning.
this implicity makes `inplace=None` -> `inplace=False` for non-assignments (which is what it currently does, and is quite reasonable). To get inplace then assignment you would have to be explict about passing `inplace=True`, which is what we would want.

yes?

cc @shoyer 

Yes, this looks great!

I agree that we should change the default behavior to `inplace=True` in a version or two. I think @jreback has the right idea on the migration path.

@chris-b1 going to mark this for 0.18.0 as putting the deprecation warning in-place (pun intended)

@chris-b1 want to update / add in deprecation warning as discussed

@jreback - this is updated - added a whatsnew and deprecation warnings.

pls review the eval docs for any updates (in enhancingperf.rst)

does this handle #8664  ?

@jreback 
Updated the enhancingperf docs, and added a fix/test for #8664

lgtm. couple of doc changes. squash & ping when green.

@jreback - updated and green

very tiny doc change. ping when pushed.

can you run: `git diff master | flake8 --diff` and fix anything. going to shortly be enforcing PEP8.

Updated for your doc comment and PEP8 - nothing major but did have to add a couple `# noqa` where a name was only used in an eval block - is that the right approach?

merged via https://github.com/pydata/pandas/commit/99c2d932c8c3e86e2311f525b412e2e335dbe6a2

thanks for the nice changes!

yeh the `# noqa` is fine (I took one out, but otherwise ok).
BUG: astype(str) on datetimelike index #10442
State: closed

closes #10442 

fix for the bug: Convert datetimelike index to strings with astype(str)
going to send pull request for test 


Labels:
name: Datetime
description: Datetime data dtype
name: Output-Formatting
description: __repr__ of pandas objects, to_string
name: Dtype Conversions
description: Unexpected or buggy dtype conversions

Comments:
I added all the tests..please let me know if I need to add anything else.

sounds fine now, let me know if I need to add anythingelse

pls add a note in whatsnew for 0.17.1 (enhancements). squash as well.

going to need tests for `Series.astype(str)` as well (put near tests in test_series or tseries/tests/test_timeseries.py wherever the existing tests for .astype are), add the issue as a comment in the tests.

@jreback 
there is no v0.17.1.txt for 0.17.1 in whatsnew folder.

rebase on master
i put it their 2 days ago

@hamedhsn can you update

@jreback sorry, what do I need to update?

comments above, needs a rebase as well

@jreback did that.
Thanks

couple of comments. pls squash as well

@jreback sorry that I am asking..how can I squash the last few commits to one. I tried a few different approaches without a luck.

http://pandas.pydata.org/pandas-docs/stable/contributing.html#contributing-your-changes-to-pandas

@jreback squashed all the commits into one.

you need to make sure you force push too.

```
git checkout thisbranch
git rebase -i master
```

squash (put a s by each commit)

```
git push yourremote thisbranch -f
```

@jreback did that. thanks

@jreback should be fine now.

ok @hamedhsn ping on green.

@jreback strange, not sure why test is failed. both sides are object type but it raises this error..
AssertionError: [index] Expected object to be of type class 'pandas.core.index.Index', 
found class 'pandas.core.series.Series' instead 

I tried the test with both side same nothing to do with astype but still get the same error
result = pd.Series(pd.date_range('2012-01-01', periods=3))
expected = result
tm.assert_index_equal(result, expected)

@jreback it is fine now.

thanks!
PERF: improves performance in SeriesGroupBy.transform
State: closed

```
-------------------------------------------------------------------------------
Test name                                    | head[ms] | base[ms] |  ratio   |
-------------------------------------------------------------------------------
groupby_transform_series2                    |   6.3464 | 167.3844 |   0.0379 |
groupby_transform_multi_key3                 |  99.4797 | 1078.2626 |   0.0923 |
groupby_transform_multi_key1                 |  10.9547 |  98.5500 |   0.1112 |
groupby_transform_multi_key2                 |   9.0899 |  67.2680 |   0.1351 |
groupby_transform_series                     |   5.4360 |  30.5537 |   0.1779 |
groupby_transform_multi_key4                 |  39.9816 | 187.1406 |   0.2136 |
-------------------------------------------------------------------------------
Test name                                    | head[ms] | base[ms] |  ratio   |
-------------------------------------------------------------------------------

Ratio < 1.0 means the target commit is faster then the baseline.
Seed used: 1234

Target [0470a10] : PERF: improves performance in SeriesGroupBy.transform
Base   [7b9fe14] : revers import .pandas_vb_common
```


Labels:
name: Groupby
name: Performance
description: Memory or execution speed performance

Comments:
thanks!
PERF: infer_datetime_format without padding #11142
State: closed

Closes #11142


Labels:
name: Datetime
description: Datetime data dtype
name: Performance
description: Memory or execution speed performance

Comments:
@sinhrks pushed changes for your notes.  The perf impact on `_guess_datetime_format` is negligible because it is only ever called once.  Nothing showed up in `asv`, here are a couple timings.

```
In [1]: s, l = pd.date_range('2014-1-1', periods=10), pd.date_range('2014-1-1', periods=100000)
In [2]: s = s.strftime('%Y-%m-%d')
In [3]: l = l.strftime('%Y-%m-%d')

# master
In [4]: %timeit pd.to_datetime(l, infer_datetime_format=True)
10 loops, best of 3: 30.3 ms per loop

In [5]: %timeit pd.to_datetime(s, infer_datetime_format=True)
1000 loops, best of 3: 300 µs per loop

# PR
In [5]: %timeit pd.to_datetime(l, infer_datetime_format=True)
10 loops, best of 3: 30.3 ms per loop

In [6]: %timeit pd.to_datetime(s, infer_datetime_format=True)
1000 loops, best of 3: 308 µs per loop
```

thanks!
COMPAT: Support for MPL 1.5
State: closed

closes https://github.com/pydata/pandas/issues/11111  (nice issue number)

Testing with tox locally I get 1 failure using either matplotlib 1.4 or 1.5. I may have messed up my tox config though. Still need to cleanup a bunch of stuff, I just threw in changes to get each test passing as I went.

release notes et. al coming tonight or tomorrow. Are we making any changes to the build matrix to run a matplotlib 1.5?


Labels:
name: Visualization
description: plotting
name: Compat
description: pandas objects compatability with Numpy or Python functions

Comments:
u can add to the channel list (I think I showed this above)

it will thrn pick up the latest version if it's not specified exactly

is this not tested on travis? I don't see the icon

Travis was flaky yesterday morning. I just pushed some changes, and it's picked up now.

@jreback travis is green. Looks like https://travis-ci.org/pydata/pandas/jobs/81429896 and https://travis-ci.org/pydata/pandas/jobs/81429897 both ran with matplotlib RC1.

make the 3.4 regular build not specify a matplotlib (in requirements-3.4.run). All of the slow tests are running (e.g. 2.7) just fine. But I don't think the fast ones actually are run by travis (in theory they are on 3.3, but that is not build ATM for 1.5.0rc1).

@jreback done. We'll want to followup and pin it at 1.5 once that's released. I'm assuming the `conda-forge` channel will continue to track master and we won't want changes there breaking our builds. 

thanks @TomAugspurger 

@TomAugspurger I don't think we plan to update conda forge from master, just tags.
BUG: remove midrule in latex output with header=False
State: closed

This bug fix addresses issue #7124. If `to_latex` is called with option `header=False`, the output should not contain a `\midrule` which is used to separate the column headers from the data.


Labels:
name: Output-Formatting
description: __repr__ of pandas objects, to_string
name: IO LaTeX
description: to_latex

Comments:
@TomAugspurger ?

pls add a whatsnew note for 0.17.1

Yeah this looks good (looked at this code in a while though). @loicseguin if you could rebase and add a release note, thanks.

Done. Let me know if there's anything else I should add.

merged via https://github.com/pydata/pandas/commit/924c419070bc6d7c4edc6779b501377a22bce58d

thanks!
DEPR: Series.is_timeseries
State: closed

Follow-up for #10890. Deprecate `Series.is_timeseries`.


Labels:
name: Datetime
description: Datetime data dtype
name: Deprecate
description: Functionality to remove in pandas

Comments:
lgtm

just check that we are not actually using this anywhere (tests or otherwise)

Based on the search result, it only exists in `series.py` and `test_series.py`. Both fixed.

gr8 merge when ready
BUG: nested construction with timedelta #11129
State: closed

Closes #11129

Per discussion in issue this adds similar hash equality to `_Timedelta` / `datetime.timedelta`
that exists for `_Timestamp` / `datetime.datime`.

I tried inlining `_Timedelta._ensure_components` and it actually hurt performance a bit, so I left it
as a plain `def` but did build a seperate `_has_ns` check for hashing.

Adds a little overhead to hashing:

```
In [1]: tds = pd.timedelta_range('1 day', periods=100000)

# PR
In [2]: %timeit [hash(td) for td in tds]
1 loops, best of 3: 810 ms per loop

# Master
In [2]: %timeit [hash(td) for td in tds]
1 loops, best of 3: 765 ms per loop
```


Labels:
name: Bug
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: Timedelta
description: Timedelta data type

Comments:
try making it a `cpdef`

hmm, no reason ever to do `hash` like that in a loop though..

changing to a `cpdef` didn't seem to matter, assuming I'm profiling the right thing (I think component access would take a call to `_ensure_components()`)?

```
In [2]: %timeit tds.microseconds
#cpdef
1 loops, best of 3: 776 ms per loop
#def
1 loops, best of 3: 775 ms per loop
```

lgtm. pls change `_has_ns` to be a method of `_Timedelta`

@jreback made that change, Travis green.

thank u sir!
BUG: Fix computation of invalid NaN indexes for interpolate.
State: closed

Fixes issue #11115.

This change fixes up a couple edge cases for the computation of invalid NaN indexes for interpolation limits.

I've also added a test explicitly for the reported bug (and similar behaviors at the opposite end of the series) in #11115.

/cc @jreback 


Labels:
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate

Comments:
in the whatsnew add this issue on (where the `limit_direction` change exists, just add this issue number on)

Done!

gr8. don't worry about eh failing builds on travis. fixing shortly

thanks!
DOC: Fixed outdated doc-string, added missing default values, added missing optional parameters for some io classes
State: closed

Some doc-strings are outdated, in a sense that there are missing newly added optional parameters and default values. 


Labels:
name: Docs

Comments:
@terrytangyuan The changes look good, but, the `data.py` is now maintained at https://github.com/pydata/pandas-datareader. So maybe you can do a PR for that file there

@jorisvandenbossche Nice catch! Thanks for checking. I've made the changes. I'll submit another PR to make changes on the new file. 

@jorisvandenbossche On green now. Can you merge? 

Thanks @kawochen Fixed and on green now. 

I guess this is fine. @terrytangyuan prob strip off the stuff for `pandas-datareader` and submit there as well.
ERR/API: Raise NotImplementedError when method is not implemented (issue: 7692)
State: closed

Fixed #7692 


Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Error Reporting
description: Incorrect or improved errors from pandas

Comments:
@jreback Does this need a whatsnew note or test? Would the test to check whether this raises `NotImplementedError`? 

tests

@jreback  thanks could you check the update again?

Thanks. Done @jreback 

On green. Could you merge? @jreback 

@terrytangyuan just a single ping is enough.

@jreback I apologize for the repetitive ping and thanks for the feedback. I've made the changes. 

pls add a whatsnew note (Bug fixes) for 0.17.0

@jreback  Done. On green now. Thanks.

Thanks for the review @jreback . Fixed and on green now.

@kawochen @jreback  Thanks. Revised. On green now. 

merged via https://github.com/pydata/pandas/commit/597aa42e2b0eeffd60d68b923ac88eee347c6658

thanks!
TST: More thorough test to check correctness of to_html_unicode
State: closed

The current test for `test_to_html_unicode` only checks for whether the function works but does not check for its correctness.  


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Unicode
description: Unicode strings
name: IO HTML
description: read_html, to_html, Styler.apply, Styler.applymap

Comments:
On green. Can this be merged?

looks good. will merge soon.

thanks!
TST: Fix skipped unit tests in test_ga. Install python-gflags using p…
State: closed

closes #11090 

This commit should resolve the following error in the travis build log.

```
-------------------------------------------------------------
#15 nose.failure.Failure.runTest: need httplib2 and auth libs
-------------------------------------------------------------
```

Valid Google credentials are required to run the ga unit tests. 


Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
Here is the output from my local test session:

```
test_getdata (pandas.io.tests.test_ga.TestGoogle) ... ok
test_iterator (pandas.io.tests.test_ga.TestGoogle) ... ok
test_remove_token_store (pandas.io.tests.test_ga.TestGoogle) ... ok
test_segment (pandas.io.tests.test_ga.TestGoogle) ... ok
test_v2_advanced_segment_format (pandas.io.tests.test_ga.TestGoogle) ... ok
test_v2_dynamic_segment_format (pandas.io.tests.test_ga.TestGoogle) ... ok
test_v3_advanced_segment_common_format (pandas.io.tests.test_ga.TestGoogle) ... ok
test_v3_advanced_segment_weird_format (pandas.io.tests.test_ga.TestGoogle) ... ok
test_v3_advanced_segment_with_underscore_format (pandas.io.tests.test_ga.TestGoogle) ... ok

----------------------------------------------------------------------
Ran 9 tests in 3.370s

OK
```

I've updated the tests as per the comments. The import error is gone in my latest commit. See job https://travis-ci.org/parthea/pandas/jobs/80171740

The new reason for skipped tests is `authentication error` as the tests require valid google credentials.

```
#16 pandas.io.tests.test_ga.TestGoogle.test_getdata: authentication error
#17 pandas.io.tests.test_ga.TestGoogle.test_iterator: authentication error
#18 pandas.io.tests.test_ga.TestGoogle.test_segment: authentication error
```

I've confirmed that the tests pass with valid credentials.

thanks!
DOC: Comparison with SAS
State: closed

closes #11075
xref #4052 

There doesn't seem to be a straightforward way to syntax highlight the SAS code, so it will format as generic code blocks.


Labels:
name: Docs

Comments:
lots of lexers here: http://pygments.org/docs/lexers/#lexers-for-the-r-s-languages

isn't it actual S syntax?

No, SAS has its own syntax.

can you render and post this page 

can also add to the highlites section in v0.17.0 (and add to the release.rst the same line).

Here's the rendered output - for some reason in-line code segments don't get rendered correctly locally for me (on all doc pages).
![docs](https://cloud.githubusercontent.com/assets/1924092/9838038/928815ee-5a18-11e5-814e-65e7a133c267.png)

@chris-b1 looks pretty good.

I need a more full look, but @jorisvandenbossche ?

@chris-b1 Really cool!!

just added some minor comments

@jorisvandenbossche - pushed changes for your comments, thanks for taking a look.  Rendered output above is also updated.

@chris-b1 just a couple of minor edits / suggestions.

really nice!

ping when ready for merge.

Thanks!  @jreback pushed changes for your notes, should be good to merge.

awesome. pls have a look once they are built and do a follow-up if needed!
BUG: Fixed bug in groupby.std changing target column when as_index=False
State: closed

Fixed #10355


Labels:

Comments:
this is not going to be oerformant

is there a reason you didn't follow the soln from the issue?

I think the code structure changed somewhat. I couldn't figure out how to
use it.
On Sep 12, 2015 8:54 PM, "Jeff Reback" notifications@github.com wrote:

> this is not going to be oerformant
> 
> is there a reason you didn't follow the soln from the issue?
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/pydata/pandas/pull/11085#issuecomment-139832786.

They are refactored to `_cython_functions` I guess. I tried to do similar thing using `cython` but failed. Maybe this is good for now and I'll figure out how to do it in a separate PERF PR some time later? 

no that will just change it again 

@jreback any updated soln for the soln mentioned in issue? I tried similar way and failed, maybe I missed something.  Thanks.

you can just do the `.var`, then the `np.sqrt` only applies to the action columns.

@jreback  Could you take a look at the new push? Did I get what you meant?

no u can't use the lamba
that goes to a python path

compute var then apply the sqrt

How do I apply it only to action columns without lambda?

Since the result of applying `.var` would be a `DataFrame` and it lost its original groups, which is why the original implementation causes this bug. 

this needs a similar approach to what I suggest (and like what `.first` does)

So I removed the original definition of `std()` and the diff is as follows:

```
@@ -828,6 +828,8 @@ class GroupBy(PandasObject):
                               numeric_only=False, _convert=True)
     last = _groupby_function('last', 'last', _last_compat, numeric_only=False,
                              _convert=True)
+    def std(self):
+        return self._cython_agg_general('std')

     def ohlc(self):
         """
@@ -1485,6 +1487,8 @@ class BaseGrouper(object):
             'f': lambda func, a, b, c, d: func(a, b, c, d, 1)
         },
         'last': 'group_last',
+        'std': {'name' : 'group_var',
+                'f' : lambda func, a: np.sqrt(func(a))}
     }
```

Still couldn't get it working. (tried other similar things too) The code structure of groupby.py must have changed a lot since the issue has been posted. I think this PR at least fixed the bug. I won't be the best person to fix the perf issues. 

then let's close this for now. this is non performant.

https://github.com/pandas-dev/pandas/pull/25315
BUG: Fix Series nunique groupby with object dtype
State: closed

closes #11077


Labels:
name: Bug
name: Blocker
description: Blocking issue or pull request for an upcoming release

Comments:
cc @behzadnouri 

or you want to use do a factorize?

``` diff
diff --git a/pandas/core/groupby.py b/pandas/core/groupby.py
index f34fd6e..39be706 100644
--- a/pandas/core/groupby.py
+++ b/pandas/core/groupby.py
@@ -2565,13 +2565,21 @@ class SeriesGroupBy(GroupBy):
         ids, _, _ = self.grouper.group_info
         val = self.obj.get_values()

-        sorter = np.lexsort((val, ids))
+        try:
+            sorter = np.lexsort((val, ids))
+        except TypeError:
+            val, _ = algos.factorize(val, sort=False)
+            sorter = np.lexsort((val, ids))
+            isnull = lambda a: a == -1
+        else:
+            isnull = com.isnull
+
         ids, val = ids[sorter], val[sorter]

         # group boundries are where group ids change
         # unique observations are where sorted values change
-        idx = com._ensure_int64(np.r_[0, 1 + np.nonzero(ids[1:] != ids[:-1])[0]])
-        inc = com._ensure_int64(np.r_[1, val[1:] != val[:-1]])
+        idx = np.r_[0, 1 + np.nonzero(ids[1:] != ids[:-1])[0]]
+        inc = np.r_[1, val[1:] != val[:-1]]

         # 1st item of each group is a new unique observation
         mask = isnull(val)
```

@behzadnouri If you submit that as a PR to my fork, I'll happily merge it in

the error says `TypeError: merge sort not available for item 0` even though numpy _does_ have merge sort for objects (both `np.argsort` and `np.sort`). so it may be a bug on numpy side. so, it is better to implement try catch in case future numpy releases do implement lexsort for this case

once, values are factorized, original values are not needed and it is more efficient to work with integer factors, as long as `isnull` function is adjusted accordingly. 

@jreback getting a somewhat strange travis failure here that seems unrelated to my change: https://travis-ci.org/cpcloud/pandas/jobs/80036566

any ideas what that might be?

hmm someone else reported that as well
try restarting it and see

this is passing on my travis ci fork

couple comments, squash and merge ok

@cpcloud can you update (e.g. revert those 2 lines). ok to merge on green after that.

ty sir!