BrandonFors/pandas-github-issues huggingface dataset:
BUG: series.fillna() fails with quoted numbers
State: closed

Here's a simple example:

``` python
import pandas as pd
import numpy as np

x = pd.Series(range(5))
x.iloc[1] = np.nan
x.fillna(0) # works
x.fillna('foo') # works
x.fillna('0') # crash
```

in the function `_putmask_smart()`, two numpy arrays are compared to determine if they have compatible dtypes. However if one array contains quoted numbers, then the test does NOT resolve as element wise compare, but rather as a "normal" python comparison. This means the result is `False`, rather than an array, and the attempt to get `False.all()` obviously fails.

Not sure exactly why the arrays don't compare as expected -- perhaps there are other criteria that would trigger it -- but an easy fix is to add `AttributeError` to the try/except clause as a "valid" error. 


Labels:
name: Bug
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate

Comments:
hmm, that is odd. Pls add a test for this.

@jlowin this needs a test and release note.

closing pls reopen if/when updated
Calling qcut with too many duplicates now gives an informative error
State: closed

Closes #7751. 


Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Error Reporting
description: Incorrect or improved errors from pandas

Comments:
- needs a test
- needs a release note (bug fix section)

pls move the release note to v0.16.0. and rebase. thxs

pls move the release note to 0.16.0 and rebase

closing pls reopen if/when updated
Implement timedeltas for to_json
State: closed

Closes #9027


Labels:
name: Bug
name: IO Data
description: IO issues that don't fit into a more specific label
name: IO JSON
description: read_json, to_json, json_normalize

Comments:
@jreback refactored to have a single `total_seconds` function in `datetime_helper.h`

cool rebase and merge in green 

looks good. ready to merge?
BUG: Datareader index name shows unicode characters
State: closed

Fixes #8967 


Labels:
name: Bug
name: Unicode
description: Unicode strings

Comments:
is this just google weirdness?

ping on green

failing on py3

@jreback  Passes on py3 now.  Can rebase if you like.  Solution is a big hackist - happy to take some suggestions on dealing with unicode between py3 and py2.

@dstephens99 this looks ok, pls rebase.

alos note that we are moving the dataread stuff to a separate package, pandas-datareade, i'll send you an invite to the repo

Can this get ported to datareader? Or should I submit a pull request there as well?

@dstephens99 this needs to be merged to pandas? or no

No, it got merged in with #9358 

@dstephens99 gr8 thankns!
Gh9010 yahoo options parsing bug
State: closed

closes #9010. Fix for Yahoo Options parse error for underlying prices of 1,000.00 or larger.


Labels:
name: Bug

Comments:
even better is to construct the frame and then try to coerce the columns with a try except around astype If it fails then do the replace and try again - will be much faster

So you would prefer a solution where I leave Options's underlying_price member as a string, then in the _process_data method do an astype conversion and filter out the comma if an exception is thrown?

yep will be more efficient 

Correcting the column type in _process_data won't work -- it breaks get_near_stock_price(), because its call into chop_data() relies on underlying_price already being a float, and doesn't use _process_data. It also breaks the test_sample_page_price_quote_time1 and test_sample_page_price_quote_time2 tests, for the same reasons.

Doing the type checking / possible conversion in the _get_underlying_price() method will make the code cleaner and more cohesive, versus sprinkling around the checking and conversion login in multiple methods. I understand your performance concerns, especially about the locale methods ... but wouldn't simply filtering out the commas be nearly as quick?

Let me know your thoughts.

I understand your point about simplicity, but string.replace() is deprecated in 2.x python and doesn't exist in python 3.x, and the pandas docs (http://pandas.pydata.org/pandas-docs/stable/install.html) claim to support both branches of python. String.join() works in both. 

Thanks for the report, sturmd.

Is there any more housekeeping I need to do here to make sure this is ready for 0.16.0?

@PollyP  `replace` was removed from the `string` helper module but still exists on the string type

i.e. 

``` python
In [1]: s = 'fof'

In [2]: s.replace('o', 'f')
Out[2]: 'fff'
```

will work fine in Python 3 but

``` python
In [3]: import string

In [4]: s = 'fof'

In [5]: string.replace(s, 'o', 'f')
```

won't. It's just some cruft they cleared up, the `replace` method has not been removed or deprecated on the string type.

@PollyP 

pls address the above, rebase and squash. ping when green.

OK, will do this this weekend.

On Fri, Jan 2, 2015 at 9:23 AM, jreback notifications@github.com wrote:

> @PollyP https://github.com/PollyP
> 
> pls address the above, rebase and squash. ping when green.
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/pydata/pandas/pull/9024#issuecomment-68543990.

I wound up syncing and reapplying my changes to create a new PR #9198. I believe it is ready to go.

replaced by #9198
Make Timestamp('now') equivalent to Timestamp.now()
State: closed

closes #9000 
I opted to not change np_datetime_strings but instead short-circuit in Timestamp.  I don't have a strong preference one way or another.


Labels:
name: Bug
name: Datetime
description: Datetime data dtype
name: Timezones
description: Timezone data dtype

Comments:
this looks good

@shoyer @jorisvandenbossche any other comments?

Looks good to me

@rockg thanks!
reindex multi-index at level with reordered labels
State: closed

closes https://github.com/pydata/pandas/issues/4088

on branch:

```
>>> df
                     vals
first second third       
mid   3rd    992     1.96
             562    12.06
      1st    73     -6.46
             818   -15.75
             658     5.90
btm   2nd    915     9.75
             474    -1.47
             905    -6.03
      1st    717     8.01
             909   -21.12
      3rd    616    11.91
             675     1.06
             579    -4.01
top   1st    241     1.79
             363     1.71
      3rd    677    13.38
             238   -16.77
             407    17.19
      2nd    728   -21.55
             36      8.09
>>> df.reindex(['top', 'mid', 'btm'], level='first')
                     vals
first second third       
top   1st    241     1.79
             363     1.71
      3rd    677    13.38
             238   -16.77
             407    17.19
      2nd    728   -21.55
             36      8.09
mid   3rd    992     1.96
             562    12.06
      1st    73     -6.46
             818   -15.75
             658     5.90
btm   2nd    915     9.75
             474    -1.47
             905    -6.03
      1st    717     8.01
             909   -21.12
      3rd    616    11.91
             675     1.06
             579    -4.01
>>> df.reindex(['1st', '2nd', '3rd'], level='second')
                     vals
first second third       
mid   1st    73     -6.46
             818   -15.75
             658     5.90
      3rd    992     1.96
             562    12.06
btm   1st    717     8.01
             909   -21.12
      2nd    915     9.75
             474    -1.47
             905    -6.03
      3rd    616    11.91
             675     1.06
             579    -4.01
top   1st    241     1.79
             363     1.71
      2nd    728   -21.55
             36      8.09
      3rd    677    13.38
             238   -16.77
             407    17.19
>>> df.reindex(['top', 'btm'], level='first').reindex(['1st', '2nd'], level='second')
                     vals
first second third       
top   1st    241     1.79
             363     1.71
      2nd    728   -21.55
             36      8.09
btm   1st    717     8.01
             909   -21.12
      2nd    915     9.75
             474    -1.47
             905    -6.03
```


Labels:
name: Bug
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: MultiIndex

Comments:
looks good. just need a release note (bug fixes).

added the release notes; all green

@behzadnouri cool thanks!

was a bit more complicated that first glance....!
BUG: Prevent index header column from being added on MultiIndex when ind...
State: closed

Fix for #8452 

Index column was added when index=False.


Labels:
name: Output-Formatting
description: __repr__ of pandas objects, to_string
name: IO HTML
description: read_html, to_html, Styler.apply, Styler.applymap

Comments:
@cpcloud quick look?

@shoyer @jorisvandenbossche looks ok to me

@papaloizouc thanks!
TST: better testing for test_where (GH8997)
State: closed

closes #8996 


Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
FWIW, as I have mentioned, not sure how this was supposed to help and here it is (not on current master but on v0.15.1-176-g526f33c as before

```
(Pdb) tm.assert_numpy_array_equal(result, expected)
*** AssertionError: [(0L, 'a') (1L, 'b') (2L, 'c')] is not equal to [(0L, 'a') (1L, 'b') (2L, 'c')]
```

Hello,

I can confirm this od Debian build.
I have used a patch from #8996 and I did get the same result as yarikoptic

<pre>
======================================================================
FAIL: test_to_records (pandas.tests.test_categorical.TestCategoricalAsBlock)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/build/pandas-HjPP3c/TESTIRAJ_PANDU/2/pandas-0.15.1+git125-ge463818/debian/tmp/usr/lib/python2.7/dist-packages/pandas/tests/test_categorical.py", line 2393, in test_to_records
    tm.assert_numpy_array_equal(result,expected)
  File "/build/pandas-HjPP3c/TESTIRAJ_PANDU/2/pandas-0.15.1+git125-ge463818/debian/tmp/usr/lib/python2.7/dist-packages/pandas/util/testing.py", line 632, in assert_numpy_array_equal
    '{0} is not equal to {1}.'.format(np_array, assert_equal))
AssertionError: [(0L, 'a') (1L, 'b') (2L, 'c')] is not equal to [(0L, 'a') (1L, 'b') (2L, 'c')].
</pre>

Regards,
Jurica
ENH: Implement Series.StringMethod.slice_replace
State: closed

closes #8888 

This does not implement immerrr's suggestion of implementing `__setitem__`, as all `StringMethod` methods return a new `Series` (or `DataFrame`) rather than mutating. 


Labels:
name: Enhancement
name: Strings
description: String extension data type and string data

Comments:
This looks pretty good to me. We are about to release 0.15.1, so after we do that please add release notes for this new feature.

Sure, I'll add a release note and rebase after the release drops.

@billletson 0.15.2 has been released, so you can go ahead and add release notes now for 0.16.0.

@billletson sorry, I was not paying attention earlier. Could you please rebase again and ping me when the tests pass so I can do the merge? Github does not send out notifications of new commits without a comment :/.

@shoyer rebased and build passing, good to go

Thanks!

@billletson this is nice.

maybe stick an example in here somewhere: http://pandas-docs.github.io/pandas-docs-travis/text.html ?
Fix groupby().transform() example in cookbook.rst
State: closed

@jorisvandenbossche : Fixes code in example, and makes final averaging by len(x) stylistically similar to previous adds.
New pull request replacing my earlier one, which contained some other erroneous commits.
closes #8944 


Labels:
name: Docs

Comments:
@SteveSimmons thanks!

fyi, you don't need to open a new PR in general, just edit and force push the old one to replace.

Thanks Jeff. I have some time this weekend and will try to do another couple of 0.15.2 issues. Can you let me know if I should focus on particular ones? Don't want to duplicate effort of others

On 5 December 2014 02:11:25 GMT+00:00, jreback notifications@github.com wrote:

> Merged #9007.
> 
> ---
> 
> Reply to this email directly or view it on GitHub:
> https://github.com/pydata/pandas/pull/9007#event-203047697

## 

Sent from my Android device with K-9 Mail. Please excuse my brevity.
FIX: use decorator to append read_frame/frame_query docstring (GH8315)
State: closed

Closes #8315


Labels:
name: IO SQL
description: to_sql, read_sql, read_sql_query

Comments:
is #8315 the right issue?

@jreback yes, you even suggested the fix (https://github.com/pydata/pandas/issues/8315#issuecomment-63912121) :-)

duh! thxs
User facing AssertionError in empty groupby GH5289
State: closed

closes https://github.com/pydata/pandas/issues/5289 
Is this all we need to do?
Or am I missing something?


Labels:
name: Groupby
name: Error Reporting
description: Incorrect or improved errors from pandas

Comments:
would be nice to have a test that hits this

also need to remove the .deb file that seemed to be added to the repo

Ok doing that - might need  abit of help with the tests - since I'm not an expert on these yet.

@springcoil can you revisit

Ok I'll have a look and see what is going wrong. 

closing pls reopen if/when updated
BUG: Dynamically created table names allow SQL injection
State: closed

Working with the SQL code, I realized that the legacy code does not properly validate / escape passed in table and column names.

E.g.:

```
n [116]:

import pandas as pd
import pandas.io.sql as sql
import sqlite3

df = pd.DataFrame({'a':[1,2],'b':[2,3]})
con = sqlite3.connect(':memory:')
db = sql.SQLiteDatabase(con, 'sqlite')

t=sql.SQLiteTable('a; DELETE FROM contacts; INSERT INTO b', db, frame=df)

print t.insert_statement()
```

results in:

```
INSERT INTO a; DELETE FROM contacts; INSERT INTO b ([index],[a],[b]) VALUES (?,?,?)
```

This fixes the issues.


Labels:
name: IO SQL
description: to_sql, read_sql, read_sql_query

Comments:
@jorisvandenbossche Is there anything else that should be done here?

Also, @jorisvandenbossche --- this is now failing tests because of the rather brittle `_get_sqlite_column_type` inserted in #9109 . Is there a reason you are parsing the sql code directly instead of inserting the table and then reading out the column type from PRAGMA (as done by existing `_get_sqlite_column_type`)?

@artemyk Hmm, I was first using the existing `_get_sqlite_column_type`, but then wrote that new function because I wanted to mimic the same test in the `TestSQLApi` (so just testing the type mapping that happens in `SQLTable` without the roundtrip to the actual database, so testing `_sqlalchemy_type`/`_sql_type_name` for sqlalchemy/fallback mode). But I agree it is rather brittle and not very good practice. 
You can easily update it so the tests pass (I assume changing the `'[]'` to `'""'` would be enough), but I can also change the test.

@jorisvandenbossche I changed the test to `'""'` so that it passes now and this can be merged.  Personally I think a roundtrip to sqlite is better than a brittle test, so I would vote for changing it (I can do it in this PR if you'd like)

@jorisvandenbossche Can we merge?

@jorisvandenbossche  Now catching UnicodeError...

@artemyk Thanks!
BUG in read_csv skipping rows after a row with trailing spaces, #8983
State: closed

Update tokenizer.c to fix a BUG in read_csv skipping rows after tokenizing a row with trailing spaces, Closes #8983


Labels:
name: Bug
name: IO CSV
description: read_csv, to_csv

Comments:
pls add the issue number into the same release note that dealt with this

otherwise looks good

ping when ready

added the issue number to whatsnew/v0.15.2.txt and to test_parsers.py

@selasley awesome! thanks for the fixes. not many want to delve into the c-parser!

if you are interested:

#8985, #6710

This may not be the appropriate place for this comment and you probably hear it often, but thank you for pandas.  I converted some python scripts we wrote to plot our near-real time SOHO MTOF data on the web from numpy+matplotlib to pandas.  The scripts are much easier to follow now and I was able to add a few features that would have been cumbersome to do with straight numpy.  I'll see if I can do anything with 8985 and 6710.  I believe there are still some bugs in read_csv with certain combinations of options.  I'll work on fixing any that I can find as time allows.

@selasley always appropriate for praise :)

glad to see you are using it!

and feel free to chime in on any issues...thanks again
FIX: Workaround for integer hashing
State: closed

Force conversion to integer for missing values when
they must be integer to avoid hash errors on 32 bit
platforms.

closes #8968


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: IO Stata
description: read_stata, to_stata

Comments:
@yarikoptic This might fix it by forcing int.  I'm not completely sure if this isn't a NumPy bug (or at least idiosyncracy) on 32 bit platforms with hashing or casting this value to native Python types.

iirc this is a numpyism

that dtype hashes are not the same

@yarikoptic can you try this out?

well i'll merge and see what happens

@bashtage thanks

There is some cleaning to do in the stata code, so I'll think about how to make this more robust in the future - perhaps making all integer values always longs when creating the dictionary, and then doing the same when doing a look up might be the more stable across bitnesses.

ok, if this fixes @yarikoptic then ok to put in 

Have made a few minor changes to ensure that the hashes in the missing value dictionary should always have the correct type for any bitness.
BUG: Bug in Timestamp-Timestamp not returning a Timedelta type (GH8865)
State: closed

closes #8865 


Labels:
name: Bug
name: Datetime
description: Datetime data dtype
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: Timedelta
description: Timedelta data type

Comments:
cc @shoyer

was a rabbit-hole!

not 100% convinced that the auto-broadcasting strategy will work here, but go for it.

generally looks pretty good to me!

@shoyer great!

still would love to have you refactor when you have to time to use more array broadcasting stuff.
BUG: StataWriter uses incorrect string length
State: closed

Fixes bug where StataWriter always writes strings with
a size of 244.

closes #8969


Labels:
name: Bug
name: Output-Formatting
description: __repr__ of pandas objects, to_string
name: IO Stata
description: read_stata, to_stata

Comments:
@jreback I have had to use a workaround since `max_len_string_array` does not work with unicode - it alway returns 0.  Any ideas on getting the maximum length for a unicode array (when `infer_dtype` returns 'unicode')?

https://github.com/pydata/pandas/pull/8978

should fix this (give it a try and lmk)

just wasn't implemented.

Once #8978 gets in, then I'll rebase and remove the workaround.

does that fix it for you? it's ready to go

Yes, all fixed.  Once I can rebase on it this should be ready.

did you need any addtl tests for this? looks back-compat?

I adapted an existing test to check for this in addition to a previous bug

On Dec 3, 2014 8:32 AM, jreback notifications@github.com wrote:

did you need any addtl tests for this? looks back-compat?

—
Reply to this email directly or view it on GitHubhttps://github.com/pydata/pandas/pull/8977#issuecomment-65406670.

thanks!
Implement Categorical.searchsorted(v, side, sorter) #8420
State: closed

closes #8420

Continuation of #8928


Labels:
name: Enhancement
name: Categorical
description: Categorical Data Type

Comments:
looks good. just need a release note, enhancements section. ping when you have pushed. thanks.

can you add a release note?

Added in whatsnew for 0.15.2

merged via https://github.com/pydata/pandas/commit/5928075f58715207da9c21f03c9da9ea10d4082b

thanks!
PERF: astype(str) on object dtypes GH8732
State: closed

Closes #8732

In most cases it looks like, we need to iterate over array
and coerce each element. This is so that the appropriate
exception can be raised, or we can deal with nulls.
So the original case of casting ints to strings, has to
work the way it does, unless we change the underlying behaviour.
So when astype(str) is called on ints. Then each element
is first cast as a string then made into a numpy object.
If we relied on numpy it wouldn't cast it to string, just
return it as an object. This breaks existing behaviour.

It is possible to bypass iterating over the array, when we are
coercing to int. Assuming that there are no NaNs and the type
of the array is a numeric.


Labels:
name: Performance
description: Memory or execution speed performance
name: Strings
description: String extension data type and string data

Comments:
not sure what this is actually fixing. All of the tests you show work AFAICT in master.

@jreback #8732 is not a bug, but a performance issue

@vikram can you show the output of the benchmark?

@vikram can you show the benchmark results?

@vikram can you show the vbench results for this?

closing pls reopen if/when updated
docfix: add boolean array as option for indexing with .iloc
State: closed

resolves #8956 Trivial documentation addition of boolean array as option for .iloc


Labels:
name: Docs
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves

Comments:
@cswarth can you update to the comments?

not sure what you need for in an update - I was just getting back to this when you mail came in.  I'll modify other doc sections on `.iloc` as requested.    Do you think it worth mentioning in close proximity that boolean array indexers should span the full axis?   Personally prefer to document what is, not what might be.

@cswarth their are 2 mentions of the iloc description (one at the very top and one where you modified). Need them both to be the same. Nothing additional is necessary w.r.t. what you wrote.

merged via https://github.com/pydata/pandas/commit/7be0236fa56429ccdd98b26dd3ab115b31025cb2

thanks!
BUG: ValueError raised by cummin/cummax when datetime64 Series contains NaT.
State: closed

closes #8965


Labels:
name: Bug
name: Datetime
description: Datetime data dtype
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: Timedelta
description: Timedelta data type

Comments:
@jreback: Sorry, I don't understand what part of nanops I should be using, or what makes cummax more complicated. Could you give me some more guidance?

`cummin` with `skipna=False` will fail in your current impl because the `NaT` will always be the min (as its the smallest value)

thanks!