BrandonFors/pandas-github-issues huggingface dataset:
FIX/ENH: attempt soft conversion of object series before raising a TypeError when plotting
State: closed


Labels:

Comments:
this leads the user to believe that `object` types are ok; I think you previous solution (of raising) is better as it has a failure that is more obvious

@wesm ?

i like the raise too since if you have plottable series that are not numeric you should convert, but @wesm gave the raise-on-object a strong :-1: in #3911 .

oh I see

ok maybe do the conversion if needed (and warn if u can do the conversion but let it proceed - PerformanceWarning)
and raise like u do now (if u can't convert it)

ok i'll add the perf warning.

@jreback ok 2 merge?

@jreback any objections? going 2 merge if not...

No please do not put in a performance warning! Thank you for fixing this though

sure thing. hope i didn't ruffle too many feathers. trying to avoid :-1:s like the plague! almost finished

@wesm merge-able?

@jreback merge? anyone, beuller? trying to assist in as many 0.11.1 issues as i can...

ok then
BUG/API: remove infer_types from replace and fix compiled regex bug
State: closed

closes #3907.


Labels:

Comments:
@jreback merge-able? i can remove `_is_sequence` in a later pr if u want but that is low priority...

u know what let me add some tests for the things i added to `common.py`

did Travis check this?

it did but i just pushed another commit so rerunning :|

(passed the first time)

ok now it's ready

ok just recall if I changed something really low level  sometimes things break

oh. what did u change that is low-level?

no was a while ago 

ah len is there for `StringMethods`. putting it back...

and i will write a test so that someone doesn't take it out in the future

although a test still passes because of my addition of `__iter__` to `StringMethods` a while back. i'm just going to add the test and keep `len` out of `is_list_like`

oh i also need 2 change the replace docs i think...

nope nothing about `infer_types` in there

ok build is passing.

@jreback merge-able round 2?

looks good
very minor point - can u put next to where (or even include) as part of your other replace changes in release notes - then they r all together
BLD: add setup_requires in setup.py so pandas can be used with buildout
State: closed

Numpy is a setup-time dependency due to some .h files, but is not declared as such. This causes tools like buildout ( http://buildout.org/ ) to fail.

Adding the `setup_requires` kwarg to the setuptools config shoud fix this: buildout will correctly assemble any declared setup-time dependencies, before executing the pandas build.

closes #3861


Labels:

Comments:
@wesm any problems with this?

:thumbsup: 

thanks @ejnens 

related https://github.com/statsmodels/statsmodels/issues/588 / https://github.com/pydata/patsy/pull/16, but :thumbsup: .
BUG: fix unicode -> str cast in tslib
State: closed

This should use format since calling str on a unicode string is a _bad_ idea 
because it may or may not repr correctly.

closes #3875.

another error is created from fixing this issue.


Labels:

Comments:
the other error created is #3899 

@jreback this is ready 2 go can i merge?

how was this breaking?

when the `TypeError` was raised the hebrew unicode would throw a `UnicodeEncodeError` because `str` was being called on it

this was also hiding the infinite recursion bug

ok then
API: (GH3888) more consistency in the to_datetime return types (given string/array of string inputs)
State: closed

closes #3888

resolves the following inconsistencies in the Timestamp/to_datetime interface
Things that do the same thing will now _do the same thing_!

```
In [1]: to_datetime = pd.to_datetime

In [2]: to_datetime('')
Out[2]: NaT

In [3]: to_datetime(['', ''])
Out[3]: 
<class 'pandas.tseries.index.DatetimeIndex'>
[NaT, NaT]
Length: 2, Freq: None, Timezone: None

In [4]: Timestamp(0)
Out[4]: Timestamp('1970-01-01 00:00:00', tz=None)

In [5]: to_datetime(0)
Out[5]: Timestamp('1970-01-01 00:00:00', tz=None)

In [6]: to_datetime(['2012'])[0]
Out[6]: Timestamp('2012-01-01 00:00:00', tz=None)

In [8]: to_datetime('2012')
Out[8]: Timestamp('2012-01-01 00:00:00', tz=None)

In [9]: array = ['20120101','20120101 12:01:01']

In [11]: list(to_datetime(array))
Out[11]: 
[Timestamp('2012-01-01 00:00:00', tz=None),
 Timestamp('2012-01-01 12:01:01', tz=None)]

In [13]: map(Timestamp,array)
Out[13]: 
[Timestamp('2012-01-01 00:00:00', tz=None),
 Timestamp('2012-01-01 12:01:01', tz=None)]
```

Note that the following is still inconsisten and will be fixed in a future PR

```
In [14]: Timestamp('2012')
Out[14]: Timestamp('2012-06-18 00:00:00', tz=None)

In [15]: to_datetime('2012')
Out[15]: Timestamp('2012-01-01 00:00:00', tz=None)
```


Labels:

Comments:
Look forward to this begin merged as I ran into problems due to the very inconsistencies this PR addresses. Thanks for the nice work!

great.....if you have any other things you have notice, pls let me know!
FIX hash of DataFrame raises Typerror
State: closed

fixes #3882

raise TypeError if trying to hash a DataFrame (or Panel etc.).


Labels:

Comments:
~~Should any PandasObjects be hashable? (pretty sure answer is yes)~~

Currently this makes NDFrame not hashable. Changing the message for Series too with your improvement and adding tests :)

All tests pass, so I guess we aren't hashing anything. There were know hashing tests in the docs before.

Merge?

ok by me
ENH: do not convert mixed-integer type indexes to datetimeindex
State: closed

closes #3877.


Labels:

Comments:
@jreback any thoughts/opinions here? i'm not sure if i really want this because of `eval` or because it makes sense, feel free to tell me that i'm too gung-ho :) on the one hand just because datetimeindexes have join precedence over integers they shouldn't then assume that the joined index of integers refer to dates. on the other hand how often (or not) does one have an object index containing mixed integers and datetimes?

hmm....maybe this should be changed a bit to only allow certain types of indicies on a time-series join.

e.g. string, integer (only if time-series convertible)

I would raise I think if inferred_type != object or the TypeError is triggered

of course see what this breaks

one problem with the second point is that i believe all integers except for reserved ones like `nan`/`inf` are time-series convertible. i think mixed-integer datetimeindexes are too ambiguous to just assume that the ints are datetimes thus only this change...i will play around to see if being more strict with joins breaks the entirety of pandas

sweet only breaks 4 tests 2 of which are trying to convert, easy enough to change, need to look at others

@jreback legacy support is broken if you raise and only allow strings to be converted. currently it's really only an issue with converting mixed-integer type indexes. unless one of the goals is to raise whenever an object index is being used...thus discouraging users from using it. my change here preserves legacy support and fixes the issue of conversion i can do the raise if u want but is it too soon for htat?

that's fine then
I always want to have general solutions rather that specific related to types but
sometimes unavoidable

i agree that it's annoying to have to special case this but the int<->datetime mapping makes it necessary when u already have an object index :(

@jreback anything else u think i should add here? if not should i merge?

go ahead

@jreback u were right after all, all these issues are related to this ....

the fix or the symtom?

symptom
FIX: change initObjToJSON return type
State: closed

This is necessary because clang complains about the return type. There's a
call to the macro import_array() which injects a return statement into
wherever it's used.

closes #3872.


Labels:

Comments:
also cc @Komnomnomnom 

what do u think?

Sounds good to me, as a fix in ultrajson I think it should be pushed upstream to them as well though. Which I can do once I've merged in their recent changes, as I might have more stuff for them.
DOC add to_datetime to api.rst
State: closed

Either I'm being thick or `to_datetime` isn't in the docs (does adding it like this add it?)

Should also put something in basics...?


Labels:

Comments:
there should be an example in timeseries.rst and/or basics.rst
but can't remember one

should be generated by the API docs. grin shows nothing in those that @jreback just mentioned

@jreback http://pandas.pydata.org/pandas-docs/dev/search.html?q=to_datetime&check_keywords=yes&area=default only result is "what's new".

Should one go in basics or timeseries? (I'd vote for basics.)

I would put near the top of timeseries (before time spans)
prob could create a separate section (conversions?)
as several options need to be explained, format, day first, tz, 
and a link / mention in basics / dtypes

Can I include unit ? :)

yeh...goign to merge that right now in fact......

I think its a reasonable api, yes?

units defined in  #3818

@hayd fyi...I believe that `format` option is currentlly ignored, remember that from somewhere...?

I was worried this was going to be a barrel of worms. maybe there is a reason this function isn't documented... :p

is it a barrel?

Nah... putting something together now. (sorry, just got flustered with above issue)

Also that dayfirst isn't strict. :S

@jreback @hayd merge?

I don't think this is ready yet

k

It _could_ be merged as is, it would at least get `to_datetime` to the docs.

I've written "something" about using `to_datetime` in tseries api, but not pushed it yet...

I've done a little more on this, feedback welcome...

`pd.to_datetime([1, '2'])` is possibly surprising maybe note that?

`dateutil` defaults on making a date string of just an integer midnight on the current day iirc

@cpcloud woah! I'd never done: `pd.to_datetime(['2'])`, er!! Will add it in.

That is also strange, and the thing which means that `to_datetime` isn't just `map(Timestamp, x)`, I dislike examples where it isn't (but they need to be included).

this is good!

see #3890, which resolves almost all of the inconsistencies

Ah cool! Didn't realise hasn't been merged yet! :)

This is still whacky:

```
In [4]: pd.to_datetime([1, '1'])
Out[4]:
<class 'pandas.tseries.index.DatetimeIndex'>
[1970-01-01 00:00:00.000000001, 2013-06-01 00:00:00]
Length: 2, Freq: None, Timezone: None
```

so will mention it

not sure there's any other way to go about it. need to use ints for representation but want to be able to parse semi human readable dates and `int`s...dateutil is _extremely_ lenient except e.g., it won't parse an `int`. hopefully no one is abusing pandas in this way

can u add that last to #3944?

I think we are all agreed that the default to_ datetime should be more strict

done

that why I separated out #3944 

dateutil lets everything thru

so have to do some most common case parsing first

need to expand c date parser then can pretty much drop need for dateutil except as fallback

(and it's not strict for dayfirst grrr!)
ENH Prefer requests over urllib2
State: closed

At the moment we use urllib2 for http requests, this branch prefers to use requests if it's installed. For one thing it means you can read gzipped json, which is more difficult in urllib2:

```
url = 'https://api.stackexchange.com/2.1/search?page=1&pagesize=10&order=desc&sort=activity&tagged=pandas&site=stackoverflow'
result = pd.read_json(url)  # in master this raises ValueError: Expected object or value
```

cc #3804 @jreback

Thoughts?


Labels:

Comments:
Probably needs some testing with and without requests installed (quite a few io tests skip for ~~me~~ travis).

maybe a mention in install.rst of the requests module as an optional dep

might want to note also that `read_html` hasn't been tested with `requests` as well...

I had a go with html, but got some [weird lxml error](https://travis-ci.org/hayd/pandas/builds/7997369)... (removed them in 2nd commit).

u have to catch `Exception` since you must be able to catch exceptions that are not due to invalid HTML and propagate them, even when `lxml` is not installed, the `ValueError` clause u changed probably caused the `XMLSyntaxError` to not be caught

it's a bit annoying since i actually have to call `isinstance(e, XMLSyntaxError)` but _only_ when i know that `lxml` is installed (since that's where that is defined) thus all of the kludge around parsing...

Will have another play later in the week.

I actually had a similar thing in the request, checking for exceptions which the user may not have installed (I hid it behind an ImportError: pass). I definitely got into a muddle on that particular Exception...

i think there needs to be a mother-of-all-read-functions in `io.common` since there seems to be lots of repetition of the function with very tiny variations...i have an issue somewhere

yep...including the options NOT to read (e.g. I think html and read_csv want to read line-by-line), most others (e.g. json), io.data, etc. just need the buffer already read

html reads the buffer or punts the file-like object to the backend if possible. i think csv is the only iterable one so far

Do I need to worry about testing in Travis both with and without the requests module?

I'm concerned this might have some weird issues either way so push to 0.12?

there's the `FULL_DEPS` variable...check out the build matrix on a past run i don't remember which runs it's used on.

you could add to all if u want
update ci/install.sh
and ci/print_versions

see also #2426

related #3811

@hayd going to resurrect this for primary web access (with fallback to urllib2)
?

and put it io/common if not already there

then need to change uses of web acces across io to use this

and then docs!

see #4140 Google BigQuery is dependent on httplib2 is that a viable alternative here?

@jreback will have a loook

Not sure what to do with this one (certainly a lot has changed in io.common)
TST: Fix missing import in io/tests/test_json
State: closed

Nose import is missing. If you get to the error at the last line, it throws an error because nose is never imported.


Labels:

Comments:
were line endings screwed up?

no, I mean that it raises `nose.skipTest` but never has `import nose`. Just ran the test and it failed that way, which is how I noticed it.

however, it looks like github screwed up the commit

so I'll redo it on my computer :-/

hmm I ran these using nosetests directly so maybe that's why didn't have a problem 

You have to get a urllib2.UrlError to see it...

Much better now...as you can see, it's a oneline change.

yesterday: https://dev.twitter.com/blog/api-v1-is-retired lol
Fixing get_data_yahoo/google pause and retry
State: closed

Provide pause and retry_count to _get_hist_yahoo/google when single value given for symbols. Now pause option works, as described in docs.


Labels:

Comments:
Thanks for this!

@gliptak cc @jtratner
this looks independten of your network catching stuff?

@Rambominator this was essentially a bug fix, right? (e.g. the retry didn't work before).....

can you add a test (e.g. maybe test a bogus symbol), then just a real symbol to validate its working

pls add a mention in release notes

> @Rambominator this was essentially a bug fix, right? (e.g. the retry didn't work before).....

Yes, a bug fix. Users provided arguments: pause and retry_count, was not provided to the function which realy uses them, so function always worked with default params retry_count=3, pause=0.

> can you add a test (e.g. maybe test a bogus symbol), then just a real symbol to validate its working

Symbols has nothing with this fix (with wrong symbol get will raise 404), as I see it, the only possible test is mocking urllib2.get, so it will not return 200 response, and compare time >= retry_count \* pause

> pls add a mention in release notes

added

@Rambominator sorry i didn't see this before. in my clean up of data.py this has been addressed.

thanks
Io to clipboard
State: closed

Added documentation for to_clipboard().
Closes #3784


Labels:

Comments:
~~Is it worth adding a note in the APIs about system requirements aka #3837 (not if below pull is merged.)~~

@nipunreddevil  If you update your banch that will be reflected here.

OK. Whats the note to be now put in the documentation?

On Thu, Jun 13, 2013 at 5:18 PM, Andy Hayden notifications@github.comwrote:

> @nipunreddevil https://github.com/nipunreddevil If you update your
> banch that will be reflected here.
> 
> —
> Reply to this email directly or view it on GitHubhttps://github.com/pydata/pandas/pull/3845#issuecomment-19386728
> .

Updated to put code in one ipython block and note reflects documentation from docstring.

Thanks for spotting the typo!

Fixed the typo. The docs still take ever so long to build!

you should rebase on master, that prob was fixed a few commits ago

```
git co master
git pull
git br yourbranch --set-upstream master
git co yourbranch
git pull --rebase
```
ENH: add figsize argument to DataFrame and Series hist methods
State: closed

closes #3834 


Labels:

Comments:
@jreback ok with this?

don't we always just pass kwds to matplotlib anyhow? (I am not disagreeing with this, because its nice
to do this on the front-end), just curios

yes but they weren't being propagated correctly because they were being passed into funcs that don't use them e.g., grouped_hist. plus hist_series was calling gca which creates a figure before u can give it a figsize. u might be able to adjust that post creation but my method requires less mpl api munging

oh i just saw something that i need to change can't merge yet..

minor point, figsize is in inches I think? (subject to dpi), (may want to mention that)

units there are customizable i believe but ok

@jreback this is good 2 go...shall i merge?

ok
Add msgpack as submodule for pandas
State: closed

Let's get the ball rolling on better high perf serialization. Note this is based on the 0.3.0 release of msgpack-python, should be updated whenever another stable release of the library is made. 


Labels:

Comments:
@wesm is there a reason you are using `pytest`? rather than nose

Unit tests are directly from the msgpack module, should be adapted to nose I guess

closing as this is rolled into #3831
CLN: remove relative imports
State: closed


Labels:

Comments:
This fixed the (motivating) syntax error from #3821, and has the benefit of working on py3 (when not built is show ImportError rather than syntax error). Merge?

will rebase and then i willl merge

@jreback good 2 merge?

what exactly is the reason the relative imports are bad?

i don't think they are bad necessarily, but they're certainly not needed here since they were used to refer to `pandas`. see [here](http://stackoverflow.com/questions/4209641/absolute-vs-explicit-relative-import-of-python-module) for short pro/con discussion. also the exception being caught here was too general...whenever i have an installation issue it's hidden by a cryptic error message... about not being able to import `hashtable`

Your error message is a big improvement, (and solves the even less descriptive "syntax error" in py2.5+3).

also importerrors should not be caught by `except Exception` clauses they should be caught by `except ImportError:` let's call a spade a spade... 

sounds good then
ENH: DataFrame.corr(method='spearman') is cythonized.
State: closed

It should be more than 10 times faster than the old version.


Labels:

Comments:
ok....can you put a release notes mention? (enhancement section),

and if you are gung ho, pls add...a vbench entry (vb_suite/stat_ops.py)....it could have a start date of say 2013/1/1 (or earlier if you can figure out about when spearman was added....was sometime last year, but I don't remember)

@jreback is there any consensus about fused types/memoryviews? if there's a consensus to use should we "require" that new cython use it?

should use them
but in this case doesn't matter as float64 is all that is needed

fused types sure but what about memoryviews here?

if u can get them to work
should do it

@jniznan does @cpcloud change make sense?

Yes, I will push the changes in the next day or two.

@jniznan can you see if you can finish this up ? thanks

@jreback  sorry about the delay, 
it seems that the spearman functionality was added on 2011/12/3 (GH #428) , so i will put 2011/12/4 in the benchmark. 

@jreback i put in the release notes mention and the vbench entry, i have also implemented the change @cpcloud proposed

@jniznan can u rebase?

@cpcloud it should be rebased now.

Merged...thanks.....nice improvements

```
-------------------------------------------------------------------------------
Test name                                    | head[ms] | base[ms] |  ratio   |
-------------------------------------------------------------------------------
stats_corr_spearman                          |  83.9807 | 3735.8600 |   0.0225 |

Target [a70b48a] : PERF: stats_corr a smaller vbsuite
Base   [c4a218c] : Merge pull request #3929 from y-p/PR_PTF_no_more
```
ENH: Add unit keyword to Timestamp and to_datetime
State: closed

to enable passing of integers or floats that are in an epoch unit of s, ms, us, ns
    (e.g. unix timestamps or epoch s, with fracional seconds allowed)

closes #3540

```
In [5]: pd.to_datetime(Series([ 1370745748 + t for t in range(5) ]),unit='s')
Out[5]: 
0   2013-06-09 02:42:28
1   2013-06-09 02:42:29
2   2013-06-09 02:42:30
3   2013-06-09 02:42:31
4   2013-06-09 02:42:32
dtype: datetime64[ns]
```


Labels:

Comments:
(I wonder if it makes it less ambiguous here to have the default unit for `to_datetime` as 'ns' rather than None?)

they are both treated the same (but i'll change it in any event)

actually...going to leav e it as None, becuase it doens' tapply in most cases and might be confusing?

I was thinking that too, _but_ I think a little rewording may solve it, I was going to suggest something like:

```
unit : unit of the arg (s,ms,us,ns) if passed an epoch time # numerical?
    (e.g. a unix timestamp)
```

ok...that's reasonable....sure

@wesm any thoughts?

This used to cause an exception, but maybe now is a good time for this to return NaT? :)

In your branch:

```
In [2]: pd.Timestamp(np.nan)
Exception ValueError: 'cannot convert float NaN to integer' in 'pandas.tslib.cast_from_unit' ignored
Out[2]: Timestamp('1970-01-01 00:00:00', tz=None)
```

think I got all of the cases, thanks!

```
In [2]: Timestamp(np.nan)
Out[2]: NaT

In [3]: Timestamp(pd.NaT)
Out[3]: NaT

In [5]: Timestamp(pd.tslib.iNaT)
Out[5]: NaT

In [6]: Timestamp(float(pd.tslib.iNaT))
Out[6]: NaT
```

@wesm ok API by you?