BrandonFors/pandas-github-issues huggingface dataset:
Tag yahoo data tests as @network only
State: closed


Labels:

Comments:
this ok to merge? 

https://www.travis-ci.org/pydata/pandas/jobs/7992099

@gliptak @hayd can you look at this.....

need to handle better a failure to get data....not sure of the current standard, raise? 

@jreback Could you be a bit more specific? I didn't see the actual error in the logs you are pointing to.

I'm also trying to understand why #3817 fails (as it was running without errors locally ...)

it's an intermittent failure I think because on one test the connection failed
click on the Travis link above
look at the failed build
scroll to the bottom you will see where the test failed
prob just need to put in a better way to handle the None coming back 

What is a good way to deal with it without it just being skipped? (or is just skipping it if it's None ok?)

if in general their is a network error what do u do raise some urlopen error?
just catch that and skip in that case

in fact should probably have your new requests pr raise a common NetworkError or something

@jreback These are new tests for existing (and functionally unmodified) code, which did not have tests before.

OK, I see it now. I will change the tests to check for None.

```
ERROR: test_get_quote (pandas.io.tests.test_yahoo.TestYahoo)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/travis/virtualenv/python3.2_with_system_site_packages/lib/python3.2/site-packages/pandas-0.11.1.dev_e958833-py3.2-linux-x86_64.egg/pandas/io/tests/test_yahoo.py", line 46, in test_get_quote
    assert_series_equal(df.ix[0], df.ix[2])
AttributeError: AttributeError: 'NoneType' object has no attribute 'ix'
-------------------- >> begin captured stdout << ---------------------
Failed to download:
<urlopen error [Errno 111] Connection refused>
```
DOC/BLD: Fix a bunch of doc build warnings that weren't being previously caught
State: closed


Labels:

Comments:
@jreback wut do u think about the deprecation message here? merge-able?

looks fine

@cpcloud nothing to do with theis PR...but can you make some of the read_html ipython blocks actually be code blocks (so for example it doesn't do some of the ops., maybe the last 2 (lxml parsing and using fallback parsing), maybe even a couple of others....e.g. just give the commands, don't actually do them....

building io docs takes a lot longer now

alternatively, maybe there is a way to build docs while disabling network type things? (e.g when you are working on other docs...)?

oh yeah duh sorry i had a thought to do this b/c of the ntwk issue.

i wonder if there's a sphinx ext for this

nope *sigh*

@jreback what about using the data i've provided in the tests? (will still mark a bunch as code-block)

eh nvm i 'll just give an exmaple using that..i need to write one for file readin ganyway

for the docs maybe cache it (read from a static file)

should i put in `doc/data`?

not sure how to cache things in sphinx

i could make a single network call, download it and trhen clean up... or two network calls...to show that passing a url works and then another to download...

put in source/_static

where the png files r

ok...do i need to pass the exact path or is there some sphinx magic there to just pas the filename alone and it will look there...?

look in doc for HDF5
I do this to read in a legacy file

thanks
Implement historical finance data from Google Finance
State: closed


Labels:

Comments:
This looks pretty cool! Good to have something in the docs / release notes.

If this is OK to be merged, I can update the docs (and I might need some help with the release notes)

It looks pretty good to me, will see what other people think, but ti seems to work.

(I tagged it for consideration for 0.11.1).

Merged. I'll add a release note

@gliptak see here for a possible bug...https://travis-ci.org/cpcloud/pandas/jobs/8040142

Caused this build to fail too (with the same error I guess) ... https://travis-ci.org/jtratner/pandas/jobs/8035403

@cpcloud  @jtratner please see #3893
DOC: add link to tips data set in rplot docs
State: closed

closes #3799.


Labels:

Comments:
oh darn also the `'totbill'` is not a column anymore

@jreback ok to merge?

i wonder why the doc build was not catching this...

hmmm.....we can include that file in the static sources (or is right?)

ok now i'm getting a key error i think there are disparate versions of tips.csv...let me investigate...

yep they are different, i'm going to put the one that i've linked to here, which is the one wes links to on his blog

@jreback going to merge if u have no issues here.

@jreback actually can u merge this and the ipython cache fix if there are no issues, i have to run out. thx!

thanks again
BUG: (GH3795) better error messages for invalid dtype specifications in read_csv
State: closed

ENH: accept 'str' as a dtype in read_csv to provide correct parsing

closes #3795

```
In [1]: df = DataFrame(np.random.rand(5,2),columns=list('AB'),index=['1A','1B','1C','1D','1E'])

In [2]: df
Out[2]: 
           A         B
1A  0.096563  0.761440
1B  0.623102  0.538810
1C  0.498820  0.277789
1D  0.113544  0.723437
1E  0.381104  0.061758

In [3]: df.dtypes
Out[3]: 
A    float64
B    float64
dtype: object

In [4]: path='test.csv'

In [5]: df.to_csv(path)

In [6]: pd.read_csv(path, dtype=str, index_col=0)
Out[6]: 
                      A                    B
1A  0.09656290409114332    0.761440208545324
1B   0.6231015058315575   0.5388097714651147
1C  0.49881957371373464  0.27778943212477014
1D  0.11354443109778356    0.723437196012621
1E  0.38110436826261596  0.06175758774696094

In [7]: pd.read_csv(path, dtype=str, index_col=0).dtypes
Out[7]: 
A    object
B    object
dtype: object
```

Invalid dtype specifciation, all `TypeError`, slightly different messages depending on what you are doing (e.g. if its completely invalid or really a date spec)

```
In [8]: pd.read_csv(path, index_col=0, dtype={'A' : 'foo'})
TypeError: data type "foo" not understood

In [9]: pd.read_csv(path, index_col=0, dtype={'A' : 'datetime64'})
TypeError: the dtype <M8 is not supported for parsing, pass this column using parse_dates instead

In [10]: pd.read_csv(path, index_col=0, dtype={'A' : 'timedelta64'})
TypeError: the dtype <m8 is not supported for parsing

```


Labels:

Comments:
@cpcloud @hayd in regards to #3795, reasonable?

maybe a test for this `read_csv(dtype={'A': str, 'B': 'datetime64'}, parse_dates=['B'])`?

or maybe it will never make it to parse_dates anyways..

I added that test, but you are right, won't make it

looks ok to me

@jreback wut do u think about an "Error Reporting" issues label. there seem to be and have been a lot of these kinds of issues often times there's another underlying issue but it might be nice to have this label

sure

Excellent! :)
link to a numerical integration recipe (Issue #3759)
State: closed


Labels:

Comments:
Hm... Something is strange in here. I think GitHub must have changed the line endings or something (I used their web editor). I'll try to fix it locally.

OK, now it's better. For some reason, using the web editor deleted and added every line in the file. A shame, because that would make adding changes to this document super fast.

I edit on windows (though run everything on linux)

I added

`git config --global core.autocrlf input` seems to _ignore_ the line endings issue (IOW it will leave it as whatever it was and not change to your standard), I think if you edit on linux, you want this as True

thank you!

My pleasure :)

Yeah, I mean it also wasn't a problem on my local env (Linux + Emacs), but only in the web editor you get when clicking [the edit button](https://github.com/pydata/pandas/edit/master/doc/source/cookbook.rst) on [the file view](https://github.com/pydata/pandas/blob/master/doc/source/cookbook.rst). I'll report it to GitHub support.

that's exactly the point it was rewriting the line endings
turn that autocrlf option on and you will be good

@kermit666 i feel ur pain. scripts just refuse to run on my linux machine with windows line endings. if u run `dos2unix` on them and then `git add` you should be fine. for some reason github recommends the opposite which is that linux users set to `input` while windows users should set to `true` not sure exactly why.
Added examples to CSV section in cookbook
State: closed

- Edited link to timezone handling with epochs
- Added link for dealing with bad lines 


Labels:

Comments:
thanks!

Thanks for merging. When do the changes reflect on the main site?

On Fri, Jun 7, 2013 at 6:40 PM, jreback notifications@github.com wrote:

> thanks!
> 
> —
> Reply to this email directly or view it on GitHubhttps://github.com/pydata/pandas/pull/3789#issuecomment-19105765
> .

i think docs built around 5pm est....so after that
DOC: document the pitfalls of different byte orders
State: closed

closes #3778 .


Labels:

Comments:
fast!

Really digging hub and I've been using shortcut foo. Still need to check out typing of the dead... 

@jreback i think this is good 2 go. now that i have push rights, is there anything to know about merging (other than git), conventions, etc.? i'm going to defer merging with master to you for the time being (i think that is wise since u've been doing it for a while) i think a practice run on something like this is good...do u use the gh interface? or do u do it thru cmd line? i'm comfortable either way, just wanted to see how u do it...

I generally use the merge button....however there are times when I merge on the command line 
which allows you to edit if needed

I also do PRs for almost everything, once in a while I will push directly to master....just have to be careful not to break things (so e.g. if its only a doc change then no biggie)

just be careful and take time

other thing is you can now edit issues, assign labels and close issues.....

I generally am pretty cautious about closing, I ttry to give people a little time to responsd and close themselves

assign labels is pretty striaghtforward

try to assign labels (and milestones) to things with no milestones

on my own PR's I try to to have someone else take a look before mergine...esp if its something new....

go ahead and click the merge button here...!

also...I try to prod people to do test cases and add release notes/docs but sometimes just easier to do it 
(that's when I would do a command line), to preserve their ownership of things....

cool thanks. maybe i will go on a milestone/issue labeling "binge" (where i'm sure of it) after making some more progress with `eval`...

sweet i like the confirm

great....it also auto-closes the mentioned issue (if you put closes issue number in the issue description)

there is also a build on travis for pandas fyi.....click on the green button in the readme (on the code tab)

bingeing....purging.....

what I think we need is to sort thru some 0.12 issues and rate flag some for priority
(as well as remaining no milestone issues)

e.g. prio-high,med,low (prob wouldn't flag everything though)

yeah the travis build link is cool. i have a travis chrome extension that tries to put those tags on everything on gh. kinda neat.

coool....fyi....if you are looking for something to do :)

I think sql module needs some tlc....mainly to do proper type conversion (and I think there are a couple of open issues about it, see labels for data i/o), e.g. doesn't handle NaT properly...etc

actually right now i'm trying to repro #3763; sql-related actually, will check out the stuff you've mentioned. i wish there were personal task lists on gh *sigh*.
ENH: Provide a default key name in read_hdf and to_hdf
State: closed

For convenience, provide a default name ("data") to quickly store and retrieve data from HDF5.

Now for csv:

``` python
df3 = pd.DataFrame(np.random.randn(10, 5), columns=['a', 'b', 'c', 'd', 'e'])
df3.to_csv("df3.csv")
print df3.from_csv("df3.csv")
```

for HDF5:

``` python
df3 = pd.DataFrame(np.random.randn(10, 5), columns=['a', 'b', 'c', 'd', 'e'])
df3.to_hdf("df3.h5", "data")
print pd.read_hdf("df3.h5", "data")
```

I would suggest to provide a default key name that would make it more convenient to read and write to HDF5 when it contains only one dataset.

In this PR:

``` python
df3 = pd.DataFrame(np.random.randn(10, 5), columns=['a', 'b', 'c', 'd', 'e'])
df3.to_hdf("df3.h5")
print pd.read_hdf("df3.h5")
```

Instead I would not provide a default key in `pd.read_hdf` because it would break backward compatibility.


Labels:

Comments:
this is too specific of a change. You might want to do this in your local installation (via monkey-patching these 2 functions), but as a general rule, this can introduce unwanted behavior.

thanks for the idea (I actually do this locally as well too)

Thanks for your feedback,
I think that the most common and natural user case is to store 1 dataset for each file, so having to specify a key seems redundant and annoying. More experienced users that want to store more datasets in the same HDF5 file can do that easily anyway.

if you're just tinkering around with an h5 file then you're probably right about the use case.

but if you have huge frames that are related in some way (this basically describes all of the data of the entire field of neurophysiology and i imagine many other sciences as well) i think storing multiple data sets per file is more common. personally, i tend to store "related" data frames (e.g., a collection of experiments related by something like the age of an animal or the experimental condition or something like that) in a single file and then i have CSV file that i read into a frame that i use to basically index into the collection of h5 files, a sort of poor man's scientific database. 

i would rather be forced to provide a key and get an error when i didn't than accidentally overwrite a bunch of data because i forgot to put a key.

I'm not sure if i'm misinterpreting what you wrote above but as for `to/read_csv` you just shouldn't have more than a single frame per file because there's really no general way to tell the two frames apart without just introducing a new serialization format that is _not_ CSV, and so it's difficult to say "`to_csv` doesn't require a key so `to_hdf` shouldn't" because `to_csv` is a single data set format.

Ok, I would still prefer to have a default key, and get an error if I try to overwrite stuff if I don't set mode to "w", but I see your point, I'll close this issue.
ENH: allow fallback when lxml fails to parse
State: closed


Labels:

Comments:
why do you have raise_on_error and flavor?

because if `raise_on_error=False` and `flavor='lxml'` throws an `XMLSyntaxError` then a parse will be attempted with bs4 + html5lib if installed. if `raise_on_error=True` then the error will propagated

i thought that was the desired behavior per the many exchanges we had + the mailing list, no?

do you still want me to implement the flavor list functionality? i can still do that...

I thought it was clearer to do:

`flavor=None` : lxml, then fallback (deafult)
`flavor=[bs]` : just do bs
`flavor=[lxml,bs]` same as default

gets you same resutl, but only 1 kw

you just try with all the flavors you are given (and default to both), then if you have reached the end of your flavors list and still haven't parsed, raise.....

ok doing it now...

u r right that is much more clear

looks good...ready to merge?

not just yet running tests i missed a format string and changed an exception type, almost finished

ok...np..lmk

AH ur killing me smalls!

isn't lxml supposed to install? https://www.travis-ci.org/pydata/pandas/jobs/7842837

i see it installed by apt-get...i though it was installed by pip. let me take a look

yes but print_versions not showing it
weird

i got it. `bs4` is installing it. there might be a build caching issue that i didn't take into account 

i really admire whoever spent the many many hours it probably took to get travis working. really. sheer willpower.

GAAH! i didn't know python3 has changed the scope such that exceptions raised in a loop don't leak to their outer scope!!!

wow and it even clobbers an existing binding, this is so annoying.

``` python
for i in range(10):
    try:
        raise Exception()
    except Exception as e:
        pass
print(e) # raises a NameError or UnboundLocalError
```

now i see it in the [docs](http://docs.python.org/3.0/whatsnew/3.0.html#changes-to-exceptions)

either way if this passes i will squash and it is ready 2 merge (FINALLY!)

YES IT PASSED!!!!! @jreback this can probably be merged BUT, i think it would be prudent to wait a few hours just in case...or i think of some more tests to add.

hahha...ok...lmk

ok...ready to merge this?

yep this is good 2 go. shall i merge?

yes...fyi you can assign yourself to issues

then you can see your assigned tasks! a personal list

oh yes! i forgot about that. thanks.

you assign other people too! (but I don't think anyone actually does that)
DataFrame.corr() now computing correlations only once
State: closed

DataFrame.corr() for 'spearman' and 'kendall' was computing some correlations twice.


Labels:

Comments:
can you add a test for this? e.g. generate a random frame, compare this results vs. numpy?

@jniznan the following is probably faster as u avoid the explicit check and u don't create the `enumerate` objects

``` python
n = mat.shape[0]
for i in xrange(n):
    ac = mat[i]
    for j in xrange(i + 1):
        bc = mat[j]
        valid = mask[i] & mask[j]
        if valid.sum() < min_periods:
            c = NA
        elif not valid.all():
            c = corrf(ac[valid], bc[valid])
        else:
            c = corrf(ac, bc)
        correl[i, j] = correl[j, i] = c
```

future note that this could be cyhonized

yes looks very similar to the cython code for pearson

The test is already in `pandas/tests/test_frame.py`, function `test_corr_spearman(self)`.
Anyway, I have cythonized the spearman function so I will create a new pull request.

closinging in favor of #3823
TST: Fix assert_almost_equal error message
State: closed

Most tests (in pandas' test suite and in general) are of form
`assert_almost_equal(result, expected)`.  Verbose error message was
treating its first argument as expected, this is now fixed.

Previous message was:

``` python
err_msg = lambda a, b: 'expected %.5f but got %.5f' % (a, b)
```

But a, the first argument, is actually `actual`, not expected.

Everywhere else, it makes sense to display `"%s != %s" % (a,b)`


Labels:

Comments:
can you hook up travis? (I know this is a trivial PR, but for the future)

I set it up...hadn't realized I needed to enable it in travis-ci. Do you want me to redo this commit to trigger it?

sure
git commit --amend -C HEAD
will force a Travis update

got it - thanks!

@wesm this looks fine for 0.11
API: deprecate unused DataFrame.replace arguments
State: closed

@jreback i forgot to do this as per our conversation


Labels:

Comments:
cool...

@jreback i just rebased this and all of my other prs just in case u were wanting to merge any of them :)

hmm...this might need to be trivially rebase as well

done

thanks for all the PRs!

trying to finish up with 0.11.1....so unless critical....let's hold off (docs always good though)
DOC: document read_html and to_html
State: closed


Labels:

Comments:
FYI after #3693 u won't need to import read_html explicitly anymore

ok i will remove

i changed a few things around in `html.py` nothing major, but now `read_html` attempts to infer row and columns index types if they are not `MultiIndex`es. rls notes for this? it doesn't try to infer dates

you might need to rebase this after #3693

done.

Where you show reading with an attribute, I think you are missing something?

It looks the same as the prvious example

http://pandas.pydata.org/pandas-docs/dev/io.html#io-html

where  r u looking i don't see the attrs anywhere...also a problem

oh i c

it's missing -u prob copied the prior example

man the io docs take 4ever 2 compile

there's probably a very tight correlation btwn amt docs amt of tests and amt of code all pairwise

true!

for 0.11 I had to keep rereading the docs
For all of the indexing chnges
as no tests for language!

pr is up
ENH: allow to_html and to_latex to take a path for their first argument
State: closed


Labels:

Comments:
not so sure that `to_string` should take this argument; its for a pure string rep, while `to_csv` has all the options for outputing to a file (eg. specialized nan rep, separators, etc)

@y-p ?

Yeah that's right. More things to maintain for u guys and probably would be confusing for new users. Will take out 

Agree with jeff, doesn't belong in `to_string`.

I would opt for `isinstance(basestring)` here, 'better to ask forgiveness" not withstanding.
`to_html(FooObject)` should not try to `open(FooObject)`, it should give a sane error message.

@y-p indeed, thanks.
BUG: DataFrame.to_csv fails silently if nanRep not None
State: closed

The nanRep argument to DataFrame.to_csv() is deprecated and na_rep should be used instead. .to_csv should succeed with a warning if nanRep is specified. Before this fix, to_csv threw a warning but did not save the file.


Labels:

Comments:
merged  via16b28f186d9810d50486ea488493ec1f810a1bdb
thanks!
ENH/API: implement __nonzero__ for NDFrame
State: closed

closes #3691.


Labels:

Comments:
this didn't break any tests? (other than the one that is meant to catch it)

Nope. I changed that test to reflect the api change. 

I figured that it would be more difficult to do this but it seems like the object hierarchy is very clean and only getting better :) 

can u rebase?

done