BrandonFors/pandas-github-issues huggingface dataset:
ENH: IO World Bank WDI
State: closed

The World Bank's World Development Indicators (WDI) is a collection of nearly 8000 data series in country-year format. These data cover a wide range of topics (e.g. demography, economics), and they are freely available freely through the World Bank API.

http://data.worldbank.org/

This PR introduces a very simple io module to search and downloading data from the WDI. Usage example: http://nbviewer.ipython.org/4365089/ 

I don't know the best way to test features such as this one, so I may need a bit of help there. 


Labels:

Comments:
?

I noticed that lots of data series weren't downloading properly. I'm investigating. If there's interest on your side, I will reopen as soon as I figure out what's happening. 

OK. It looks like the World Bank advertises more data series than they actually publish through the API. I don't see an easy way to distinguish between available and unavailable series in the search() function, short of pinging each series. 

Thus, I think we can consider that this function works as-is. I will contact the World Bank people to see what can be done to improve, but this should still be useful now. 

thanks!

@vincentarelbundock I added issues #4354, do provide docs for this (could be pretty simple, really just advertising), maybe in io.rst in the Data section
Update force_unicode documentation
State: closed

Should resolve #2053.

For consistency, I've re-added the `force_unicode` parameter to `to_latex()` and have it now raise a deprecation warning. I've also updated the common docstring for the `to_*` methods to have a brief explanation as to why the parameter is deprecated.

There were a couple other missing deprecation warnings (`force_unicode` in `to_html()` and `colSpace` in `to_latex()`) that should now have consistent behavior with other methods.


Labels:

Comments:
merged, thanks!
Add some default parameters to TextParser in _parse_options_data()
State: closed

The Options class can return some awkward dtypes:

``` python
In [1]: from pandas.io.data import *

In [2]: aapl = Options('aapl')

In [3]: a = aapl.get_put_data(1,2013)     

In [4]: a.dtypes
Out[4]: 
Strike       object
Symbol       object
Last        float64
Chg         float64
Bid          object
Ask         float64
Vol          object
Open Int     object
```

There are two causes for Strike/Bid/Vol/Open Int having `object` dtypes: not parsing the NaN value and not passing a thousands separator.

``` python
In [5]: a.Bid[:5]
Out[5]: 
0    0.01
1     N/A
2     N/A
3     N/A
4     N/A
Name: Bid

In [6]: a['Open Int'][:5]
Out[6]: 
0    8,744
1    2,381
2    1,117
3    4,586
4    1,535
Name: Open Int
```

I've checked with a couple non-US proxies and it seems that finance.yahoo.com (the hardcoded url) does not redirect to the local version (yahoo.com does redirect). Since we're getting the US site, we can pass `','` as the thousands separator and `'N/A'`as the NaN value.

These changes yield the following dtypes:

``` python
In [4]: a.dtypes
Out[4]: 
Strike      float64
Symbol       object
Last        float64
Chg         float64
Bid         float64
Ask         float64
Vol           int64
Open Int      int64

In [5]: a[:5]
Out[5]: 
   Strike               Symbol  Last   Chg   Bid   Ask  Vol  Open Int
0     135  AAPL130119P00135000  0.02  0.01  0.01  0.03    5      8744
1     140  AAPL130119P00140000  0.02  0.00   NaN  0.04   11      2381
2     145  AAPL130119P00145000  0.01  0.03   NaN  0.11    3      1117
3     150  AAPL130119P00150000  0.01  0.00   NaN  0.11   20      4586
4     155  AAPL130119P00155000  0.01  0.00   NaN  0.11    4      1535
```


Labels:

Comments:
thanks!
CLN: Remove a useless parameter in Series.iteritem method.
State: closed

Hi,

I was reading some docstring of some `Series` & `DataFrame` methods from IPython. I wondered what the parameter `index` of the method `Series.iteritems` -- and thus the `iterkv` method -- did. I think it's not used. Tests passed.

Cheers,
Damien G.


Labels:

Comments:
thanks!
BUG: DatetimeIndex.unique numpy 1.6 repr #2563
State: closed

#2563


Labels:

Comments:
Boxing + dtype=object hash table pass (vs. int64) --> 300x slowdown. we can make this into a vbenchmark

```
In [48]: rng = date_range('1/1/2000', periods=10000, freq='T')

In [49]: arr = rng.repeat(10)

In [50]: timeit arr.unique()
1000 loops, best of 3: 1.11 ms per loop

In [51]: timeit arr.asobject.unique()
1 loops, best of 3: 320 ms per loop
```
ENH: added support for data column queries
State: closed

- can construct searches on the actual columns of the data, by passing keyword `data_columns` to `append`
  e.g. store.select('df_dc',[ Term('B>0'), Term('string=foo') ]) 
         (where B and string are columns in the frame)
- added `nan_rep` for supporting string columns with nan's in them
- corectly determine dtypes of data columns that we cannot deal with (date,unicode), raise UnImplementedError
- added support for datetime64 in columns
- added method `unique` to fast retrieval of indexables or data columns
- added keyword `index=True` to automagically create indicies on all indexables and data columns!
- performance enhancements on string columns
- added `chunksize` parameter to append to allow write chunking, significantly lower memory usage on writes
- added `expectedrows` parameter to append to allow specification of the TOTAL expected rows in a table (to optimize performance)
- added `start/stop` parameters to select to allow limiting of the selection space
- added multi-index dataframe support (closes GH #1277 )
- added `append_to_multple`, `select_as_multiple` and `select_as_coordinates` methods to support multiple-table creation & selection
- removed compression kw from put, replaced with complib (to make more consistent across library)
- Term parsing a little more robust
- more tests & docs for data columns
- added whatsnew 0.10.1 docs


Labels:

Comments:
@wesm i think this is ready to merge...also...if at all possible, once merged, can put up a dev build for 2.7 amd-64 on the site for testing....

@wesm I believe these changes will allow you to close: GH #512, GH #1277 
GH #698 and GH #2397 can also be closed (as of 0.10.0)

@wesm done adding things - ready to merge when u r

Hey jeff, one problem here is that the legacy test file is about 10 megs-- I don't want to bloat the size of the git repo or source archive if at all possible. fixing it may not be so bad using the interactive rebase approach described here:

http://stackoverflow.com/questions/2100907/how-to-purge-a-huge-file-from-commits-history-in-git

if you don't have time i could take a crack at it. will need a smaller test h5 file, though, i guess

I can make smaller np

will repost in a few

On Dec 28, 2012, at 8:59 AM, Wes McKinney notifications@github.com wrote:

> Hey jeff, one problem here is that the legacy test file is about 10 megs-- I don't want to bloat the size of the git repo or source archive if at all possible. fixing it may not be so bad using the interactive rebase approach described here:
> 
> http://stackoverflow.com/questions/2100907/how-to-purge-a-huge-file-from-commits-history-in-git
> 
> if you don't have time i could take a crack at it. will need a smaller test h5 file, though, i guess
> 
> —
> Reply to this email directly or view it on GitHub.

posted a revised file much smaller now
thxs

ok I did the interactive rebase
then added the new file 
had to force update so hopefully merge works ok 

Oops. I had already done the rebase. All is good and merged now, closing the PR
New Pandas Cheat Sheet
State: closed

Pandas cheatsheet 
- created during the PyData Workshop-Sprint 2012
- Hannah Chen, Henry Chow, Eric Cox, Robert Mauriello


Labels:

Comments:
Can you guys make this into something that people can build into a landscape PDF document?

Sure, assuming there's some Sphinx support. 

It's still very rough. All I did was merge and reformat. But I can do some work on it. 

On Dec 17, 2012, at 7:01 PM, "Chang She" notifications@github.com wrote:

> Can you guys make this into something that people can build into a landscape PDF document?
> 
> —
> Reply to this email directly or view it on GitHub.

portions of this included in cf9bf08b914b33025a26558fee68f7fb891411f1

close this?

isn't this a duplicate of #1618 ?

in 0.11

http://pandas.pydata.org/pandas-docs/dev/10min.html

@michaelaye my comment hit the other issue

looks like a dupe 2 me

I can do some work to merge and/or update the 10min.html doc?

On Mar 14, 2013, at 10:51 PM, jreback wrote:

> in 0.11
> 
> http://pandas.pydata.org/pandas-docs/dev/10min.html
> 
> @michaelaye my comment hit the other issue
> 
> looks like a dupe 2 me
> 
> —
> Reply to this email directly or view it on GitHub.

feel free.....love to have any updates to 10min.html (maybe close this PR and open another when you are ready?)

Thanks for the effort guys, but closing since this has stalled.

My apologies if you needed something from me. I thought it was closed also. 

On Jul 29, 2013, at 1:14 AM, y-p wrote:

> Thanks for the effort guys, but closing since this has stalled.
> 
> —
> Reply to this email directly or view it on GitHub.
BUG: pytables minor changes
State: closed

- ENH: moved Term to top-level panas namespace (like HDFStore)
- BUG: writing a Panel4D in cython (lib.pyx)
- BUG: interpreting axes paremeter to orient data
- BUG/ENH: allow appending of a different order of non_index_axes data 
  which then is ordered according to on-disk data
- DOC: explanation of how to orient data to provide optimal append/delete operations
- BUG/TST: more complete coverage of ndim and non-default axes ordering
- removed debugging memory calls


Labels:

Comments:
@wesm bug list getting shorter!
ENH: vbench support for HDFStore
State: closed

added benchmarcks to compare (100,000) rows:
- read/write store
- read/write store mixed
- read/write table
- read/write table wide (200 columns)
- read/write table mixed
- query wide/table


Labels:

Comments:
The start dates on these are too early (they use newer apis)

true....so a lot will fail, but read/write store should work
what date is a good start point?

i started with io_bench.py....that's where they came from

Maybe like 2012-07-01 or so. Anytime 0.8.0 onward should be OK. If you go to some effort you can make the tests work on earlier pandas but prolly not worth the effort

just check 7.0...should work for that (not the mixed types or queries...but evrything else)...2/1/12?

done

thanks!
option callback on value change, use an option to control use_inf_as_null
State: closed

I'm not married to the "mode.X" option key, but be aware that the dynamic generated 
docstrings for the options API favour a small number of top level keys with multiple keys
under them and so i consolidated `test.interactive` into `mode.sim_interactive`.
Some sane taxonomy needs to be established early on to keep things tidy,
Be an option-naming fascist I say.

http://martinfowler.com/bliki/TwoHardThings.html

related: #1919


Labels:

Comments:
@wesm , just want to make sure this and the other 0.10 PR's won't get left behind,
so we avoid back-compat issues.

don't worry, i'll definitely give fair warning before cutting 0.10 final...
BUG: create correctly named indexables in HDFStore Tables
State: closed

- indexable columns were created and named in a legacy format, now named like the
  indexable in the object, e.g. 'index' for DataFrame, or 'major_axis'/'minor_axis' for Panel
- fixes an issue if we want to support say a 'column' oriented Table
   (e.g. instead of storing the transpose,
  store with a 'column' indexables) - not implemented yet, but supported now


Labels:

Comments:
Cool, cherry-picking now

cool....btw...i potentially have 1 more pr for pytables....adding support for Panel4D....when are you going to freeze for release?

I think we can slip it in anytime this week
ENH: Add batteries-included index and DataFrame generators to tm
State: closed


Labels:

Comments:
``` python
In [1]: from pandas.util.testing import makeCustomDataframe as mkdf
   ...:   mkdf(5,3)
Out[1]: 
C0      C_l0_g0 C_l0_g1 C_l0_g2
R0                             
R_l0_g0    R0C0    R0C1    R0C2
R_l0_g1    R1C0    R1C1    R1C2
R_l0_g2    R2C0    R2C1    R2C2
R_l0_g3    R3C0    R3C1    R3C2
R_l0_g4    R4C0    R4C1    R4C2

In [2]: from random import randint
In [2]: mkdf(5,3,data_gen_f=lambda r,c:randint(1,100))
Out[9]: 
C0       C_l0_g0  C_l0_g1  C_l0_g2
R0                                
R_l0_g0       38        9       64
R_l0_g1       42       45       94
R_l0_g2       15       94       16
R_l0_g3       80       39       33
R_l0_g4       94       30       63

In [3]: mkdf(5,3,c_idx_names=False,r_idx_names=False,
   ...:                              r_idx_type="dt",c_idx_type="u")
Out[3]: 
           ס4ןדםבדפגמ ץי4ףצ8י1נ2 3מקףיזשד4ר
2000-01-03       R0C0       R0C1       R0C2
2000-01-04       R1C0       R1C1       R1C2
2000-01-05       R2C0       R2C1       R2C2
2000-01-06       R3C0       R3C1       R3C2
2000-01-07       R4C0       R4C1       R4C2

In [4]: mkdf(5,3,r_idx_nlevels=2,r_ndupe_l=[2])
Out[4]: 
C0              C_l0_g0 C_l0_g1 C_l0_g2
R0      R1                             
R_l0_g0 R_l1_g0    R0C0    R0C1    R0C2
        R_l1_g1    R1C0    R1C1    R1C2
R_l0_g1 R_l1_g2    R2C0    R2C1    R2C2
        R_l1_g3    R3C0    R3C1    R3C2
R_l0_g2 R_l1_g4    R4C0    R4C1    R4C2

In [5]: mkdf(5,3,r_idx_nlevels=4,
   ...:                              r_idx_names=["FEE","FI","FO","FAM"],
   ...:                              c_idx_nlevels=2)
Out[5]: 
C0                              C_l0_g0 C_l0_g1 C_l0_g2
C1                              C_l1_g0 C_l1_g1 C_l1_g2
FEE     FI      FO      FAM                            
R_l0_g0 R_l1_g0 R_l2_g0 R_l3_g0    R0C0    R0C1    R0C2
R_l0_g1 R_l1_g1 R_l2_g1 R_l3_g1    R1C0    R1C1    R1C2
R_l0_g2 R_l1_g2 R_l2_g2 R_l3_g2    R2C0    R2C1    R2C2
R_l0_g3 R_l1_g3 R_l2_g3 R_l3_g3    R3C0    R3C1    R3C2
R_l0_g4 R_l1_g4 R_l2_g4 R_l3_g4    R4C0    R4C1    R4C2

```
to_execl xlwt fails on large files
State: closed

``` python
import pandas as pd
df = pd.DataFrame([tuple([i for i in range(200)]) for i in range(10000)], columns=["%s" % i for i in range(200)])
df.to_excel('test.xls')

python testlarge.py                                                                      ⏎
Traceback (most recent call last):
  File "testlarge.py", line 9, in <module>
    df.to_excel('test.xls')
  File "/Volumes/Locodrive/Dev/.virtualenvs/pandasdev/lib/python2.7/site-packages/pandas-0.10.0.dev_67ef67f-py2.7-macosx-10.8-x86_64.egg/pandas/core/frame.py", line 1445, in to_excel
    startrow=startrow, startcol=startcol)
  File "/Volumes/Locodrive/Dev/.virtualenvs/pandasdev/lib/python2.7/site-packages/pandas-0.10.0.dev_67ef67f-py2.7-macosx-10.8-x86_64.egg/pandas/io/parsers.py", line 2068, in write_cells
    self._writecells_xls(cells, sheet_name, startrow, startcol)
  File "/Volumes/Locodrive/Dev/.virtualenvs/pandasdev/lib/python2.7/site-packages/pandas-0.10.0.dev_67ef67f-py2.7-macosx-10.8-x86_64.egg/pandas/io/parsers.py", line 2131, in _writecells_xls
    val, style)
  File "/Volumes/Locodrive/Dev/.virtualenvs/pandasdev/lib/python2.7/site-packages/xlwt/Worksheet.py", line 1032, in write
    self.row(r).write(c, label, style)
  File "/Volumes/Locodrive/Dev/.virtualenvs/pandasdev/lib/python2.7/site-packages/xlwt/Row.py", line 236, in write
    style_index = self.__parent_wb.add_style(style)
  File "/Volumes/Locodrive/Dev/.virtualenvs/pandasdev/lib/python2.7/site-packages/xlwt/Workbook.py", line 303, in add_style
    return self.__styles.add(style)
  File "/Volumes/Locodrive/Dev/.virtualenvs/pandasdev/lib/python2.7/site-packages/xlwt/Style.py", line 90, in add
    return self._add_style(style)[1]
  File "/Volumes/Locodrive/Dev/.virtualenvs/pandasdev/lib/python2.7/site-packages/xlwt/Style.py", line 149, in _add_style
    raise ValueError("More than 4094 XFs (styles)")
V
```


Labels:

Comments:
Test suite fails

my bad sry. all green now

@wesm should we add a check that openpyxl 1.6.1 is installed. 1.5.3 for example does not have the ability to merge cells

That's probably a good idea

FWIW you might consider turning on Travis CI on your fork for continuous integration happiness

thanks will do.  This pull fixes the above issue and should fix #2396 (2nd issue)

Merged, thanks!
repr for wider DataFrames
State: closed

by default no effect,

set_option('print.expand_frame_repr', True) to have it print out wider DataFrames instead of just info repr.

by default uses 80 characters as the width of the line and can be set like set_option('print.line_width', 120)

also adds set_option('test.interactive', False) option for testing purposes


Labels:

Comments:
Travis failures unrelated.
Probably should set expand_frame_repr to True actually.
@wesm any additional thought?

This looks great-- I'll have a look and post comments
Resample interval
State: closed

inference for close and label parameter values


Labels:

Comments:
I agree this is the right thing to do. We'll need to make sure people see the API change!

hmmm do we want to wait for 0.11 and get feedback from the mailing list?

On Dec 3, 2012, at 9:57 AM, Wes McKinney notifications@github.com wrote:

> I agree this is the right thing to do. We'll need to make sure people see the API change!
> 
> —
> Reply to this email directly or view it on GitHub.

We could wait, but I worry about digging the hole deeper

Agreed. Let's include this in the RC and see the reaction. I'll be sure to add more docs and examples in the release notes.

On Dec 3, 2012, at 12:07 PM, Wes McKinney notifications@github.com wrote:

> We could wait, but I worry about digging the hole deeper
> 
> —
> Reply to this email directly or view it on GitHub.
Index format
State: closed

e.g., formatters={'**index**' : <func>}


Labels:

Comments:
Py3 failures are from parser and panel4d tests. 

how do I run py3 tests? 

On Dec 3, 2012, at 9:28 AM, Chang She notifications@github.com wrote:

> Py3 failures are from parser and panel4d tests.
> 
> —
> Reply to this email directly or view it on GitHub.

check on travis-ci.org
You have to sign up and enable to tests, then anything you push to your fork and also PRs will be automatically tested. Github has hooks for it so under PRs there's usually a "merge status" thing (e.g., https://github.com/pydata/pandas/pull/2409)

I have a new PR that may fix the py3 issues (PR #2407)....if this is testing for unicode on the **str** then it would 

ok I pushed your branch into my fork so it'll run on travis

status: https://travis-ci.org/changhiskhan/pandas/builds/3477163

can you try this commit on travis.....I think this should fix it (had to add boolean comparison methods to panel).....
panel > 0 was not doing the right thing (now returns a panel with booleans as it should)
32b37322bd56332e9a99b1cd93524d4cef0411b4

Does it align unequal sized input?

unfortunately no didn't write align 
comparisons accepts scalars and same indexed objs, lower dim objects case remains

new commit...
3818d0132a10db5dc202912450aca28cc7891f1c
BLD: New build matrix for Travis
State: closed

- Moved travis build logic to ci directory.
- Added testing for 3.3  (with numpy 1.7b2).
- Prints version strings for all installed components.
- added FULL_DEPS build for 2.7 and 3.2, which includes
  scipy and matplotlib tests.
- Made the log less verbose.
- Travis now shows an "env" columnn.


Labels:

Comments:
wes, "announcing" "official" 3.3 support in 0.10 might be nice, now that the CI is in place.

I shall do that

still needs fixing.

should be good now.

We can probably drop 3.1 soon (does anyone use it, really?)
Docs for Panel4D/panelnd and Minor Enhancements
State: closed

updated whatsnew/RELEASE
added boolean comparison methods to panel.py (didn't exist before)

all tests pass

need to update the link to dsintro-panel4d section in the whatsnew docs....


Labels:

Comments:
travis build is good - ready to merge

great....fyi...there seems to be a weird bolding in RELEAST.rst (in experimental features)....
Pytables support for hierarchical keys
State: closed

GH #2391

support for keys of the form: foo/bar/bah
includes docs & tests


Labels:

Comments:
ready for merging 

good to go  - I am sure that I will find more bugs...but will add to the issue...

Cool. 

Aside: could you be careful about not merging master into feature branches? it makes the commit history very unclean

I was able to get these commits to cherry-pick cleanly. Merged

yes np

On Dec 1, 2012, at 3:34 PM, Wes McKinney notifications@github.com wrote:

> Cool.
> 
> Aside: could you be careful about not merging master into feature branches? it makes the commit history very unclean
> 
> —
> Reply to this email directly or view it on GitHub.
Dti toperiod tz
State: closed


Labels:

Comments:
Sweet, thanks. I'll get on the Cython refactor today

hmm…failing on travis for 2x but works for 3x, and also works on 2.7.3 locally…

#2380?

Not sure why that would work for me locally then.

I had the same issue, there's a random.seed() call in the code, which when combined with
the travis image makes randint deterministic, and it just so happens that the seed triggers
the problem on travis everytime. that's my take on it, anyway.
I was able to reproduce intermittently locally, but it was somewhat rare.

got it. thanks.
BLD: temporary workaround for travis numpy/py3 woes
State: closed

note that py3 is now tested against numpy recent git, not pypi, just until
the travis people sort things out.


Labels:

Comments:
oh, the [irony](https://travis-ci.org/y-p/pandas/builds/3376439) of it all.

That buglet popped up when I expanded Travis to run _all_ of the pandas unit tests (instead of just `pandas.tests`). I need to find a place to reproduce it and debug.

travis fixed this on their side a while back.
endianess fixes for #2318
State: closed

pending fix verifcation from @juliantaylor


Labels:

Comments:
I started a build, powerpc is big endian
https://launchpad.net/ubuntu/+source/pandas/0.9.1-1ubuntu2
unfortunately some other component broke so its build dependencies are currently uninstallable, I'll retry it when its fixed.

the package also has python3 tests enabled, you might be interested in those results (as they currently fail)
ignore the scpy specfun failures, thats a problem in the scipy package

the ppc build succeeded now, so the fix works fine, thanks

merged. thanks!
BUG: hash random. in 3.3 exposed reliance on dict traversal order. #2331
State: closed

fixes all failing tests mentioned in #2331, perhaps others lurking.


Labels:

Comments:
still failing.

Seems to be passing now. Still the python3 numpy issue though

The fix in groupby.aggregate is incorrect. The test passes, but the problem is still there.
I'm trying out a fix with OrderedDict, (2.7+, will need a shim)

Thanks for the badge btw. :)

Should be good now. There are still two failing tests  on 3.3, that don't look like
a result of hash randomization, so #2331 is still not resolved.

```
FAIL: test_quoting (pandas.io.tests.test_parsers.TestPythonParser)
FAIL: test_cant_compare_tz_naive_w_aware (pandas.tseries.tests.test_timeseries.TestTimestamp
```

Thanks. I'll take the PR and address the failing tests separately
Support for datetime columns with rpy2
State: closed

Check to see if the values we're looking at are numpy datetime64
objects. If they are, we can assume we're looking at a Timestamp object,
which subclasses Python's datetime class, and we return a struct_time
which is understood by rpy2's POSIXct class.

Attempt at a fix for  #2351


Labels:

Comments:
Nice. I left those out when I wrote support for conversions mainly because I did not use any of the datetime features (on R and Python). As one who wrote the code, you get a +1 from me.

Cherry-picked this commit. It'd be great to see some tests for this at some point (separate PR is fine)