DOC: example of pandas to R transfer of DataFrame using HDF5 file
State: closed

closes #9636 
I had some trouble getting the HDF5 import of DataFrame objects to work on the R side (pandas is fine). I've added an example to the documentation with a simple load function in R which assembles an R data.frame from the corresponding *_items and *_values nodes in the HDF5 file.

NOTE: Any improvements to the R code are welcome, I'm just starting to work with R. :-)


Labels:
name: Docs
name: IO HDF5
description: read_hdf, HDFStore

Comments:
Thanks for this. I couldn't actually test, since apparently `rhdf5` isn't available for my version of R (3.1.1). 

I wonder if the [odo](http://github.com/ContinuumIO/odo) project is stable enough where we should start linking to them (in the docs, not for code) for this kind of thing. I think `h5py` HDF5 flies are in more of a vanilla format, making it easier to read elsewhere.

`rhdf5` should be compatible with version 3.1.1, I'm using 3.1.2 myself. Note, that it is a package by Bioconductor and not available on CRAN (see [package site](http://www.bioconductor.org/packages/release/bioc/html/rhdf5.html)). For installation you have to use:

``` R
source("http://bioconductor.org/biocLite.R")
biocLite("rhdf5")
```

@joschkazj this looks nice. 2 things.

1) pls add a note (in v0.16.0.txt) referencing this new doc section (you may need to add a refernce label under the heading, e.g. in the `:ref:` part)
2) I think a link / short explanation in this section: http://pandas-docs.github.io/pandas-docs-travis/comparison_with_r.html would also be a useful reference

> @joschkazj this looks nice.

Thank you!

> 1) pls add a note (in v0.16.0.txt) referencing this new doc section (you may need to add a refernce label under the heading, e.g. in the :ref: part)

Done.

> 2) I think a link / short explanation in this section: http://pandas-docs.github.io/pandas-docs-travis/comparison_with_r.html would also be a useful reference

I've added a short paragraph about data transfer of `DataFrame` objects in the introduction. I think an extra section is too much, considering this part is about a comparison to R.

P.S. I'm looking into a rebase to avoid merge conflicts. Haven't done that before, so it could take a while.

I hope everything is still working, the rebase using Mercurial and hg-git was a bit confusing.

merged via https://github.com/pydata/pandas/commit/dab318f166d73bec13ee3d3f07e361ff5b0b1e5c

@joschkazj thanks for this!

the docs will be built (give it 2-3 hours), and will be updated here: http://pandas-docs.github.io/pandas-docs-travis/

pls review and issue a followup pr if necessary

You're welcome. Thank you for this great package, which keeps on adding more and more useful features, since I've first used it (v0.11 I think).
I've built the documentation locally to check the output, the only change was the new position in the "What's New" document, which had to be adapted after the reorganization (4004d79cdb87be9e54699be7394fde5c06734746).

> pls review and issue a followup pr if necessary

The documentation built by Travis looks ok to me, so I'm closing this PR.

I am not sure it is the best place to write this but I was not able to contact you @joschkazj. 
I have the following warning with your script that was not able to recognize the column name: 
`h5read for type 'VLEN' not yet implemented. Values replaced by NA's`
I cananot reproduce with simple data and cannot trace the origin of the error (I don't understand what VLEN means here)

@benjello `VLEN` refers to "variable length" variables in HDF5. Typically, this is used for (unicode) strings. If you don't nee unicode, it should be possible to write ASCII instead. Someone who knows the pandas/pytables specific HDF5 conventions can hopefully tell you if that's possible here (see [this page](http://h5py.readthedocs.org/en/latest/strings.html) for more details on how you do that with h5py).

@shoyer Thanks a lot ! Everything is ok after renaming every column_name to str(column_name).

@benjello I'm curious as to how you did that. I'm getting the same warning where all my strings are replaced with N/As. What's column_name in your context? I'm trying to read a hdf5 file saved from pandas's .to_hdf() method, where some columns have strings as their values.
IIRC column_name is the name of a DataFrame column. My dataframe columns had unicode strings. Going back to ASCII solved the problem.  
@benjello I've tried converting my column names and values into ASCII (if they weren't already). Still facing the same problem. I'm on python 2 with pandas version 0.20.1. If it's not too much to ask, I'd like to know how did you convert them back to ASCII.
@jfong361 : I converted the column names using `str(column_name)`
@benjello Hm, that doesn't seem to work for me. It's probably some other problem in my case. Anyhow, thanks for your help!