Repository: extendr/extendr
Pull request #128 (2021-01-14 23:57:03)

Basic function and struct metadata.
This PR provides metadata that will eventually replace the init__ functions.

The metadata is accessible from Rust or R and can be used to register functions,
build call and class wrappers and generate documentation.

This is the new `cargo expand` on the hello example:

```
#![feature(prelude_import)]
#[prelude_import]
use std::prelude::v1::*;
#[macro_use]
extern crate std;
use extendr_api::*;
fn hello() -> &'static str {
    "hello"
}
#[no_mangle]
#[allow(non_snake_case)]
pub extern "C" fn wrap__hello() -> extendr_api::SEXP {
    unsafe {
        use extendr_api::FromRobj;
        extendr_api::handle_panic("hello paniced.\u{0}", || {
            extendr_api::Robj::from(hello()).get()
        })
    }
}
#[allow(non_snake_case)]
fn init__hello(info: *mut extendr_api::DllInfo, call_methods: &mut Vec<extendr_api::CallMethod>) {
    call_methods.push(extendr_api::CallMethod {
        call_symbol: std::ffi::CString::new("wrap__hello").unwrap(),
        func_ptr: wrap__hello as *const u8,
        num_args: 0i32,
    })
}
#[allow(non_snake_case)]
fn meta__hello(metadata: &mut Vec<extendr_api::metadata::Func>) {
    let mut args = ::alloc::vec::Vec::new();
    metadata.push(extendr_api::metadata::Func {
        doc: "",
        name: "hello",
        args: args,
        return_type: "-> & \'static str",
        func_ptr: wrap__hello as *const u8,
    })
}
#[no_mangle]
#[allow(non_snake_case)]
pub fn get_hello_metadata() -> extendr_api::metadata::Metadata {
    let mut functions = Vec::new();
    let mut structs = Vec::new();
    meta__hello(&mut functions);
    extendr_api::metadata::Metadata {
        doc: "",
        functions,
        structs,
    }
}
#[no_mangle]
#[allow(non_snake_case)]
pub extern "C" fn wrap__get_hello_metadata() -> SEXP {
    unsafe { Robj::from(get_hello_metadata()).get() }
}
#[no_mangle]
#[allow(non_snake_case)]
pub extern "C" fn R_init_hello_extendr(info: *mut extendr_api::DllInfo) {
    let mut call_methods = Vec::new();
    init__hello(info, &mut call_methods);
    unsafe { extendr_api::register_call_methods(info, call_methods.as_ref()) };
}
```


andy-thomason: Next step is to add some tests and remove the old `init_` functions and pass the metadata directly to `register_call_methods`.
clauswilke: Any idea what's going on with the failing Windows test? All the recent PRs have the same problem, so it doesn't seem to be specific to this PR.
andy-thomason: Metadata is now working from the Rust side.

In theory, you should be able to `.Call(wrap__get_my_module_metadata)` from the R side
to get an R named list with functions and impls.

I've tightened up the `#[extendr] impl` rules for now. Only impl <typename> is allowed.

The metadata contains doc strings which can be used to generate R documentation.
These are direct copies of the Rust doc strings and so may need some conversion.

The metadata can also be used to generate class wrappers in whatever flavour we like.

The list of impls contains list of methods which could be bound to an environment.

We could then do:

person <- new("Person")
person$name()

or use s3 classes:

person <- news3("Person")
name(person)

Of course, we would have to generate "new" and "news3".

clauswilke: Just to be clear: I think this is good to have in the code, but by itself it doesn't solve the various problem for rextendr. I would have to go back to writing regexes to parse Rust code to use these functions:

1. `rust_function()`: For this use case, I cannot know the name of the function without somehow parsing the input source code. I also don't want to require the user to add a module call to the code, so I would somehow have to generate it, and again that requires knowing the name of the function.

2. `rust_source()`: For this use case, I could require the user to add a module call, but then I don't know the name of the module without parsing the input source code, or having the user provide the name in a separate argument, which is awkward.

Both of these problems are avoided if the `extendr` macros write out data as they go, as is currently implemented.
clauswilke: Would it be possible to add a function that always has the same name (regardless of the module name) to discover the name(s) of any modules that have been defined? I believe functions in different dynamic libraries can have the same name, and we can call them by explicitly providing the name of the dynamic library in `.Call()`.
andy-thomason: The problem is that the procedural macros are not allowed to have any shared state and **should** not do I/O. We are doing this at the moment, but it is not guaranteed to work forever. For example, the compiler could run the in parallel.

I could make the metadata function just "get_module_metadata" and then you can extract the
metadata with a call to the module.

I'll do an example of this today.

I am especially keen to let us wrap libraries that are not designed for use with R without extra work.

We could also add a "generate_wrappers" function exported to R.

clauswilke: I think this would work. It removes the need for regex parsing in `rust_source()`, and I think it's fine to regex in `rust_function()`, as that's a must more constrained problem.
andy-thomason: The function registration now uses metadata instead of the old "init__" functions.

andy-thomason: We can now generate wrapper code from R and even execute it directly.

```
eval(parse(text = make_extendrtests_wrappers(use_symbols = FALSE)))
```

True use_symbols will generate code for use in libraries, if set to false
it will use strings.

In the example above, we use strings, so we can call the code interactively.

clauswilke: I'm not entirely convinced it makes sense to generate wrapper code from Rust, as it needs to be adjusted to the specific needs of the R use case. E.g., it needs the correct library name when used from rextendr, as is done here:
https://github.com/extendr/rextendr/blob/a4253f33e028d141501f5bb83846dd34233241ed/R/source.R#L168-L173
clauswilke: Sorry, I should have been more precise. This won't work on GH Actions:
https://github.com/extendr/extendr/blob/a77d863907a11ae21e410d18fb713ef1ee5bc03a/tests/extendrtests/src/rust/Cargo.toml#L19-L21
These relative paths don't exist there.

You need to write something like:
```
extendr-api = { git = "https://github.com/extendr/extendr", branch = "metadata"}
extendr-engine = { git = "https://github.com/extendr/extendr", branch = "metadata" }
extendr-macros = { git = "https://github.com/extendr/extendr", branch = "metadata" }
```
See: https://github.com/extendr/extendr/issues/89
andy-thomason: That is a shame. I did this earlier...

clauswilke: Yeah, it basically works now. You just have a few minor issues remaining in the integration tests:

1. Undocumented objects:
https://github.com/extendr/extendr/pull/128/checks?check_run_id=1718200375#step:13:75

2. Testing code somehow ended up in the compiled library:
3. https://github.com/extendr/extendr/pull/128/checks?check_run_id=1718200375#step:13:93
clauswilke: To get rid of the undocumented objects error, you'll have to rebuild the documentation with `devtools::document()`. But you'll also have to document the `use_symbols` argument (see other comment I made).
clauswilke: It still requires running `devtools::document()` in the extendrtests package to rebuild the `.Rd` files. Otherwise R will continue to complain about undocumented symbols.

And some testing code still makes it into the compiled library and needs a conditional compilation guard.


clauswilke: Please see PR #135. Relative paths work after all in extendrtests Cargo.toml, they just need to be a bit different from what one might expect naively.
clauswilke: Once you're ready, if problems with integration tests remain, I'm happy to go in and fix up this branch. Just let me know.
andy-thomason: Thanks, Claus.

Your wisdom is appreciated.


On Tue, Jan 19, 2021 at 5:21 PM Claus Wilke <notifications@github.com>
wrote:

> Once you're ready, if problems with integration tests remain, I'm happy to
> go in and fix up this branch. Just let me know.
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/extendr/extendr/pull/128#issuecomment-762993572>, or
> unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AAL36XDQLPMMFHBE23V5ICTS2W5RRANCNFSM4WDIIR4A>
> .
>

clauswilke: @andy-thomason Could you make the metadata functions public, at least in the crate? See here:
https://github.com/extendr/extendr/issues/140#issuecomment-763559061

cc @dfalbel 

andy-thomason: So nearly  there!

Thanks to @dfalbel I have just one problem to solve with the class wrappers.

The registration settings allow both symbols (packages) and strings (interactive)

MyClass$new() returns a pointer as expected with class "MyClass"

```
> library(extendrtests)
> p <- MyClass$new()
> p
<pointer: 0x559681d60490>
attr(,"class")
[1] "MyClass"
```
The dollar overload:
```
#' @export
`$.MyClass` <- function (self, name) { func <- MyClass[[name]]; environment(func) <- environment(); func }
```

Converts p$a into a modified function that steals "self" from p.

Now `p$a()` calls the function with self set.

I'm guessing copy-on-write de-duplicates the function.

It all works using metadata generated code:

```
MyClass <- new.env(parent = emptyenv())

MyClass$new <- function() .Call(wrap__MyClass__new)
MyClass$set_a <- function(x) .Call(wrap__MyClass__set_a, self, x)
MyClass$a <- function() .Call(wrap__MyClass__a, self)
MyClass$me <- function() .Call(wrap__MyClass__me, self)
#' @export
`$.MyClass` <- function (self, name) { func <- MyClass[[name]]; environment(func) <- environment(); func }
```

The only remaining problem is getting the dollar operator to export.
clauswilke: Does this solve your issue exporting `$`?

https://stackoverflow.com/a/29639694/4975218
andy-thomason: I just had to run the "document" action in R studio and it worked!

Thanks, Claus.
clauswilke: I'm glad things are moving forward. The integration tests almost pass, only a few minor issues remaining.

In particular, if you ran "document" there should be an updated `.Rd` file in this folder:
https://github.com/extendr/extendr/tree/master/tests/extendrtests/man
I think it's missing from the commits.

Patch:
diff --git a/.gitignore b/.gitignore
index 71eca2c56..25c7b19ed 100644
--- a/.gitignore
+++ b/.gitignore
@@ -21,10 +21,15 @@
 
 # CLion support files
 **/.idea
+
+# Misc RStudio & VS Code.
 .vscode/tasks.json
 
 # Rstudio temps.
 .RData
 .Rhistory
+
+# misc
 extendr.Rproj
+tests/extendrtests.Rcheck/
 tests/extendrtests_0.1.0.tar.gz
diff --git a/extendr-api/src/lib.rs b/extendr-api/src/lib.rs
index f052e82dc..9b6cd037a 100644
--- a/extendr-api/src/lib.rs
+++ b/extendr-api/src/lib.rs
@@ -199,6 +199,7 @@ mod robj;
 mod thread_safety;
 mod wrapper;
 mod matrix;
+pub mod metadata;
 
 #[cfg(feature = "ndarray")]
 mod robj_ndarray;
@@ -233,6 +234,8 @@ use libR_sys::*;
 #[doc(hidden)]
 use std::ffi::CString;
 
+pub use metadata::Metadata;
+
 #[doc(hidden)]
 pub struct CallMethod {
     pub call_symbol: std::ffi::CString,
@@ -240,18 +243,33 @@ pub struct CallMethod {
     pub num_args: i32,
 }
 
+unsafe fn make_method_def(cstrings: &mut Vec<std::ffi::CString>, rmethods: &mut Vec<libR_sys::R_CallMethodDef>, func: &metadata::Func, wrapped_name: &str) {
+    cstrings.push(std::ffi::CString::new(wrapped_name).unwrap());
+    rmethods.push(libR_sys::R_CallMethodDef {
+        name: cstrings.last().unwrap().as_ptr(),
+        fun: Some(std::mem::transmute(func.func_ptr)),
+        numArgs: func.args.len() as i32,
+    });
+}
+
+
 // Internal function used to implement the .Call interface.
 // This is called from the code generated by the #[extendr] attribute.
 #[doc(hidden)]
-pub unsafe fn register_call_methods(info: *mut libR_sys::DllInfo, methods: &[CallMethod]) {
-    let mut rmethods: Vec<_> = methods
-        .iter()
-        .map(|m| libR_sys::R_CallMethodDef {
-            name: m.call_symbol.as_ptr(),
-            fun: Some(std::mem::transmute(m.func_ptr)),
-            numArgs: m.num_args,
-        })
-        .collect();
+pub unsafe fn register_call_methods(info: *mut libR_sys::DllInfo, metadata: Metadata) {
+    let mut rmethods = Vec::new();
+    let mut cstrings = Vec::new();
+    for func in metadata.functions {
+        let wrapped_name = format!("wrap__{}", func.name);
+        make_method_def(&mut cstrings, &mut rmethods, &func, wrapped_name.as_str());
+    }
+
+    for imp in metadata.impls {
+        for func in imp.methods {
+            let wrapped_name = format!("wrap__{}__{}", imp.name, func.name);
+            make_method_def(&mut cstrings, &mut rmethods, &func, wrapped_name.as_str());
+        }
+    }
 
     rmethods.push(libR_sys::R_CallMethodDef {
         name: std::ptr::null(),
@@ -266,8 +284,10 @@ pub unsafe fn register_call_methods(info: *mut libR_sys::DllInfo, methods: &[Cal
         std::ptr::null(),
         std::ptr::null(),
     );
+
+    // This seems to allow both symbols and strings,
     libR_sys::R_useDynamicSymbols(info, 0);
-    libR_sys::R_forceSymbols(info, 1);
+    libR_sys::R_forceSymbols(info, 0);
 }
 
 /// Return true if this primitive is NA.
@@ -441,6 +461,7 @@ mod tests {
     }
 
     #[extendr]
+    /// impl comment.
     impl Person {
         fn new() -> Self {
             Self {
@@ -457,7 +478,16 @@ mod tests {
         }
     }
 
+    // see metadata_test for the following comments.
+
+    /// comment #1
+    /// comment #2
+    /**
+        comment #3
+        comment #4
+    **/
     #[extendr]
+    /// aux_func doc comment.
     fn aux_func(_person: &Person) {}
 
     // Macro to generate exports
@@ -622,4 +652,26 @@ mod tests {
         assert_eq!("NA".is_na(), false);
         assert_eq!(na_str().is_na(), true);
     }
+    
+    
+    #[test]
+    fn metadata_test() {
+        test! {
+            // Rust interface.
+            let metadata = get_my_module_metadata();
+            assert_eq!(metadata.functions[0].doc, " comment #1\n comment #2\n\n        comment #3\n        comment #4\n    *\n aux_func doc comment.");
+            assert_eq!(metadata.functions[0].name, "aux_func");
+            assert_eq!(metadata.functions[0].args[0].name, "_person");
+            assert_eq!(metadata.functions[1].name, "get_my_module_metadata");
+            assert_eq!(metadata.impls[0].name, "Person");
+            assert_eq!(metadata.impls[0].methods.len(), 3);
+
+            // R interface
+            let robj = unsafe { new_owned(wrap__get_my_module_metadata()) };
+            let functions = robj.dollar("functions").unwrap();
+            let impls = robj.dollar("impls").unwrap();
+            assert_eq!(functions.len(), 3);
+            assert_eq!(impls.len(), 1);
+        }
+    }
 }
diff --git a/extendr-api/src/metadata.rs b/extendr-api/src/metadata.rs
new file mode 100644
index 000000000..cf8df3f8b
--- /dev/null
+++ b/extendr-api/src/metadata.rs
@@ -0,0 +1,193 @@
+//! Module metadata
+//!
+//! This data is returned by get_module_metadata()
+//! which is generated by [extendr_module!].
+use crate::*;
+use std::io::Write;
+
+/// Metadata function argument.
+#[derive(Debug, PartialEq)]
+pub struct Arg {
+    pub name: &'static str,
+    pub arg_type: &'static str,
+}
+
+/// Metadata function.
+#[derive(Debug, PartialEq)]
+pub struct Func {
+    pub doc: &'static str,
+    pub name: &'static str,
+    pub args: Vec<Arg>,
+    pub return_type: &'static str,
+    pub func_ptr: *const u8,
+    pub hidden: bool,
+}
+
+/// Metadata Impl.
+#[derive(Debug, PartialEq)]
+pub struct Impl {
+    pub doc: &'static str,
+    pub name: &'static str,
+    pub methods: Vec<Func>,
+}
+
+/// Module metadata.
+#[derive(Debug, PartialEq)]
+pub struct Metadata {
+    pub name: &'static str,
+    pub functions: Vec<Func>,
+    pub impls: Vec<Impl>,
+}
+
+impl From<Arg> for Robj {
+    fn from(val: Arg) -> Self {
+        let res: Robj = List(&[r!(val.name), r!(val.arg_type)]).into();
+        res.set_attrib(names_symbol(), r!(["name", "arg_type"]));
+        res
+    }
+}
+
+impl From<Func> for Robj {
+    fn from(val: Func) -> Self {
+        let res: Robj = List(&[
+            r!(val.doc),
+            r!(val.name),
+            r!(List(val.args)),
+            r!(val.return_type),
+            r!(val.hidden),
+        ])
+        .into();
+        res.set_attrib(
+            names_symbol(),
+            r!(["doc", "name", "args", "return.type", "hidden"]),
+        );
+        res
+    }
+}
+
+impl From<Impl> for Robj {
+    fn from(val: Impl) -> Self {
+        let res: Robj = List(&[r!(val.doc), r!(val.name), r!(List(val.methods))]).into();
+        res.set_attrib(names_symbol(), r!(["doc", "name", "methods"]));
+        res
+    }
+}
+
+impl From<Metadata> for Robj {
+    fn from(val: Metadata) -> Self {
+        let res: Robj = List(&[r!(val.name), r!(List(val.functions)), r!(List(val.impls))]).into();
+        res.set_attrib(names_symbol(), r!(["name", "functions", "impls"]));
+        res
+    }
+}
+
+fn write_doc(w: &mut Vec<u8>, doc: &str) -> std::io::Result<()> {
+    if !doc.is_empty() {
+        write!(w, "#'")?;
+        for c in doc.chars() {
+            if c == '\n' {
+                write!(w, "\n#'")?;
+            } else {
+                write!(w, "{}", c)?;
+            }
+        }
+        writeln!(w, "")?;
+    }
+    Ok(())
+}
+
+/// Generate a wrapper for a non-method function.
+fn write_function_wrapper(w: &mut Vec<u8>, func: &Func, package_name: &str, use_symbols: bool) -> std::io::Result<()> {
+    if func.hidden {
+        return Ok(());
+    }
+
+    writeln!(w, "#' @export")?;
+    write_doc(w, func.doc)?;
+
+    let args = func.args.iter().map(|arg| arg.name).collect::<Vec<_>>().join(", ");
+
+    write!(w, "{} <- function({}) .Call(", func.name, args)?;
+
+    if use_symbols {
+        write!(w, "wrap__{}", func.name)?;
+    } else {
+        write!(w, "\"wrap__{}\", PACKAGE=\"{}\"", func.name, package_name)?;
+    }
+
+    if func.args.is_empty() {
+        writeln!(w, ")\n")?;
+    } else {
+        writeln!(w, ", {})\n", args)?;
+    }
+    Ok(())
+}
+
+/// Generate a wrapper for a method.
+fn write_method_wrapper(w: &mut Vec<u8>, func: &Func, package_name: &str, use_symbols: bool, class_name: &str) -> std::io::Result<()> {
+    if func.hidden {
+        return Ok(());
+    }
+
+    let actual_args = func.args.iter().map(|arg| arg.name).collect::<Vec<_>>();
+    let formal_args = if !actual_args.is_empty() && actual_args[0] == "self" {
+        // Skip a leading "self" argument.
+        // This is supplied by the environment.
+        func.args.iter().skip(1).map(|arg| arg.name).collect::<Vec<_>>()
+    } else {
+        actual_args.clone()
+    };
+    
+    let formal_args = formal_args.join(", ");
+    let actual_args = actual_args.join(", ");
+
+    write!(w, "{}${} <- function({}) .Call(", class_name, func.name, formal_args)?;
+
+    if use_symbols {
+        write!(w, "wrap__{}__{}", class_name, func.name)?;
+    } else {
+        write!(w, "\"wrap__{}__{}\", PACKAGE=\"{}\"", class_name, func.name, package_name)?;
+    }
+
+    if actual_args.is_empty() {
+        writeln!(w, ")")?;
+    } else {
+        writeln!(w, ", {})", actual_args)?;
+    }
+
+    Ok(())
+}
+
+impl Metadata {
+    pub fn make_r_wrappers(&self, use_symbols: bool) -> std::io::Result<String> {
+        let mut w = Vec::new();
+
+        writeln!(w, "#' @name {}", self.name)?;
+        writeln!(w, "#' @docType package")?;
+        writeln!(w, "#' @useDynLib {}, .registration = TRUE", self.name)?;
+        writeln!(w, "NULL")?;
+        writeln!(w, "")?;
+        writeln!(w, "# Generated by")?;
+        writeln!(w, "#")?;
+        writeln!(w, "# cat(.Call(\"wrap__make_{}_wrappers\", use_symbols={}))\n", self.name, if use_symbols {"TRUE"} else {"FALSE"})?;
+        
+        for func in &self.functions {
+            write_function_wrapper(&mut w, func, self.name, use_symbols)?;
+        }
+
+        for imp in &self.impls {
+            write_doc(& mut w, imp.doc)?;
+            writeln!(w, "#' @export")?;
+            writeln!(w, "{} <- new.env(parent = emptyenv())\n", imp.name)?;
+                        
+            for func in &imp.methods {
+                // write_doc(& mut w, func.doc)?;
+                write_method_wrapper(&mut w, func, self.name, use_symbols, imp.name)?;
+            }
+
+            writeln!(w, "#' @export")?;
+            writeln!(w, "`$.{}` <- function (self, name) {{ func <- {}[[name]]; environment(func) <- environment(); func }}\n", imp.name, imp.name)?;
+        }
+        unsafe { Ok(String::from_utf8_unchecked(w)) }
+    }
+}
diff --git a/extendr-macros/src/lib.rs b/extendr-macros/src/lib.rs
index 06ce395de..d82d9e2be 100644
--- a/extendr-macros/src/lib.rs
+++ b/extendr-macros/src/lib.rs
@@ -57,13 +57,13 @@
 extern crate proc_macro;
 use proc_macro::{TokenStream};
 use syn::punctuated::Punctuated;
-use syn::{parse_macro_input, parse_quote, Expr, FnArg, Item, ItemFn, ItemImpl, Ident, parse::ParseStream};
+use syn::{parse_macro_input, parse_quote, Expr, FnArg, Item, ItemFn, ItemImpl, Ident, Type, parse::ParseStream};
 use quote::{format_ident, quote};
 use syn::Token;
 
 mod output_r;
 
-const INIT_PREFIX: &str = "init__";
+const META_PREFIX: &str = "meta__";
 const WRAP_PREFIX: &str = "wrap__";
 
 #[derive(Debug)]
@@ -91,6 +91,41 @@ fn translate_formal(input: &FnArg, self_ty: Option<&syn::Type>) -> FnArg {
     }
 }
 
+// Generate code to make a metadata::Arg.
+fn translate_meta_arg(input: &FnArg, self_ty: Option<&syn::Type>) -> Expr {
+    match input {
+        // function argument.
+        FnArg::Typed(ref pattype) => {
+            let pat = pattype.pat.as_ref();
+            let ty = pattype.ty.as_ref();
+            let name_string = quote!{ #pat }.to_string();
+            let type_string = type_name(ty);
+            return parse_quote! {
+                extendr_api::metadata::Arg {
+                    name: #name_string,
+                    arg_type: #type_string
+                }
+            }
+        }
+        // &self
+        FnArg::Receiver(ref reciever) => {
+            if !reciever.attrs.is_empty() || reciever.reference.is_none() {
+                panic!("expected &self or &mut self");
+            }
+            if self_ty.is_none() {
+                panic!("found &self in non-impl function - have you missed the #[extendr] before the impl?");
+            }
+            let type_string = type_name(self_ty.unwrap());
+            return parse_quote! {
+                extendr_api::metadata::Arg {
+                    name: "self",
+                    arg_type: #type_string
+                }
+            }
+        }
+    }
+}
+
 // Convert SEXP arguments into Robj. This maintains the lifetime of references.
 fn translate_to_robj(input: &FnArg) -> syn::Stmt {
     match input {
@@ -153,7 +188,7 @@ fn extendr_function(args: Vec<syn::NestedMeta>, func: ItemFn) -> TokenStream {
     }
 
     let mut wrappers = Vec::new();
-    generate_wrappers(&opts, &mut wrappers, "", &func.sig, None);
+    make_function_wrappers(&opts, &mut wrappers, "", &func.attrs, &func.sig, None);
 
     TokenStream::from(quote!{
         #func
@@ -162,15 +197,103 @@ fn extendr_function(args: Vec<syn::NestedMeta>, func: ItemFn) -> TokenStream {
     })
 }
 
+// Extract doc strings from attributes.
+fn get_doc_string(attrs: &Vec<syn::Attribute>) -> String {
+    let mut res = String::new();
+    for attr in attrs {
+        if let Some(id) = attr.path.get_ident() {
+            if id.to_string() == "doc" {
+                if let Ok(meta) = attr.parse_meta() {
+                    if let syn::Meta::NameValue(nv) = meta {
+                        if let syn::Lit::Str(litstr) = nv.lit {
+                            if !res.is_empty() {
+                                res.extend("\n".chars());
+                            }
+                            res.extend(litstr.value().chars());
+                        }
+                    }
+                }
+            }
+        }
+    }
+    res
+}
+
+fn get_return_type(sig: &syn::Signature) -> String {
+    match &sig.output {
+        syn::ReturnType::Default => "()".into(),
+        syn::ReturnType::Type(_, ref rettype) => type_name( rettype),
+    }
+}
+
+fn mangled_type_name(type_: &Type) -> String {
+    let src = quote!( #type_ ).to_string();
+    let mut res = String::new();
+    for c in src.chars() {
+        if c != ' ' {
+            if c.is_alphanumeric() {
+                res.push(c)
+            } else {
+                let f = format!("_{:02x}", c as u32);
+                res.extend(f.chars());
+            }
+        }
+    }
+    res
+}
+
+
+// Return a simplified type name that will be meaningful to R. Defaults to a digest.
+// For example:
+// & Fred -> Fred
+// * Fred -> Fred
+// && Fred -> Fred
+// Fred<'a> -> Fred
+// &[i32] -> _hex_hex_hex_hex
+// 
+fn type_name(type_: &Type) -> String {
+    match type_ {
+        Type::Path(syn::TypePath { path, .. }) => {
+            if let Some(ident) = path.get_ident() {
+                ident.to_string()
+            } else if path.segments.len() == 1 {
+                let seg = path.segments.clone().into_iter().next().unwrap();
+                seg.ident.to_string()
+            } else {
+                mangled_type_name(type_)
+            }
+        }
+        Type::Group(syn::TypeGroup { elem, .. }) => {
+            type_name(elem)
+        }
+        Type::Reference(syn::TypeReference { elem, .. }) => {
+            type_name(elem)
+        }
+        Type::Paren(syn::TypeParen { elem, .. }) => {
+            type_name(elem)
+        }
+        Type::Ptr(syn::TypePtr { elem, .. }) => {
+            type_name(elem)
+        }
+        _ => {
+            mangled_type_name(type_)
+        }
+    }
+
+}
+
 // Generate wrappers for a specific function.
-fn generate_wrappers(_opts: &ExtendrOptions, wrappers: &mut Vec<ItemFn>, prefix: &str, sig: &syn::Signature, self_ty: Option<&syn::Type>) {
+fn make_function_wrappers(_opts: &ExtendrOptions, wrappers: &mut Vec<ItemFn>, prefix: &str, attrs: &Vec<syn::Attribute>, sig: &syn::Signature, self_ty: Option<&syn::Type>) {
     let func_name = &sig.ident;
 
     let raw_wrap_name = format!("{}{}{}", WRAP_PREFIX, prefix, func_name);
     let wrap_name = format_ident!("{}{}{}", WRAP_PREFIX, prefix, func_name);
-    let init_name = format_ident!("{}{}{}", INIT_PREFIX, prefix, func_name);
+    let meta_name = format_ident!("{}{}{}", META_PREFIX, prefix, func_name);
+
+    let name_str = format!("{}", func_name);
+    let doc_string = get_doc_string(attrs);
+    let return_type_string = get_return_type(&sig);
 
-    let wrap_name_str = format!("{}", wrap_name);
     let panic_str = format!("{} paniced.\0", func_name);
 
     let inputs = &sig.inputs;
@@ -218,12 +341,10 @@ fn generate_wrappers(_opts: &ExtendrOptions, wrappers: &mut Vec<ItemFn>, prefix:
         .filter_map(|input| translate_actual(input))
         .collect();
 
-    // let R_formal_args: Vec<String> = inputs
-    //     .iter()
-    //     .filter_map(|input| translate_R_formal(input))
-    //     .collect();
-
-    let num_args = inputs.len() as i32;
+    let meta_args: Vec<Expr> = inputs
+        .iter()
+        .map(|input| translate_meta_arg(input, self_ty))
+        .collect();
 
     output_r::output_r_wrapper(
         func_name,
@@ -258,27 +379,21 @@ fn generate_wrappers(_opts: &ExtendrOptions, wrappers: &mut Vec<ItemFn>, prefix:
         }
     ));
 
-    // Generate init functions which gather metadata about functions and methods.
-    //
-    // Example:
-    // #[allow(non_snake_case)]
-    // fn init__hello(info: *mut extendr_api::DllInfo, call_methods: &mut Vec<extendr_api::CallMethod>) {
-    //     call_methods.push(extendr_api::CallMethod {
-    //         call_symbol: std::ffi::CString::new("wrap__hello").unwrap(),
-    //         func_ptr: wrap__hello as *const u8,
-    //         num_args: 0i32,
-    //     })
-    // }
+    // Generate a function to push the metadata for a function.
     wrappers.push(parse_quote!(
         #[allow(non_snake_case)]
-        fn #init_name(info: *mut extendr_api::DllInfo, call_methods: &mut Vec<extendr_api::CallMethod>) {
-            call_methods.push(
-                extendr_api::CallMethod {
-                    call_symbol: std::ffi::CString::new(#wrap_name_str).unwrap(),
-                    func_ptr: #wrap_name as * const u8,
-                    num_args: #num_args,
-                }
-            )
+        fn #meta_name(metadata: &mut Vec<extendr_api::metadata::Func>) {
+            let mut args = vec![
+                #( #meta_args, )*
+            ];
+            metadata.push(extendr_api::metadata::Func {
+                doc: #doc_string,
+                name: #name_str,
+                args: args,
+                return_type: #return_type_string,
+                func_ptr: #wrap_name as * const u8,
+                hidden: false,
+            })
         }
     ));
 }
@@ -315,12 +430,38 @@ fn generate_wrappers(_opts: &ExtendrOptions, wrappers: &mut Vec<ItemFn>, prefix:
 /// }
 /// ```
 fn extendr_impl(mut item_impl: ItemImpl) -> TokenStream {
+    // Only `impl name { }` allowed
+    if item_impl.defaultness.is_some() {
+        return quote! { compile_error!("default not allowed in #[extendr] impl"); }.into();
+    }
+
+    if item_impl.unsafety.is_some() {
+        return quote! { compile_error!("unsafe not allowed in #[extendr] impl"); }.into();
+    }
+
+    if item_impl.generics.const_params().count() != 0 {
+        return quote! { compile_error!("const params not allowed in #[extendr] impl"); }.into();
+    }
+
+    if item_impl.generics.type_params().count() != 0 {
+        return quote! { compile_error!("type params not allowed in #[extendr] impl"); }.into();
+    }
+
+    // if item_impl.generics.lifetimes().count() != 0 {
+    //     return quote! { compile_error!("lifetime params not allowed in #[extendr] impl"); }.into();
+    // }
+
+    if item_impl.generics.where_clause.is_some() {
+        return quote! { compile_error!("where clause not allowed in #[extendr] impl"); }.into();
+    }
+
     let opts = ExtendrOptions {};
     let self_ty = item_impl.self_ty.as_ref();
-    let self_ty_name = quote! {#self_ty}.to_string();
+    let self_ty_name = type_name(&self_ty);
     let prefix = format!("{}__", self_ty_name);
-    let mut method_init_names = Vec::new();
-    let mut wrappers = Vec::new();
+    let mut method_meta_names = Vec::new();
+    let doc_string = get_doc_string(&item_impl.attrs);
+    
 
     // Generate wrappers for methods.
     // eg.
@@ -334,14 +475,15 @@ fn extendr_impl(mut item_impl: ItemImpl) -> TokenStream {
     //     }
     // }
     // ```
+    let mut wrappers: Vec<ItemFn> = Vec::new();
     for impl_item in &mut item_impl.items {
         if let syn::ImplItem::Method(ref mut method) = impl_item {
-            method_init_names.push(format_ident!("{}{}__{}", INIT_PREFIX, self_ty_name, method.sig.ident));
-            generate_wrappers(&opts, &mut wrappers, prefix.as_str(), &method.sig, Some(self_ty));
+            method_meta_names.push(format_ident!("{}{}__{}", META_PREFIX, self_ty_name, method.sig.ident));
+            make_function_wrappers(&opts, &mut wrappers, prefix.as_str(), &method.attrs, &method.sig, Some(self_ty));
         }
     }
 
-    let init_name = format_ident!("{}{}", INIT_PREFIX, self_ty_name);
+    let meta_name = format_ident!("{}{}", META_PREFIX, self_ty_name);
 
     let finalizer_name = format_ident!("__finalize__{}", self_ty_name);
 
@@ -380,6 +522,20 @@ fn extendr_impl(mut item_impl: ItemImpl) -> TokenStream {
                 unsafe {
                     let ptr = Box::into_raw(Box::new(value));
                     let res = Robj::makeExternalPtr(ptr, Robj::from(#self_ty_name), Robj::from(()));
+                    res.set_attrib(class_symbol(), #self_ty_name);
+                    res.registerCFinalizer(Some(#finalizer_name));
+                    res
+                }
+            }
+        }
+
+        // Output conversion function for this type.
+        impl<'a> From<&'a #self_ty> for Robj {
+            fn from(value: &'a #self_ty) -> Self {
+                unsafe {
+                    let ptr = Box::into_raw(Box::new(value));
+                    let res = Robj::makeExternalPtr(ptr, Robj::from(#self_ty_name), Robj::from(()));
+                    res.set_attrib(class_symbol(), #self_ty_name);
                     res.registerCFinalizer(Some(#finalizer_name));
                     res
                 }
@@ -398,19 +554,15 @@ fn extendr_impl(mut item_impl: ItemImpl) -> TokenStream {
             }
         }
 
-        // Init function to call the method inits for this type.
-        // Example:
-        // ```ignore
-        // #[allow(non_snake_case)]
-        // fn init__Person(info: *mut extendr_api::DllInfo, call_methods: &mut Vec<extendr_api::CallMethod>) {
-        //     init__Person__new(info, call_methods);
-        //     init__Person__set_name(info, call_methods);
-        //     init__Person__name(info, call_methods);
-        // }
-        // ```
         #[allow(non_snake_case)]
-        fn #init_name(info: *mut extendr_api::DllInfo, call_methods: &mut Vec<extendr_api::CallMethod>) {
-            #( #method_init_names(info, call_methods); )*
+        fn #meta_name(impls: &mut Vec<extendr_api::metadata::Impl>) {
+            let mut methods = Vec::new();
+            #( #method_meta_names(&mut methods); )*
+            impls.push(extendr_api::metadata::Impl {
+                doc: #doc_string,
+                name: #self_ty_name,
+                methods,
+            });
         }
     });
 
@@ -436,7 +588,7 @@ pub fn extendr(attr: TokenStream, item: TokenStream) -> TokenStream {
 struct Module {
     modname: Option<Ident>,
     fnnames: Vec<Ident>,
-    implnames: Vec<Ident>,
+    implnames: Vec<Type>,
 }
 
 // Custom parser for the module.
@@ -499,19 +651,77 @@ pub fn extendr_module(item: TokenStream) -> TokenStream {
     let module = parse_macro_input!(item as Module);
     let Module {modname, fnnames, implnames} = module;
     let modname = modname.unwrap();
+    let modname_string = modname.to_string();
     let module_init_name = format_ident!("R_init_{}_extendr", modname);
 
-    let fninitnames = fnnames.iter().map(|id| format_ident!("{}{}", INIT_PREFIX, id));
-    let implinitnames = implnames.iter().map(|id| format_ident!("{}{}", INIT_PREFIX, id));
+    let module_metadata_name = format_ident!("get_{}_metadata", modname);
+    let module_metadata_name_string = module_metadata_name.to_string();
+    let wrap_module_metadata_name = format_ident!("{}get_{}_metadata", WRAP_PREFIX, modname);
+
+    let make_module_wrappers_name = format_ident!("make_{}_wrappers", modname);
+    let make_module_wrappers_name_string = make_module_wrappers_name.to_string();
+    let wrap_make_module_wrappers = format_ident!("{}make_{}_wrappers", WRAP_PREFIX, modname);
+
+    let fnmetanames = fnnames.iter().map(|id| format_ident!("{}{}", META_PREFIX, id));
+    let implmetanames = implnames.iter().map(|id| format_ident!("{}{}", META_PREFIX, type_name(id)));
 
     TokenStream::from(quote!{
+        #[no_mangle]
+        #[allow(non_snake_case)]
+        pub fn #module_metadata_name() -> extendr_api::metadata::Metadata {
+            let mut functions = Vec::new();
+            let mut impls = Vec::new();
+            #( #fnmetanames(&mut functions); )*
+            #( #implmetanames(&mut impls); )*
+
+            // Add this function to the list, but set hidden: true.
+            functions.push(extendr_api::metadata::Func {
+                doc: "Metadata access function.",
+                name: #module_metadata_name_string,
+                args: Vec::new(),
+                return_type: "Metadata",
+                func_ptr: #wrap_module_metadata_name as * const u8,
+                hidden: true,
+            });
+
+            // Add this function to the list, but set hidden: true.
+            functions.push(extendr_api::metadata::Func {
+                doc: "Wrapper generator.",
+                name: #make_module_wrappers_name_string,
+                args: vec![extendr_api::metadata::Arg { name: "use_symbols", arg_type: "bool" }],
+                return_type: "String",
+                func_ptr: #wrap_make_module_wrappers as * const u8,
+                hidden: true,
+            });
+
+            extendr_api::metadata::Metadata {
+                name: #modname_string,
+                functions,
+                impls,
+            }
+        }
+
+        #[no_mangle]
+        #[allow(non_snake_case)]
+        pub extern "C" fn #wrap_module_metadata_name() -> SEXP {
+            unsafe { extendr_api::Robj::from(#module_metadata_name()).get() }
+        }
+
+        #[no_mangle]
+        #[allow(non_snake_case)]
+        pub extern "C" fn #wrap_make_module_wrappers(use_symbols_sexp: SEXP) -> SEXP {
+            unsafe {
+                let robj = new_borrowed(use_symbols_sexp);
+                let use_symbols : bool = <bool>::from_robj(&robj).unwrap();
+    
+                extendr_api::Robj::from(#module_metadata_name().make_r_wrappers(use_symbols).unwrap()).get()
+            }
+        }
+
         #[no_mangle]
         #[allow(non_snake_case)]
         pub extern "C" fn #module_init_name(info: * mut extendr_api::DllInfo) {
-            let mut call_methods = Vec::new();
-            #( #fninitnames(info, &mut call_methods); )*
-            #( #implinitnames(info, &mut call_methods); )*
-            unsafe { extendr_api::register_call_methods(info, call_methods.as_ref()) };
+            unsafe { extendr_api::register_call_methods(info, #module_metadata_name()) };
         }
     })
 }


Test patch:
diff --git a/tests/extendrtests/NAMESPACE b/tests/extendrtests/NAMESPACE
index f804b7780..f6b774e17 100644
--- a/tests/extendrtests/NAMESPACE
+++ b/tests/extendrtests/NAMESPACE
@@ -1,4 +1,3 @@
 # Generated by roxygen2: do not edit by hand
 
-export(hello_world)
 useDynLib(extendrtests, .registration = TRUE)
diff --git a/tests/extendrtests/R/extendrtests.R b/tests/extendrtests/R/extendrtests.R
index c64328b6e..e38c8d25d 100644
--- a/tests/extendrtests/R/extendrtests.R
+++ b/tests/extendrtests/R/extendrtests.R
@@ -1,7 +1,44 @@
-#' Integration tests for calling extendr from an R package.
-#'
-#' Integration tests for calling extendr from an R package.
 #' @name extendrtests
 #' @docType package
 #' @useDynLib extendrtests, .registration = TRUE
-NULL
\ No newline at end of file
+NULL
+
+# Generated by
+#
+# cat(.Call("wrap__make_extendrtests_wrappers", use_symbols=TRUE))
+
+# '@export
+#' Say hello
+hello_world <- function() .Call(wrap__hello_world)
+
+# '@export
+#' convert a double scalar to itself
+#' @param x a number
+double_scalar <- function(x) .Call(wrap__double_scalar, x)
+
+# '@export
+#' convert an int scalar to itself
+#' @param x a number
+int_scalar <- function(x) .Call(wrap__int_scalar, x)
+
+# '@export
+#' convert a bool scalar to itself
+#' @param x a number
+bool_scalar <- function(x) .Call(wrap__bool_scalar, x)
+
+# '@export
+#' convert a string to itself
+#' @param x a number
+char_scalar <- function(x) .Call(wrap__char_scalar, x)
+
+#' Class for testing
+# '@export
+MyClass <- new.env(parent = emptyenv())
+
+MyClass$new <- function() .Call(wrap__MyClass__new)
+MyClass$set_a <- function(x) .Call(wrap__MyClass__set_a, self, x)
+MyClass$a <- function() .Call(wrap__MyClass__a, self)
+MyClass$me <- function() .Call(wrap__MyClass__me, self)
+# '@export
+`$.MyClass` <- function (self, name) { func <- MyClass[[name]]; environment(func) <- environment(); func }
+
diff --git a/tests/extendrtests/R/wrappers.R b/tests/extendrtests/R/wrappers.R
index a8e0d0cc5..08c42466a 100644
--- a/tests/extendrtests/R/wrappers.R
+++ b/tests/extendrtests/R/wrappers.R
@@ -1,8 +1,46 @@
 #' Wrappers for Rust test functions.
 #' 
-#' Wrappers for Rust test functions.
 #' @rdname wrappers
 #' @export
 hello_world <- function() {
-  .Call("wrap__hello_world")
+  .Call(wrap__hello_world)
 }
+
+#' Get module metadata as a list.
+#' 
+#' @rdname wrappers
+#' @export
+get_extendrtests_metadata <- function() .Call(wrap__get_extendrtests_metadata)
+
+#' Build wrappers 
+#' 
+#' @param use_symbols use symbols instead of strings.
+#' @rdname wrappers
+#' @export
+make_extendrtests_wrappers <- function(use_symbols=TRUE) .Call(wrap__make_extendrtests_wrappers, use_symbols)
+
+
+hello_world <- function() .Call(wrap__hello_world)
+
+#' convert a double scalar to itself
+#' @param x a number
+double_scalar <- function(x) .Call(wrap__double_scalar, x)
+
+#' convert an int scalar to itself
+#' @param x a number
+int_scalar <- function(x) .Call(wrap__int_scalar, x)
+
+bool_scalar <- function(x) .Call(wrap__bool_scalar, x)
+
+char_scalar <- function(x) .Call(wrap__char_scalar, x)
+
+#' @export
+MyClass <- new.env(parent = emptyenv())
+
+MyClass$new <- function() .Call(wrap__MyClass__new)
+MyClass$set_a <- function(x) .Call(wrap__MyClass__set_a, self, x)
+MyClass$a <- function() .Call(wrap__MyClass__a, self)
+MyClass$me <- function() .Call(wrap__MyClass__me, self)
+#' @export
+`$.MyClass` <- function (self, name) { func <- MyClass[[name]]; environment(func) <- environment(); func }
+
diff --git a/tests/extendrtests/src/rust/Cargo.toml b/tests/extendrtests/src/rust/Cargo.toml
index a0b35022d..e7cee5540 100644
--- a/tests/extendrtests/src/rust/Cargo.toml
+++ b/tests/extendrtests/src/rust/Cargo.toml
@@ -16,7 +16,13 @@ crate-type = ["staticlib"]
 extendr-api = "*"
 
 [patch.crates-io]
-extendr-api = { path = "../../../../../../../../extendr-api" }
-extendr-engine = { path = "../../../../../../../../extendr-engine" }
-extendr-macros = { path = "../../../../../../../../extendr-macros" }
-#libR-sys = { git = "https://github.com/extendr/libR-sys" }
+
+# development verion
+#extendr-api = { path = "../../../../extendr-api/"}
+#extendr-engine = { path = "../../../../extendr-engine/"}
+#extendr-macros = { path = "../../../../extendr-macros/"}
+
+# R CMD check
+extendr-api = { git = "https://github.com/extendr/extendr", branch = "metadata"}
+extendr-engine = { git = "https://github.com/extendr/extendr", branch = "metadata" }
+extendr-macros = { git = "https://github.com/extendr/extendr", branch = "metadata" }
diff --git a/tests/extendrtests/src/rust/src/lib.rs b/tests/extendrtests/src/rust/src/lib.rs
index be75ee6a3..ca719a1ab 100644
--- a/tests/extendrtests/src/rust/src/lib.rs
+++ b/tests/extendrtests/src/rust/src/lib.rs
@@ -1,20 +1,63 @@
 use extendr_api::*;
 
+/// Say hello
 #[extendr]
 fn hello_world() -> &'static str {
     "Hello world!"
 }
 
 // functions to test input/output conversion
+
+/// convert a double scalar to itself
+/// @param x a number
 #[extendr]
 fn double_scalar(x: f64) -> f64 { x }
+
+/// convert an int scalar to itself
+/// @param x a number
 #[extendr]
 fn int_scalar(x: i32) -> i32 { x }
+
+/// convert a bool scalar to itself
+/// @param x a number
 #[extendr]
 fn bool_scalar(x: bool) -> bool { x }
+
+/// convert a string to itself
+/// @param x a number
 #[extendr]
 fn char_scalar(x: String) -> String { x }
 
+/// Class for testing
+#[derive(Default, Debug)]
+struct MyClass {
+    a: i32,
+}
+
+/// Class for testing
+#[extendr]
+impl MyClass {
+    /// Method for making new object.
+    fn new() -> Self {
+        Self { a: 0 }
+    }
+    
+    /// Method for setting stuff.
+    /// @param x a number
+    fn set_a(& mut self, x: i32) {
+        self.a = x;
+    }
+    
+    /// Method for getting stuff.
+    fn a(&self) -> i32 {
+        self.a
+    }
+    
+    /// Method for getting one's self.
+    fn me(&self) -> &Self {
+        self
+    }
+}
 
 // Macro to generate exports
 extendr_module! {
@@ -25,4 +68,6 @@ extendr_module! {
     fn int_scalar;
     fn bool_scalar;
     fn char_scalar;
+    
+    impl MyClass;
 }