Repository: pacak/bpaf
Pull request #232 (2023-07-04 16:30:40)

feat: markdown rendering
I noticed that now the documentation can be emitted in HTML, so I thought also to have a markdown format.

The code is mostly a copy-paste of the render of the HTML, but I'm happy to refactor it with some direction. Ideally, we could move it to a `markdown.rs` file.


Leaving it as a draft at the moment, asking the following questions:
- I couldn't find a proper way to test the doc gen, so I just copied the manpage test for a POC
- the environment variable is rendered strangely, probably because it has different metadata; how do you suggest tackling this part?
- what kind of test cases would you like to add?

pacak: # simple

I am a program and I do things

**Usage**: **`simple`** [**`-d`**] **`--user`**=_`USER`_

Sometimes they even work.

**Available options:**
- **`-d`**, **`--kraken`**: Unleash the kraken
- **`    --user`**=_`USER`_: Log in as this user
- : Uses environment variable **`USER`**
- **`-h`**, **`--help`**: Prints help information
pacak: Environment variables are prefixed with empty  `Block::ItemTerm` block, we probably need to handle this as a special case by skipping `-` and `:`. Need to figure out how to make it so it looks good...
pacak: Heh, github seems to be very unhappy with html I'm making even if rustdoc is totally okay with it. I'll try to get this pull request to work correctly.

https://github.com/pacak/cargo-supply-chain
ematipico: > Heh, github seems to be very unhappy with html I'm making even if rustdoc is totally okay with it. I'll try to get this pull request to work correctly.
> 
> https://github.com/pacak/cargo-supply-chain <- generated docs example

Yeah, I tried to use it inside Rome's website and it didn't work very well.

Ideally Markdown would be perfect because, all our documentation is written in Markdown 
pacak: I pushed some changes here https://github.com/pacak/bpaf/tree/md-rendering
can you check if it works for Rome website? Seems to be doing the right thing for github.
ematipico: Amazing! The end result is very well done!

Here's a small bug. No need to escape the characters in `.md`.
 
![Screenshot 2023-07-06 at 14 11 51](https://github.com/pacak/bpaf/assets/602478/d95ff320-56e9-4458-bb27-b36077374951)

A small enhancement I could suggest is to add a link to the command header even when we list the commands:

![Screenshot 2023-07-06 at 14 13 17](https://github.com/pacak/bpaf/assets/602478/e4bdcc78-8224-4776-a647-7f5573500df7)


The rest is great, thank you very much for your hard work 🙇‍♂️ 
pacak: > Here's a small bug. No need to escape the characters in .md.

Fixed, can you check?


> A small enhancement I could suggest is to add a link to the command header even when we list the commands:

Hmm.... Is is actually hard :) Well, not hard but requires a bunch of changes. There should be a list of all available commands generated somewhere on top. Rendering mechanism for all the options and commands is shared across all the representations (console, html, markdown) and mostly unaware of markdown links. Tracking that will add some binary overhead which I'm trying to minimize. I'll keep it in mind if I decide to do more refactors though.
ematipico: > Hmm.... Is is actually hard :) Well, not hard but requires a bunch of changes. There should be a list of all available commands generated somewhere on top. Rendering mechanism for all the options and commands is shared across all the representations (console, html, markdown) and mostly unaware of markdown links. Tracking that will add some binary overhead which I'm trying to minimize. I'll keep it in mind if I decide to do more refactors though.

Don't worry. It's just a nice to have. It's a great feature as is! 

>Fixed, can you check?

Yes, that's fixed
pacak: Do you want to pick the commits I made into this branch so we can merge it? I'll deal with the rest of the docs separately.

Patch:
diff --git a/.gitignore b/.gitignore
index 02cbd19..dd35385 100644
--- a/.gitignore
+++ b/.gitignore
@@ -5,3 +5,4 @@ Cargo.lock
 /tarp
 /dotfiles
 /legacy/target
+.idea
diff --git a/src/buffer/html.rs b/src/buffer/html.rs
index 712edcd..47791ff 100644
--- a/src/buffer/html.rs
+++ b/src/buffer/html.rs
@@ -27,13 +27,13 @@ fn collect_html(app: String, meta: &Meta, info: &Info) -> Doc {
         // TODO - this defines forward references to sections which are rendered differently
         // between html and markdown and never used in console...
         for section in &sections {
-            buf.token(Token::BlockStart(Block::Block));
+            buf.token(Token::BlockStart(Block::ItemBody));
             buf.text(&format!(
                 "* [`{}`↴](#{})",
                 section.path.join(" "),
-                section.path.join("-").to_lowercase(),
+                section.path.join("-").to_lowercase().replace(' ', "-"),
             ));
-            buf.token(Token::BlockEnd(Block::Block));
+            buf.token(Token::BlockEnd(Block::ItemBody));
         }
     }
 
@@ -59,6 +59,11 @@ impl<T> OptionParser<T> {
     pub fn render_html(&self, app: impl Into<String>) -> String {
         collect_html(app.into(), &self.inner.meta(), &self.info).render_html(true, false)
     }
+
+    /// Render command line documentation for the app into Markdown
+    pub fn render_markdown(&self, app: impl Into<String>) -> String {
+        collect_html(app.into(), &self.inner.meta(), &self.info).render_markdown(true)
+    }
 }
 
 #[derive(Copy, Clone, Default)]
@@ -116,13 +121,51 @@ fn change_style(res: &mut String, cur: &mut Styles, new: Styles) {
     *cur = new;
 }
 
+fn change_to_markdown_style(res: &mut String, cur: &mut Styles, new: Styles) {
+    if cur.mono {
+        res.push('`');
+    }
+
+    if cur.bold {
+        res.push_str("**");
+    }
+    if cur.italic {
+        res.push('_');
+    }
+    if new.italic {
+        res.push('_');
+    }
+    if new.bold {
+        res.push_str("**");
+    }
+
+    if new.mono {
+        res.push('`');
+    }
+    *cur = new;
+}
+
 /// Make it so new text is separated by an empty line
-fn blank_line(res: &mut String) {
+fn blank_html_line(res: &mut String) {
     if !(res.is_empty() || res.ends_with("<br>\n")) {
         res.push_str("<br>\n");
     }
 }
 
+/// Make it so new text is separated by an empty line
+fn blank_markdown_line(res: &mut String) {
+    if !(res.is_empty() || res.ends_with("\n\n")) {
+        res.push_str("\n\n");
+    }
+}
+
+/// Make it so new text is separated by an empty line
+fn new_markdown_line(res: &mut String) {
+    if !(res.is_empty() || res.ends_with('\n')) {
+        res.push('\n');
+    }
+}
+
 const CSS: &str = "
 <style>
 div.bpaf-doc {
@@ -146,8 +189,14 @@ impl Doc {
         let mut byte_pos = 0;
         let mut cur_style = Styles::default();
 
+        // skip tracks text paragraphs, paragraphs starting from the section
+        // one are only shown when full is set to true
         let mut skip = Skip::default();
+
+        // stack keeps track of the AST tree, mostly to be able to tell
+        // if we are rendering definition list or item list
         let mut stack = Vec::new();
+
         for token in self.tokens.iter().copied() {
             match token {
                 Token::Text { bytes, style } => {
@@ -182,7 +231,7 @@ impl Doc {
                     change_style(&mut res, &mut cur_style, Styles::default());
                     match b {
                         Block::Header => {
-                            blank_line(&mut res);
+                            blank_html_line(&mut res);
                             res.push_str("# ");
                         }
                         Block::Section2 => {
@@ -216,7 +265,7 @@ impl Doc {
                     stack.pop();
                     match b {
                         Block::Header => {
-                            blank_line(&mut res);
+                            blank_html_line(&mut res);
                         }
                         Block::Section2 => {
                             res.push_str("</div>");
@@ -250,6 +299,110 @@ impl Doc {
         }
         res
     }
+
+    /// Render doc into markdown document, used by documentation sample generator
+    #[must_use]
+    pub fn render_markdown(&self, full: bool) -> String {
+        let mut res = String::new();
+        let mut byte_pos = 0;
+        let mut cur_style = Styles::default();
+
+        let mut skip = Skip::default();
+        let mut stack = Vec::new();
+        let mut empty_term = false;
+        for (ix, token) in self.tokens.iter().copied().enumerate() {
+            match token {
+                Token::Text { bytes, style } => {
+                    let input = &self.payload[byte_pos..byte_pos + bytes];
+                    byte_pos += bytes;
+                    if skip.enabled() {
+                        continue;
+                    }
+
+                    change_to_markdown_style(&mut res, &mut cur_style, Styles::from(style));
+
+                    for chunk in split(input) {
+                        match chunk {
+                            Chunk::Raw(input, _) => res.push_str(input),
+                            Chunk::Paragraph => {
+                                if full {
+                                    res.push('\n');
+                                } else {
+                                    skip.enable();
+                                    break;
+                                }
+                            }
+                            Chunk::LineBreak => res.push('\n'),
+                        }
+                    }
+                }
+                Token::BlockStart(b) => {
+                    change_to_markdown_style(&mut res, &mut cur_style, Styles::default());
+                    match b {
+                        Block::Header => {
+                            blank_markdown_line(&mut res);
+                            res.push_str("# ");
+                        }
+                        Block::Section2 => {
+                            res.push_str("");
+                        }
+                        Block::ItemTerm => {
+                            empty_term = matches!(
+                                self.tokens.get(ix + 1),
+                                Some(Token::BlockEnd(Block::ItemTerm))
+                            );
+                            res.push_str(if empty_term { "  " } else { "- " });
+                        }
+                        Block::ItemBody => {
+                            new_markdown_line(&mut res);
+                        }
+                        Block::DefinitionList => {
+                            res.push_str("");
+                        }
+                        Block::Block => {
+                            res.push('\n');
+                        }
+                        Block::Meta => todo!(),
+                        Block::Section3 => res.push_str("### "),
+                        Block::TermRef => {}
+                        Block::InlineBlock => {
+                            skip.push();
+                        }
+                    }
+                    stack.push(b);
+                }
+                Token::BlockEnd(b) => {
+                    change_to_markdown_style(&mut res, &mut cur_style, Styles::default());
+                    stack.pop();
+                    match b {
+                        Block::Header => res.push('\n'),
+                        Block::Section2 => {
+                            res.push('\n');
+                        }
+
+                        Block::InlineBlock => {
+                            skip.pop();
+                        }
+                        Block::ItemTerm => res.push_str(if empty_term { " " } else { " &mdash; " }),
+                        Block::ItemBody => {
+                            if stack.last().copied() == Some(Block::DefinitionList) {
+                                res.push('\n');
+                            }
+                        }
+                        Block::DefinitionList => res.push('\n'),
+                        Block::Block => {
+                            res.push('\n');
+                        }
+                        Block::Meta => todo!(),
+                        Block::TermRef => {}
+                        Block::Section3 => res.push('\n'),
+                    }
+                }
+            }
+        }
+        change_to_markdown_style(&mut res, &mut cur_style, Styles::default());
+        res
+    }
 }
 
 #[cfg(test)]


Test patch:
diff --git a/tests/markdown.md b/tests/markdown.md
new file mode 100644
index 0000000..645ea1a
--- /dev/null
+++ b/tests/markdown.md
@@ -0,0 +1,21 @@
+# simple
+
+I am a program and I do things
+
+**Usage**: **`simple`** [**`-d`**] **`--user`**=_`USER`_
+
+Sometimes they even work.
+
+**Available options:**
+- **`-d`**, **`--kraken`** &mdash; 
+Unleash the kraken
+- **`    --user`**=_`USER`_ &mdash; 
+Log in as this user
+   
+Uses environment variable **`USER`**
+- **`-h`**, **`--help`** &mdash; 
+Prints help information
+
+
+
+Beware `-d`, dragons be here
diff --git a/tests/markdown.rs b/tests/markdown.rs
new file mode 100644
index 0000000..a4d6a68
--- /dev/null
+++ b/tests/markdown.rs
@@ -0,0 +1,44 @@
+use bpaf::*;
+
+fn write_updated(new_val: &str, path: impl AsRef<std::path::Path>) -> std::io::Result<bool> {
+    use std::io::Read;
+    use std::io::Seek;
+    let mut file = std::fs::OpenOptions::new()
+        .write(true)
+        .read(true)
+        .create(true)
+        .open(path)?;
+    let mut current_val = String::new();
+    file.read_to_string(&mut current_val)?;
+    if current_val != new_val {
+        file.set_len(0)?;
+        file.seek(std::io::SeekFrom::Start(0))?;
+        std::io::Write::write_all(&mut file, new_val.as_bytes())?;
+        Ok(false)
+    } else {
+        Ok(true)
+    }
+}
+
+#[test]
+fn simple() {
+    let kraken = short('d')
+        .long("kraken")
+        .help("Unleash the kraken")
+        .switch();
+
+    let user = long("user")
+        .env("USER")
+        .help("Log in as this user")
+        .argument::<String>("USER");
+
+    let options = construct!(kraken, user)
+        .to_options()
+        .descr("I am a program and I do things")
+        .header("Sometimes they even work.")
+        .footer("Beware `-d`, dragons be here");
+    let roff = options.render_markdown("simple");
+
+    #[cfg(unix)]
+    assert!(write_updated(&roff, "tests/markdown.md").unwrap());
+}