Repository: clap-rs/clap (language: rust)

Issue:
[Derive] Way to parse long_help from doc comments excluding 
### Please complete the following tasks

- [X] I have searched the [discussions](https://github.com/clap-rs/clap/discussions)
- [X] I have searched the [open](https://github.com/clap-rs/clap/issues) and [rejected](https://github.com/clap-rs/clap/issues?q=is%3Aissue+label%3AS-wont-fix+is%3Aclosed) issues

### Clap Version

4.0.18

### Describe your use case

In fd, there are many options where we want to have different help messages for the "short" and "long" help. When clap parses the doc comments, it interprets the first paragraph as the short help, and the entire doc comments as the long help. This means that the long help includes the short help text as its first paragraph. There isn't a way to have a different first paragraph for the long help output than for the short help output if you use doc comments. 

Relatedly, if you have an option with `hide_short_help = true`, then you don't need any short help from the doc comment. However, by default clap strips a trailing period off the end of the first paragraph, which may not be desirable. The only way to prevent that is to use `verbatim_doc_comment`, but then you lose the automatic wrapping and whitespace cleanup. 

I'd like a way to specify the long_help in the documentation comment without including the short help.

### Describe the solution you'd like

Have an attribute on the Parser struct and/or individual attributes to specify that the first paragraph in the doc comment should be passed to help and the remainder, excluding the first paragraph, should be used for the long_help. If the doc comment begins with two blank lines, then use the full comment as the long_help, and don't set a `help`:


```rust
#[derive(Parser)] 
#[command(name = "mycommand", distinct_doc_comment_help = true)]
pub struct Opts {
    /// This is the short help
    /// 
    /// This is the long help. 
    #[arg(short = 'a')]
    pub a: bool,
    ///
    ///
    /// This option only has a long_help.
    ///
    /// And it includes both paragraphs. The first period isn't 
    /// removed.
    #[arg(long, hide_short_help = true)]
    pub hidden_option: bool,
    /// This is in both.
    ///
    /// This option includes the first paragraph in both the short and long help messages.
    #[arg(short = 'c', distinct_doc_comment_help = false)]
    pub c: bool, 
}
```

The name of the attribute could probably be improved.


### Alternatives, if applicable

1. Change the behavior to never include the first paragraph in the long_help. This would break backwards compatibility, and might be inconvenient in the case you do want the same paragraph in both.
2.  Have some kind of special syntax in the doc comment itself to indicate where to split between the short help and the long help. Possibly with some way to indicate you want to include the short help at the beginning of the long help.
3. Just stop removing the trailing period when generating `long_help`. This helps in the case where `hide_short_help` is true, but not in the more general case where you want different text for the first paragraph of the long and short help.
4. Do nothing. It is possible to work around this by passing strings to `help` and `long_help`, although that is certainly less convenient than being able to use doc coumments (this is what fd currently does).

### Additional Context

Something that occurred to me, although I haven't needed it yet is, what if you want multiple paragraphs in the short help? Currently I don't think you can do that with the doc comment parser, and you have to manually use the `help=` attribute. 


Discussion / hints:
> In fd, there are many options where we want to have different help messages for the "short" and "long" help. When clap parses the doc comments, it interprets the first paragraph as the short help, and the entire doc comments as the long help. This means that the long help includes the short help text as its first paragraph. There isn't a way to have a different first paragraph for the long help output than for the short help output if you use doc comments.

Why not specify `#[arg(help = "")]` and leave the doc comment only for the long help?

> Relatedly, if you have an option with hide_short_help = true, then you don't need any short help from the doc comment. However, by default clap strips a trailing period off the end of the first paragraph, which may not be desirable. The only way to prevent that is to use verbatim_doc_comment, but then you lose the automatic wrapping and whitespace cleanup.

If short help is hidden, why does it matter what clap sets it to?
> Why not specify #[arg(help = "")] and leave the doc comment only for the long help?

I thought I tried that, and it didn't set the long_help at all. I could be misremembering though. 

That does still have the problem that it strips the "." off the end of the first paragraph though.

> If short help is hidden, why does it matter what clap sets it to?

It doesn't. In this case my main issue is that it still strips the "." off the end of the first paragraph (if it is there). 
so if I'm understanding correctly, clap is modifying the long help?
Yeah, I just retried using `#[arg(help = "...")]` witha doc comment, like this:

```
    /// Show search results from files and directories that would otherwise be
    /// ignored by '.gitignore' files. The flag can be overridden with --ignore-vcs.
    #[arg(
        long,
        hide_short_help = true,
        help = "Do not respect .gitignore files",
    )]
    pub no_ignore_vcs: bool,
```


and for the long help it shows "Do not respect .gitignore files" instead of what is in the doc comment.  (if I rem ove `hide_short_help = true` it shows the same thing in the short help).

And if I remove the `help = ` line, then the long help shows:

```
      --no-ignore-vcs
          Show search results from files and directories that would otherwise be ignored by
          '.gitignore' files. The flag can be overridden with --ignore-vcs
```

Note that while there is a period at the end of the text in the doc comment, it is missing in the help text. The [documentation for derive](https://docs.rs/clap/latest/clap/_derive/index.html#pre-processing) does mention this:

> If the first paragraph ends in exactly one period, remove the trailing period (i.e. strip trailing periods but not trailing ellipses).

If specifying a `help = ` didn't impact the use of the doc comment for the long_help, and the period at the end of the first paragraph isn't removed, then that would be a pretty good solution to this as well. 

Turns out that `help =` is not impacting the long help but is overriding the short help set by the doc comment.  As the doc comment looks like short help only (single "line"), we never set `long_help`.  Long help is visually different than short help and not every application wants long help, so we key off of the presence of `long_help` for rendering `--help` differently than `-h`.
hmm, what if `long_help` was allowed without an argument in the attribute, which would force the passing the doc comment to the long_help method, even if it is a single paragraph. And then do something about the period truncation (either don't remove it if it is long help, have a separate option for it, or just get rid of it altogether). 
I think that makes sense.  I would lean towards treating whatever is present as long help which means no period removal

Fix patch:
diff --git a/clap_derive/src/attr.rs b/clap_derive/src/attr.rs
index f97ba7d0a7f..5ea49a1dd6b 100644
--- a/clap_derive/src/attr.rs
+++ b/clap_derive/src/attr.rs
@@ -105,6 +105,8 @@ impl Parse for ClapAttr {
             "external_subcommand" => Some(MagicAttrName::ExternalSubcommand),
             "verbatim_doc_comment" => Some(MagicAttrName::VerbatimDocComment),
             "about" => Some(MagicAttrName::About),
+            "long_about" => Some(MagicAttrName::LongAbout),
+            "long_help" => Some(MagicAttrName::LongHelp),
             "author" => Some(MagicAttrName::Author),
             "version" => Some(MagicAttrName::Version),
             _ => None,
@@ -160,6 +162,8 @@ pub enum MagicAttrName {
     VerbatimDocComment,
     ExternalSubcommand,
     About,
+    LongAbout,
+    LongHelp,
     Author,
     Version,
     RenameAllEnv,
diff --git a/clap_derive/src/item.rs b/clap_derive/src/item.rs
index 94f1226787c..a068e36011a 100644
--- a/clap_derive/src/item.rs
+++ b/clap_derive/src/item.rs
@@ -19,13 +19,10 @@ use proc_macro2::{self, Span, TokenStream};
 use proc_macro_error::abort;
 use quote::{format_ident, quote, quote_spanned, ToTokens};
 use syn::DeriveInput;
-use syn::{
-    self, ext::IdentExt, spanned::Spanned, Attribute, Field, Ident, LitStr, MetaNameValue, Type,
-    Variant,
-};
+use syn::{self, ext::IdentExt, spanned::Spanned, Attribute, Field, Ident, LitStr, Type, Variant};
 
 use crate::attr::*;
-use crate::utils::{inner_type, is_simple_ty, process_doc_comment, Sp, Ty};
+use crate::utils::{extract_doc_comment, format_doc_comment, inner_type, is_simple_ty, Sp, Ty};
 
 /// Default casing style for generated arguments.
 pub const DEFAULT_CASING: CasingStyle = CasingStyle::Kebab;
@@ -46,6 +43,7 @@ pub struct Item {
     value_parser: Option<ValueParser>,
     action: Option<Action>,
     verbatim_doc_comment: bool,
+    force_long_help: bool,
     next_display_order: Option<Method>,
     next_help_heading: Option<Method>,
     is_enum: bool,
@@ -67,7 +65,7 @@ impl Item {
         let parsed_attrs = ClapAttr::parse_all(attrs);
         res.infer_kind(&parsed_attrs);
         res.push_attrs(&parsed_attrs);
-        res.push_doc_comment(attrs, "about", true);
+        res.push_doc_comment(attrs, "about", Some("long_about"));
 
         res
     }
@@ -84,7 +82,7 @@ impl Item {
         let parsed_attrs = ClapAttr::parse_all(attrs);
         res.infer_kind(&parsed_attrs);
         res.push_attrs(&parsed_attrs);
-        res.push_doc_comment(attrs, "about", true);
+        res.push_doc_comment(attrs, "about", Some("long_about"));
 
         res
     }
@@ -144,7 +142,7 @@ impl Item {
         res.infer_kind(&parsed_attrs);
         res.push_attrs(&parsed_attrs);
         if matches!(&*res.kind, Kind::Command(_) | Kind::Subcommand(_)) {
-            res.push_doc_comment(&variant.attrs, "about", true);
+            res.push_doc_comment(&variant.attrs, "about", Some("long_about"));
         }
 
         match &*res.kind {
@@ -189,7 +187,7 @@ impl Item {
         res.infer_kind(&parsed_attrs);
         res.push_attrs(&parsed_attrs);
         if matches!(&*res.kind, Kind::Value) {
-            res.push_doc_comment(&variant.attrs, "help", false);
+            res.push_doc_comment(&variant.attrs, "help", None);
         }
 
         res
@@ -217,7 +215,7 @@ impl Item {
         res.infer_kind(&parsed_attrs);
         res.push_attrs(&parsed_attrs);
         if matches!(&*res.kind, Kind::Arg(_)) {
-            res.push_doc_comment(&field.attrs, "help", true);
+            res.push_doc_comment(&field.attrs, "help", Some("long_help"));
         }
 
         match &*res.kind {
@@ -269,6 +267,7 @@ impl Item {
             value_parser: None,
             action: None,
             verbatim_doc_comment: false,
+            force_long_help: false,
             next_display_order: None,
             next_help_heading: None,
             is_enum: false,
@@ -507,6 +506,18 @@ impl Item {
                     }
                 }
 
+                Some(MagicAttrName::LongAbout) if attr.value.is_none() => {
+                    assert_attr_kind(attr, &[AttrKind::Command]);
+
+                    self.force_long_help = true;
+                }
+
+                Some(MagicAttrName::LongHelp) if attr.value.is_none() => {
+                    assert_attr_kind(attr, &[AttrKind::Arg]);
+
+                    self.force_long_help = true;
+                }
+
                 Some(MagicAttrName::Author) if attr.value.is_none() => {
                     assert_attr_kind(attr, &[AttrKind::Command]);
 
@@ -823,6 +834,8 @@ impl Item {
                 | Some(MagicAttrName::Long)
                 | Some(MagicAttrName::Env)
                 | Some(MagicAttrName::About)
+                | Some(MagicAttrName::LongAbout)
+                | Some(MagicAttrName::LongHelp)
                 | Some(MagicAttrName::Author)
                 | Some(MagicAttrName::Version)
                  => {
@@ -872,28 +885,13 @@ impl Item {
         }
     }
 
-    fn push_doc_comment(&mut self, attrs: &[Attribute], name: &str, supports_long_help: bool) {
-        use syn::Lit::*;
-        use syn::Meta::*;
-
-        let comment_parts: Vec<_> = attrs
-            .iter()
-            .filter(|attr| attr.path.is_ident("doc"))
-            .filter_map(|attr| {
-                if let Ok(NameValue(MetaNameValue { lit: Str(s), .. })) = attr.parse_meta() {
-                    Some(s.value())
-                } else {
-                    // non #[doc = "..."] attributes are not our concern
-                    // we leave them for rustc to handle
-                    None
-                }
-            })
-            .collect();
+    fn push_doc_comment(&mut self, attrs: &[Attribute], short_name: &str, long_name: Option<&str>) {
+        let lines = extract_doc_comment(attrs);
 
-        if let Some((short_help, long_help)) =
-            process_doc_comment(&comment_parts, !self.verbatim_doc_comment)
-        {
-            let short_name = format_ident!("{}", name);
+        if !lines.is_empty() {
+            let (short_help, long_help) =
+                format_doc_comment(&lines, !self.verbatim_doc_comment, self.force_long_help);
+            let short_name = format_ident!("{}", short_name);
             let short = Method::new(
                 short_name,
                 short_help
@@ -901,8 +899,8 @@ impl Item {
                     .unwrap_or_else(|| quote!(None)),
             );
             self.doc_comment.push(short);
-            if supports_long_help {
-                let long_name = format_ident!("long_{}", name);
+            if let Some(long_name) = long_name {
+                let long_name = format_ident!("{}", long_name);
                 let long = Method::new(
                     long_name,
                     long_help
diff --git a/clap_derive/src/utils/doc_comments.rs b/clap_derive/src/utils/doc_comments.rs
index 591246ff2ef..5183b6b25e1 100644
--- a/clap_derive/src/utils/doc_comments.rs
+++ b/clap_derive/src/utils/doc_comments.rs
@@ -5,36 +5,56 @@
 
 use std::iter;
 
-pub fn process_doc_comment(
-    lines: &[String],
-    preprocess: bool,
-) -> Option<(Option<String>, Option<String>)> {
+pub fn extract_doc_comment(attrs: &[syn::Attribute]) -> Vec<String> {
+    use syn::Lit::*;
+    use syn::Meta::*;
+    use syn::MetaNameValue;
+
     // multiline comments (`/** ... */`) may have LFs (`\n`) in them,
     // we need to split so we could handle the lines correctly
     //
     // we also need to remove leading and trailing blank lines
-    let mut lines: Vec<&str> = lines
+    let mut lines: Vec<_> = attrs
         .iter()
+        .filter(|attr| attr.path.is_ident("doc"))
+        .filter_map(|attr| {
+            if let Ok(NameValue(MetaNameValue { lit: Str(s), .. })) = attr.parse_meta() {
+                Some(s.value())
+            } else {
+                // non #[doc = "..."] attributes are not our concern
+                // we leave them for rustc to handle
+                None
+            }
+        })
         .skip_while(|s| is_blank(s))
-        .flat_map(|s| s.split('\n'))
+        .flat_map(|s| {
+            let lines = s
+                .split('\n')
+                .map(|s| {
+                    // remove one leading space no matter what
+                    let s = s.strip_prefix(' ').unwrap_or(s);
+                    s.to_owned()
+                })
+                .collect::<Vec<_>>();
+            lines
+        })
         .collect();
 
     while let Some(true) = lines.last().map(|s| is_blank(s)) {
         lines.pop();
     }
 
-    if lines.is_empty() {
-        return None;
-    }
-
-    // remove one leading space no matter what
-    for line in lines.iter_mut() {
-        *line = line.strip_prefix(' ').unwrap_or(line);
-    }
+    lines
+}
 
+pub fn format_doc_comment(
+    lines: &[String],
+    preprocess: bool,
+    force_long: bool,
+) -> (Option<String>, Option<String>) {
     if let Some(first_blank) = lines.iter().position(|s| is_blank(s)) {
         let (short, long) = if preprocess {
-            let paragraphs = split_paragraphs(&lines);
+            let paragraphs = split_paragraphs(lines);
             let short = paragraphs[0].clone();
             let long = paragraphs.join("\n\n");
             (remove_period(short), long)
@@ -44,20 +64,24 @@ pub fn process_doc_comment(
             (short, long)
         };
 
-        Some((Some(short), Some(long)))
+        (Some(short), Some(long))
     } else {
-        let short = if preprocess {
-            let s = merge_lines(&lines);
-            remove_period(s)
+        let (short, long) = if preprocess {
+            let short = merge_lines(lines);
+            let long = force_long.then(|| short.clone());
+            let short = remove_period(short);
+            (short, long)
         } else {
-            lines.join("\n")
+            let short = lines.join("\n");
+            let long = force_long.then(|| short.clone());
+            (short, long)
         };
 
-        Some((Some(short), None))
+        (Some(short), long)
     }
 }
 
-fn split_paragraphs(lines: &[&str]) -> Vec<String> {
+fn split_paragraphs(lines: &[String]) -> Vec<String> {
     let mut last_line = 0;
     iter::from_fn(|| {
         let slice = &lines[last_line..];
@@ -91,6 +115,10 @@ fn is_blank(s: &str) -> bool {
     s.trim().is_empty()
 }
 
-fn merge_lines(lines: &[&str]) -> String {
-    lines.iter().map(|s| s.trim()).collect::<Vec<_>>().join(" ")
+fn merge_lines(lines: impl IntoIterator<Item = impl AsRef<str>>) -> String {
+    lines
+        .into_iter()
+        .map(|s| s.as_ref().trim().to_owned())
+        .collect::<Vec<_>>()
+        .join(" ")
 }
diff --git a/clap_derive/src/utils/mod.rs b/clap_derive/src/utils/mod.rs
index 77a467c754e..9f8b6f380ab 100644
--- a/clap_derive/src/utils/mod.rs
+++ b/clap_derive/src/utils/mod.rs
@@ -2,8 +2,10 @@ mod doc_comments;
 mod spanned;
 mod ty;
 
+pub use doc_comments::extract_doc_comment;
+pub use doc_comments::format_doc_comment;
+
 pub use self::{
-    doc_comments::process_doc_comment,
     spanned::Sp,
     ty::{inner_type, is_simple_ty, sub_type, subty_if_name, Ty},
 };
diff --git a/src/_derive/mod.rs b/src/_derive/mod.rs
index f2955e50be2..f5da23632ba 100644
--- a/src/_derive/mod.rs
+++ b/src/_derive/mod.rs
@@ -154,8 +154,9 @@
 //!     - **TIP:** When a doc comment is also present, you most likely want to add
 //!       `#[arg(long_about = None)]` to clear the doc comment so only [`about`][crate::Command::about]
 //!       gets shown with both `-h` and `--help`.
-//! - `long_about = <expr>`: [`Command::long_about`][crate::Command::long_about]
+//! - `long_about[ = <expr>]`: [`Command::long_about`][crate::Command::long_about]
 //!   - When not present: [Doc comment](#doc-comments) if there is a blank line, else nothing
+//!   - When present without a value: [Doc comment](#doc-comments)
 //! - `verbatim_doc_comment`: Minimizes pre-processing when converting doc comments to [`about`][crate::Command::about] / [`long_about`][crate::Command::long_about]
 //! - `next_display_order`: [`Command::next_display_order`][crate::Command::next_display_order]
 //! - `next_help_heading`: [`Command::next_help_heading`][crate::Command::next_help_heading]
@@ -200,8 +201,9 @@
 //!   - When not present: will auto-select an action based on the field type
 //! - `help = <expr>`: [`Arg::help`][crate::Arg::help]
 //!   - When not present: [Doc comment summary](#doc-comments)
-//! - `long_help = <expr>`: [`Arg::long_help`][crate::Arg::long_help]
+//! - `long_help[ = <expr>]`: [`Arg::long_help`][crate::Arg::long_help]
 //!   - When not present: [Doc comment](#doc-comments) if there is a blank line, else nothing
+//!   - When present without a value: [Doc comment](#doc-comments)
 //! - `verbatim_doc_comment`: Minimizes pre-processing when converting doc comments to [`help`][crate::Arg::help] / [`long_help`][crate::Arg::long_help]
 //! - `short [= <char>]`: [`Arg::short`][crate::Arg::short]
 //!   - When not present: no short set


Test patch:
diff --git a/tests/derive/doc_comments_help.rs b/tests/derive/doc_comments_help.rs
index 8aa4fa1e567..f8b3743a940 100644
--- a/tests/derive/doc_comments_help.rs
+++ b/tests/derive/doc_comments_help.rs
@@ -288,3 +288,18 @@ Options:
 ";
     utils::assert_output::<Cmd>("cmd --help", OUTPUT, false);
 }
+
+#[test]
+fn force_long_help() {
+    /// Lorem ipsum
+    #[derive(Parser, PartialEq, Debug)]
+    struct LoremIpsum {
+        /// Fooify a bar
+        /// and a baz.
+        #[arg(short, long, long_help)]
+        foo: bool,
+    }
+
+    let help = utils::get_long_help::<LoremIpsum>();
+    assert!(help.contains("Fooify a bar and a baz."));
+}