Repository: dtolnay/syn
Pull request #1231 (2022-10-11 16:21:18)

Support full let-else statement parsing
Currently let-else statements are parsed as `ExprVerbatim` (i.e. they are not parsed internally).

*let-else* statements are in beta now, so they will most probably appear in stable in November. Therefore, I think, now is the right moment for adding full parsing implementation.

This PR adds such parsing implementation to syn.
Fixes: https://github.com/dtolnay/syn/issues/1159

deg4uss3r: Hey everyone! Now that let/else is stable I'd love to try to get this PR in, @wkordalski is there anything I can help you with to get CI to pass? 

@dtolnay does this PR pass your initial review? Is there another direction you'd like to take (or another PR that I missed in the queue)?
wkordalski: Hi!
I miss let-else support very much and wait for adding its support to `syn`.
I need some information on the workflow of adding such features to `syn`, so that tests will pass on CI.
And I hope that someone (@dtolnay?) can give me some hints.

## The CI problem

One of the tests is to run `syn`'s internal codegen.
The codegen at the end of its work formats code using `prettyplease`, which internally uses `syn` to parse code to be formatted.
In the failing syn test (that runs syn's internal codegen) we compile `prettyplease` crate [with overwritten `syn` version](https://github.com/wkordalski/syn/blob/ddcaf7341c2e813949dcce0140f451e1860aecc5/codegen/Cargo.toml#L27) (i.e. `syn` from this branch (with let-else support), not the recently *released* version). However current version of `prettyplease` is not compatible with "`syn` from this branch" and it fails to compile.

We could update `prettyplease` to be compatible with "`syn` from this branch", but then `prettyplease` won't be compatible with currently-released version of `syn`, and `prettyplease` CI will fail.

## Simplest solution

I think the simplest solution is:
1. Add to `prettyplease` information that it supports `syn` up to version "currently-released-version" (via Cargo.toml)
2. Remove the overwrite of `syn` crate in codegen (simply remove [this line](https://github.com/wkordalski/syn/blob/ddcaf7341c2e813949dcce0140f451e1860aecc5/codegen/Cargo.toml#L27)). Then codegen from "`syn` from this branch" will happily use `prettyplease` that will use "currently-released-version" of `syn`.
3. Now all tests from `syn` CI pass, and we can merge this PR and publish (possibly as prerelease version).
4. Update `prettyplease` to use and compile happily with the prerelease version of `syn` (also modify the version requiremest in Cargo.toml of `prettyplease`)
5. The CI of `prettyplease` should pass, so we can publish it.
6. Finally, we can publish the prerelease version of `syn` as normal release.

### Some other thoughts

- We can skip the publishing prerelease version step by stating in `prettyplease`'s Cargo.toml to use `syn` from this branch, but I feel this won't be the better option.
- Should we make minor/major version bump as we change publicly-accessible data structures, what is a breaking change?
- The "simplest solution" is quite complicated and I would love to hear from maintainers of this crate if there is another solution that is simpler.
wkordalski: Ok... Thanks to https://github.com/dtolnay/syn/commit/ec9de4642d70105d2a73f1eb6654405ce5ff8be5 all tests pass and we can move forward.
I would appreciate some review, esp. of the changes to data structures.
After merging this I can also update `prettyplease`.

Patch:
diff --git a/examples/trace-var/trace-var/src/lib.rs b/examples/trace-var/trace-var/src/lib.rs
index 67059416..3934f52c 100644
--- a/examples/trace-var/trace-var/src/lib.rs
+++ b/examples/trace-var/trace-var/src/lib.rs
@@ -84,7 +84,7 @@ impl Args {
     ///     let VAR = { let VAR = INIT; println!("VAR = {:?}", VAR); VAR };
     fn let_and_print(&mut self, local: Local) -> Stmt {
         let Local { pat, init, .. } = local;
-        let init = self.fold_expr(*init.unwrap().1);
+        let init = self.fold_expr(*init.unwrap().expr);
         let ident = match pat {
             Pat::Ident(ref p) => &p.ident,
             _ => unreachable!(),
diff --git a/src/gen/clone.rs b/src/gen/clone.rs
index 3a65617f..56fe7acc 100644
--- a/src/gen/clone.rs
+++ b/src/gen/clone.rs
@@ -1356,6 +1356,17 @@ impl Clone for Local {
         }
     }
 }
+#[cfg(feature = "full")]
+#[cfg_attr(doc_cfg, doc(cfg(feature = "clone-impls")))]
+impl Clone for LocalInitializer {
+    fn clone(&self) -> Self {
+        LocalInitializer {
+            eq_token: self.eq_token.clone(),
+            expr: self.expr.clone(),
+            else_block: self.else_block.clone(),
+        }
+    }
+}
 #[cfg(any(feature = "derive", feature = "full"))]
 #[cfg_attr(doc_cfg, doc(cfg(feature = "clone-impls")))]
 impl Clone for Macro {
diff --git a/src/gen/debug.rs b/src/gen/debug.rs
index ea444236..e9511129 100644
--- a/src/gen/debug.rs
+++ b/src/gen/debug.rs
@@ -1866,6 +1866,17 @@ impl Debug for Local {
         formatter.finish()
     }
 }
+#[cfg(feature = "full")]
+#[cfg_attr(doc_cfg, doc(cfg(feature = "extra-traits")))]
+impl Debug for LocalInitializer {
+    fn fmt(&self, formatter: &mut fmt::Formatter) -> fmt::Result {
+        let mut formatter = formatter.debug_struct("LocalInitializer");
+        formatter.field("eq_token", &self.eq_token);
+        formatter.field("expr", &self.expr);
+        formatter.field("else_block", &self.else_block);
+        formatter.finish()
+    }
+}
 #[cfg(any(feature = "derive", feature = "full"))]
 #[cfg_attr(doc_cfg, doc(cfg(feature = "extra-traits")))]
 impl Debug for Macro {
diff --git a/src/gen/eq.rs b/src/gen/eq.rs
index 30abd41e..f6004223 100644
--- a/src/gen/eq.rs
+++ b/src/gen/eq.rs
@@ -1287,6 +1287,16 @@ impl PartialEq for Local {
         self.attrs == other.attrs && self.pat == other.pat && self.init == other.init
     }
 }
+#[cfg(feature = "full")]
+#[cfg_attr(doc_cfg, doc(cfg(feature = "extra-traits")))]
+impl Eq for LocalInitializer {}
+#[cfg(feature = "full")]
+#[cfg_attr(doc_cfg, doc(cfg(feature = "extra-traits")))]
+impl PartialEq for LocalInitializer {
+    fn eq(&self, other: &Self) -> bool {
+        self.expr == other.expr && self.else_block == other.else_block
+    }
+}
 #[cfg(any(feature = "derive", feature = "full"))]
 #[cfg_attr(doc_cfg, doc(cfg(feature = "extra-traits")))]
 impl Eq for Macro {}
diff --git a/src/gen/fold.rs b/src/gen/fold.rs
index 9d68c0ca..6756111c 100644
--- a/src/gen/fold.rs
+++ b/src/gen/fold.rs
@@ -465,6 +465,10 @@ pub trait Fold {
     fn fold_local(&mut self, i: Local) -> Local {
         fold_local(self, i)
     }
+    #[cfg(feature = "full")]
+    fn fold_local_initializer(&mut self, i: LocalInitializer) -> LocalInitializer {
+        fold_local_initializer(self, i)
+    }
     #[cfg(any(feature = "derive", feature = "full"))]
     fn fold_macro(&mut self, i: Macro) -> Macro {
         fold_macro(self, i)
@@ -2348,12 +2352,23 @@ where
         attrs: FoldHelper::lift(node.attrs, |it| f.fold_attribute(it)),
         let_token: Token![let](tokens_helper(f, &node.let_token.span)),
         pat: f.fold_pat(node.pat),
-        init: (node.init)
+        init: (node.init).map(|it| f.fold_local_initializer(it)),
+        semi_token: Token![;](tokens_helper(f, &node.semi_token.spans)),
+    }
+}
+#[cfg(feature = "full")]
+pub fn fold_local_initializer<F>(f: &mut F, node: LocalInitializer) -> LocalInitializer
+where
+    F: Fold + ?Sized,
+{
+    LocalInitializer {
+        eq_token: Token![=](tokens_helper(f, &node.eq_token.spans)),
+        expr: Box::new(f.fold_expr(*node.expr)),
+        else_block: (node.else_block)
             .map(|it| (
-                Token![=](tokens_helper(f, &(it).0.spans)),
-                Box::new(f.fold_expr(*(it).1)),
+                Token![else](tokens_helper(f, &(it).0.span)),
+                Box::new(f.fold_expr_block(*(it).1)),
             )),
-        semi_token: Token![;](tokens_helper(f, &node.semi_token.spans)),
     }
 }
 #[cfg(any(feature = "derive", feature = "full"))]
diff --git a/src/gen/hash.rs b/src/gen/hash.rs
index 0bb2d1dc..20a65240 100644
--- a/src/gen/hash.rs
+++ b/src/gen/hash.rs
@@ -1734,6 +1734,17 @@ impl Hash for Local {
         self.init.hash(state);
     }
 }
+#[cfg(feature = "full")]
+#[cfg_attr(doc_cfg, doc(cfg(feature = "extra-traits")))]
+impl Hash for LocalInitializer {
+    fn hash<H>(&self, state: &mut H)
+    where
+        H: Hasher,
+    {
+        self.expr.hash(state);
+        self.else_block.hash(state);
+    }
+}
 #[cfg(any(feature = "derive", feature = "full"))]
 #[cfg_attr(doc_cfg, doc(cfg(feature = "extra-traits")))]
 impl Hash for Macro {
diff --git a/src/gen/visit.rs b/src/gen/visit.rs
index b2167031..205cfde4 100644
--- a/src/gen/visit.rs
+++ b/src/gen/visit.rs
@@ -464,6 +464,10 @@ pub trait Visit<'ast> {
     fn visit_local(&mut self, i: &'ast Local) {
         visit_local(self, i);
     }
+    #[cfg(feature = "full")]
+    fn visit_local_initializer(&mut self, i: &'ast LocalInitializer) {
+        visit_local_initializer(self, i);
+    }
     #[cfg(any(feature = "derive", feature = "full"))]
     fn visit_macro(&mut self, i: &'ast Macro) {
         visit_macro(self, i);
@@ -2610,11 +2614,22 @@ where
     tokens_helper(v, &node.let_token.span);
     v.visit_pat(&node.pat);
     if let Some(it) = &node.init {
-        tokens_helper(v, &(it).0.spans);
-        v.visit_expr(&*(it).1);
+        v.visit_local_initializer(it);
     }
     tokens_helper(v, &node.semi_token.spans);
 }
+#[cfg(feature = "full")]
+pub fn visit_local_initializer<'ast, V>(v: &mut V, node: &'ast LocalInitializer)
+where
+    V: Visit<'ast> + ?Sized,
+{
+    tokens_helper(v, &node.eq_token.spans);
+    v.visit_expr(&*node.expr);
+    if let Some(it) = &node.else_block {
+        tokens_helper(v, &(it).0.span);
+        v.visit_expr_block(&*(it).1);
+    }
+}
 #[cfg(any(feature = "derive", feature = "full"))]
 pub fn visit_macro<'ast, V>(v: &mut V, node: &'ast Macro)
 where
diff --git a/src/gen/visit_mut.rs b/src/gen/visit_mut.rs
index 1bb2ea15..5ef4e112 100644
--- a/src/gen/visit_mut.rs
+++ b/src/gen/visit_mut.rs
@@ -465,6 +465,10 @@ pub trait VisitMut {
     fn visit_local_mut(&mut self, i: &mut Local) {
         visit_local_mut(self, i);
     }
+    #[cfg(feature = "full")]
+    fn visit_local_initializer_mut(&mut self, i: &mut LocalInitializer) {
+        visit_local_initializer_mut(self, i);
+    }
     #[cfg(any(feature = "derive", feature = "full"))]
     fn visit_macro_mut(&mut self, i: &mut Macro) {
         visit_macro_mut(self, i);
@@ -2610,11 +2614,22 @@ where
     tokens_helper(v, &mut node.let_token.span);
     v.visit_pat_mut(&mut node.pat);
     if let Some(it) = &mut node.init {
-        tokens_helper(v, &mut (it).0.spans);
-        v.visit_expr_mut(&mut *(it).1);
+        v.visit_local_initializer_mut(it);
     }
     tokens_helper(v, &mut node.semi_token.spans);
 }
+#[cfg(feature = "full")]
+pub fn visit_local_initializer_mut<V>(v: &mut V, node: &mut LocalInitializer)
+where
+    V: VisitMut + ?Sized,
+{
+    tokens_helper(v, &mut node.eq_token.spans);
+    v.visit_expr_mut(&mut *node.expr);
+    if let Some(it) = &mut node.else_block {
+        tokens_helper(v, &mut (it).0.span);
+        v.visit_expr_block_mut(&mut *(it).1);
+    }
+}
 #[cfg(any(feature = "derive", feature = "full"))]
 pub fn visit_macro_mut<V>(v: &mut V, node: &mut Macro)
 where
diff --git a/src/lib.rs b/src/lib.rs
index 86b72db7..287b6522 100644
--- a/src/lib.rs
+++ b/src/lib.rs
@@ -463,7 +463,7 @@ pub mod spanned;
 #[cfg(feature = "full")]
 mod stmt;
 #[cfg(feature = "full")]
-pub use crate::stmt::{Block, Local, Stmt};
+pub use crate::stmt::{Block, Local, LocalInitializer, Stmt};
 
 mod thread;
 
diff --git a/src/stmt.rs b/src/stmt.rs
index f3f17827..33676b47 100644
--- a/src/stmt.rs
+++ b/src/stmt.rs
@@ -30,7 +30,7 @@ ast_enum! {
 }
 
 ast_struct! {
-    /// A local `let` binding: `let x: u64 = s.parse()?`.
+    /// A local `let` binding: `let x: u64 = s.parse()?` (including let-else construction).
     ///
     /// *This type is available only if Syn is built with the `"full"` feature.*
     #[cfg_attr(doc_cfg, doc(cfg(feature = "full")))]
@@ -38,16 +38,31 @@ ast_struct! {
         pub attrs: Vec<Attribute>,
         pub let_token: Token![let],
         pub pat: Pat,
-        pub init: Option<(Token![=], Box<Expr>)>,
+        pub init: Option<LocalInitializer>,
         pub semi_token: Token![;],
     }
 }
 
+ast_struct! {
+    /// An initializer of local `let` binding (with optional diverging else-block)
+    ///
+    /// `LocalInitializer` represents `= s.parse()?` in `let x: u64 = s.parse()?`
+    /// and `= r else { return }` in `let Ok(x) = r else { return }`.
+    ///
+    /// *This type is available only if Syn is built with the `"full"` feature.*
+    #[cfg_attr(doc_cfg, doc(cfg(feature = "full")))]
+    pub struct LocalInitializer {
+        pub eq_token: Token![=],
+        pub expr: Box<Expr>,
+        pub else_block: Option<(Token![else], Box<ExprBlock>)>,
+    }
+}
+
 #[cfg(feature = "parsing")]
 pub mod parsing {
     use super::*;
     use crate::parse::discouraged::Speculative;
-    use crate::parse::{Parse, ParseBuffer, ParseStream, Result};
+    use crate::parse::{Parse, ParseStream, Result};
     use proc_macro2::TokenStream;
 
     struct AllowNoSemi(bool);
@@ -152,7 +167,6 @@ pub mod parsing {
     }
 
     fn parse_stmt(input: ParseStream, allow_nosemi: AllowNoSemi) -> Result<Stmt> {
-        let begin = input.fork();
         let mut attrs = input.call(Attribute::parse_outer)?;
 
         // brace-style macros; paren and bracket macros get parsed as
@@ -170,7 +184,7 @@ pub mod parsing {
         }
 
         if input.peek(Token![let]) {
-            stmt_local(input, attrs, begin)
+            stmt_local(input, attrs)
         } else if input.peek(Token![pub])
             || input.peek(Token![crate]) && !input.peek2(Token![::])
             || input.peek(Token![extern])
@@ -227,7 +241,7 @@ pub mod parsing {
         })))
     }
 
-    fn stmt_local(input: ParseStream, attrs: Vec<Attribute>, begin: ParseBuffer) -> Result<Stmt> {
+    fn stmt_local(input: ParseStream, attrs: Vec<Attribute>) -> Result<Stmt> {
         let let_token: Token![let] = input.parse()?;
 
         let mut pat = Pat::parse_single(input)?;
@@ -246,17 +260,19 @@ pub mod parsing {
             let eq_token: Token![=] = input.parse()?;
             let init: Expr = input.parse()?;
 
-            if input.peek(Token![else]) {
-                input.parse::<Token![else]>()?;
-                let content;
-                braced!(content in input);
-                content.call(Block::parse_within)?;
-                let verbatim = Expr::Verbatim(verbatim::between(begin, input));
-                let semi_token: Token![;] = input.parse()?;
-                return Ok(Stmt::Expr(verbatim, Some(semi_token)));
-            }
+            let else_block = if input.peek(Token![else]) {
+                let else_token = input.parse::<Token![else]>()?;
+                let expr_block = input.parse::<ExprBlock>()?;
+                Some((else_token, Box::new(expr_block)))
+            } else {
+                None
+            };
 
-            Some((eq_token, Box::new(init)))
+            Some(LocalInitializer {
+                eq_token,
+                expr: Box::new(init),
+                else_block,
+            })
         } else {
             None
         };
@@ -338,11 +354,21 @@ mod printing {
             expr::printing::outer_attrs_to_tokens(&self.attrs, tokens);
             self.let_token.to_tokens(tokens);
             self.pat.to_tokens(tokens);
-            if let Some((eq_token, init)) = &self.init {
-                eq_token.to_tokens(tokens);
+            if let Some(init) = &self.init {
                 init.to_tokens(tokens);
             }
             self.semi_token.to_tokens(tokens);
         }
     }
+
+    impl ToTokens for LocalInitializer {
+        fn to_tokens(&self, tokens: &mut TokenStream) {
+            self.eq_token.to_tokens(tokens);
+            self.expr.to_tokens(tokens);
+            if let Some((else_token, expr_block)) = &self.else_block {
+                else_token.to_tokens(tokens);
+                expr_block.to_tokens(tokens);
+            }
+        }
+    }
 }
diff --git a/syn.json b/syn.json
index 62cba985..a51e0190 100644
--- a/syn.json
+++ b/syn.json
@@ -3416,21 +3416,44 @@
           "syn": "Pat"
         },
         "init": {
+          "option": {
+            "syn": "LocalInitializer"
+          }
+        },
+        "semi_token": {
+          "token": "Semi"
+        }
+      }
+    },
+    {
+      "ident": "LocalInitializer",
+      "features": {
+        "any": [
+          "full"
+        ]
+      },
+      "fields": {
+        "eq_token": {
+          "token": "Eq"
+        },
+        "expr": {
+          "box": {
+            "syn": "Expr"
+          }
+        },
+        "else_block": {
           "option": {
             "tuple": [
               {
-                "token": "Eq"
+                "token": "Else"
               },
               {
                 "box": {
-                  "syn": "Expr"
+                  "syn": "ExprBlock"
                 }
               }
             ]
           }
-        },
-        "semi_token": {
-          "token": "Semi"
         }
       }
     },


Test patch:
diff --git a/tests/debug/gen.rs b/tests/debug/gen.rs
index 1108de86..415a78d6 100644
--- a/tests/debug/gen.rs
+++ b/tests/debug/gen.rs
@@ -3501,13 +3501,13 @@ impl Debug for Lite<syn::Local> {
         if let Some(val) = &_val.init {
             #[derive(RefCast)]
             #[repr(transparent)]
-            struct Print((syn::token::Eq, Box<syn::Expr>));
+            struct Print(syn::LocalInitializer);
             impl Debug for Print {
                 fn fmt(&self, formatter: &mut fmt::Formatter) -> fmt::Result {
                     formatter.write_str("Some")?;
                     let _val = &self.0;
                     formatter.write_str("(")?;
-                    Debug::fmt(Lite(&_val.1), formatter)?;
+                    Debug::fmt(Lite(_val), formatter)?;
                     formatter.write_str(")")?;
                     Ok(())
                 }
@@ -3517,6 +3517,30 @@ impl Debug for Lite<syn::Local> {
         formatter.finish()
     }
 }
+impl Debug for Lite<syn::LocalInitializer> {
+    fn fmt(&self, formatter: &mut fmt::Formatter) -> fmt::Result {
+        let _val = &self.value;
+        let mut formatter = formatter.debug_struct("LocalInitializer");
+        formatter.field("expr", Lite(&_val.expr));
+        if let Some(val) = &_val.else_block {
+            #[derive(RefCast)]
+            #[repr(transparent)]
+            struct Print((syn::token::Else, Box<syn::ExprBlock>));
+            impl Debug for Print {
+                fn fmt(&self, formatter: &mut fmt::Formatter) -> fmt::Result {
+                    formatter.write_str("Some")?;
+                    let _val = &self.0;
+                    formatter.write_str("(")?;
+                    Debug::fmt(Lite(&_val.1), formatter)?;
+                    formatter.write_str(")")?;
+                    Ok(())
+                }
+            }
+            formatter.field("else_block", Print::ref_cast(val));
+        }
+        formatter.finish()
+    }
+}
 impl Debug for Lite<syn::Macro> {
     fn fmt(&self, formatter: &mut fmt::Formatter) -> fmt::Result {
         let _val = &self.value;
diff --git a/tests/test_stmt.rs b/tests/test_stmt.rs
index f444e5b4..1a08ac25 100644
--- a/tests/test_stmt.rs
+++ b/tests/test_stmt.rs
@@ -15,7 +15,9 @@ fn test_raw_operator() {
     snapshot!(stmt, @r###"
     Local(Local {
         pat: Pat::Wild,
-        init: Some(Verbatim(`& raw const x`)),
+        init: Some(LocalInitializer {
+            expr: Verbatim(`& raw const x`),
+        }),
     })
     "###);
 }
@@ -27,15 +29,17 @@ fn test_raw_variable() {
     snapshot!(stmt, @r###"
     Local(Local {
         pat: Pat::Wild,
-        init: Some(Expr::Reference {
-            expr: Expr::Path {
-                path: Path {
-                    segments: [
-                        PathSegment {
-                            ident: "raw",
-                            arguments: None,
-                        },
-                    ],
+        init: Some(LocalInitializer {
+            expr: Expr::Reference {
+                expr: Expr::Path {
+                    path: Path {
+                        segments: [
+                            PathSegment {
+                                ident: "raw",
+                                arguments: None,
+                            },
+                        ],
+                    },
                 },
             },
         }),
@@ -85,8 +89,65 @@ fn test_let_dot_dot() {
     snapshot!(tokens as Stmt, @r###"
     Local(Local {
         pat: Pat::Rest,
-        init: Some(Expr::Lit {
-            lit: 10,
+        init: Some(LocalInitializer {
+            expr: Expr::Lit {
+                lit: 10,
+            },
+        }),
+    })
+    "###);
+}
+
+#[test]
+fn test_let_else() {
+    let tokens = quote! {
+        let Some(x) = None else { return 0; };
+    };
+
+    snapshot!(tokens as Stmt, @r###"
+    Local(Local {
+        pat: Pat::TupleStruct {
+            path: Path {
+                segments: [
+                    PathSegment {
+                        ident: "Some",
+                        arguments: None,
+                    },
+                ],
+            },
+            pat: PatTuple {
+                elems: [
+                    Pat::Ident {
+                        ident: "x",
+                    },
+                ],
+            },
+        },
+        init: Some(LocalInitializer {
+            expr: Expr::Path {
+                path: Path {
+                    segments: [
+                        PathSegment {
+                            ident: "None",
+                            arguments: None,
+                        },
+                    ],
+                },
+            },
+            else_block: Some(ExprBlock {
+                block: Block {
+                    stmts: [
+                        Expr(
+                            Expr::Return {
+                                expr: Some(Expr::Lit {
+                                    lit: 0,
+                                }),
+                            },
+                            Some,
+                        ),
+                    ],
+                },
+            }),
         }),
     })
     "###);