Repository: pest-parser/pest (language: rust)

Issue:
Put the include_str into generated code rather than require users to write down it
It should be trivial to put an `include_str` into the parser impl, probably as a const inside the parse function, to inform the compiler that it has a dependency to the specific grammar file.

With that, users wouldn't need to write an `include_str` outside like demonstrated in the document, and they don't need to know the complexity about how compiler tracks such dependency.


Discussion / hints:
Unfortunately, this does not currently work, because the generator only knows the Cargo path and the `include_str` only knows the current file path and there seems to be no way to get one information in the other place. Might be a good idea to lobby for file tainting functionality in `proc_macro2`.
Maybe pass the absolute path to `include_str`?
Does that work? The documentation only specifies relative paths.
Yes, it does work. `include_str` can use absolute path. Whether cargo would correctly track that as dependency is unknown, though. May be worth playing with it a bit.
Will look into it. Would be really nice of this worked out. Thanks for the tip. :smile: 
But `include_str!` requires string literal. And `Path` can only convert to `String` and `&str`.
I think there's no way to convert `Path` to string literal(`&'static str`).
So how to solve this problem?
In the derive, we would take the `Path` and turn it into a `String`, then use that to create a [`LitStr`](https://docs.rs/syn/0.14.9/syn/struct.LitStr.html) which is then put in the generated output.
 @CAD97 what is "generated output"?
The way that `pest` works, it uses a [procedural #[derive] macro](https://doc.rust-lang.org/book/2018-edition/appendix-04-macros.html#procedural-macros-for-custom-derive) to generate the code which does the actual parsing. This is code run _before_ your crate is compiled, and it works via receiving a sequence of tokens representing the structure you're deriving on, and outputting a sequence of tokens to be inserted into your code.

This means that it can work around a lot of the restrictions of macro-by-example (`macro_rules!`), such as generating string literals from non-string-literal content, or reading an external grammar file and generating a PEG parser in a reasonable time.

Side-note: [proc-macro](https://github.com/rust-lang/rust/issues/38356)/[macros 1.2](https://internals.rust-lang.org/t/help-stabilize-a-subset-of-macros-2-0/7252?u=cad97)/attribute macros have stabilized (in 1.29 IIUC); we don't _have to_ use the derive mechanism anymore. It's probably best to stick with it for the upcoming 2.0 release, but it's something to keep in mind for the 🦄future.

Fix patch:
diff --git a/derive/src/generator.rs b/derive/src/generator.rs
index 93469452..c9c00de8 100644
--- a/derive/src/generator.rs
+++ b/derive/src/generator.rs
@@ -8,6 +8,7 @@
 // modified, or distributed except according to those terms.
 
 use std::collections::HashMap;
+use std::path::Path;
 
 use proc_macro2::{Span, TokenStream};
 use syn::{self, Generics, Ident};
@@ -19,12 +20,14 @@ use pest_meta::UNICODE_PROPERTY_NAMES;
 pub fn generate(
     name: Ident,
     generics: &Generics,
+    path: &Path,
     rules: Vec<OptimizedRule>,
     defaults: Vec<&str>
 ) -> TokenStream {
     let uses_eoi = defaults.iter().any(|name| *name == "EOI");
 
     let builtins = generate_builtin_rules();
+    let include_fix = generate_include(&name, &path.to_str().expect("non-Unicode path"));
     let rule_enum = generate_enum(&rules, uses_eoi);
     let patterns = generate_patterns(&rules, uses_eoi);
     let skip = generate_skip(&rules);
@@ -71,6 +74,7 @@ pub fn generate(
     };
 
     quote! {
+        #include_fix
         #rule_enum
         #parser_impl
     }
@@ -144,6 +148,16 @@ fn generate_builtin_rules() -> HashMap<&'static str, TokenStream> {
     builtins
 }
 
+// Needed because Cargo doesn't watch for changes in grammers.
+fn generate_include(name: &Ident, path: &str) -> TokenStream {
+    let const_name = Ident::new(&format!("_PEST_GRAMMAR_{}", name), Span::call_site());
+    quote! {
+        #[allow(non_upper_case_globals)]
+        #[cfg(debug_assertions)]
+        const #const_name: &'static str = include_str!(#path);
+    }
+}
+
 fn generate_enum(rules: &Vec<OptimizedRule>, uses_eoi: bool) -> TokenStream {
     let rules = rules.iter().map(|rule| Ident::new(rule.name.as_str(), Span::call_site()));
     if uses_eoi {
@@ -883,8 +897,12 @@ mod tests {
         let defaults = vec!["ANY"];
 
         assert_eq!(
-            generate(name, &generics, rules, defaults).to_string(),
+            generate(name, &generics, Path::new("test.pest"), rules, defaults).to_string(),
             quote! {
+                #[allow(non_upper_case_globals)]
+                #[cfg(debug_assertions)]
+                const _PEST_GRAMMAR_MyParser: &'static str = include_str!("test.pest");
+
                 #[allow(dead_code, non_camel_case_types)]
                 #[derive(Clone, Copy, Debug, Eq, Hash, Ord, PartialEq, PartialOrd)]
                 pub enum Rule {
diff --git a/derive/src/lib.rs b/derive/src/lib.rs
index 9452413e..b0a65cd4 100644
--- a/derive/src/lib.rs
+++ b/derive/src/lib.rs
@@ -21,16 +21,7 @@
 //! relative to `src` and is specified between the `derive` attribute and empty `struct` that
 //! `Parser` will be derived on.
 //!
-//! Because of a limitation in procedural macros, there is no way for Cargo to know that a module
-//! needs to be recompiled based on the file that the procedural macro is opening. This leads to the
-//! case where modifying a `.pest` file without touching the file where the `derive` is does not
-//! recompile it if it already has a working binary in the cache. To avoid this issue, the grammar
-//! file can be included in a dummy `const` definition while debugging.
-//!
 //! ```ignore
-//! #[cfg(debug_assertions)]
-//! const _GRAMMAR: &'static str = include_str!("path/to/my_grammar.pest"); // relative to this file
-//!
 //! #[derive(Parser)]
 //! #[grammar = "path/to/my_grammar.pest"] // relative to src
 //! struct MyParser;
@@ -338,7 +329,7 @@ pub fn derive_parser(input: TokenStream) -> TokenStream {
     let defaults = unwrap_or_report(validator::validate_pairs(pairs.clone()));
     let ast = unwrap_or_report(parser::consume_rules(pairs));
     let optimized = optimizer::optimize(ast);
-    let generated = generator::generate(name, &generics, optimized, defaults);
+    let generated = generator::generate(name, &generics, &path, optimized, defaults);
 
     generated.into()
 }
diff --git a/pest/src/lib.rs b/pest/src/lib.rs
index 56f899df..3629f675 100644
--- a/pest/src/lib.rs
+++ b/pest/src/lib.rs
@@ -36,16 +36,7 @@
 //! relative to `src` and is specified between the `derive` attribute and an empty `struct` that
 //! `Parser` will be derived on.
 //!
-//! Because of a limitation in procedural macros, there is no way for Cargo to know that a module
-//! needs to be recompiled based on the file that the procedural macro is opening. This leads to the
-//! case where modifying a `.pest` file without touching the file where the `derive` is does not
-//! recompile it if it already has a working binary in the cache. To avoid this issue, the grammar
-//! file can be included in a dummy `const` definition while debugging.
-//!
 //! ```ignore
-//! #[cfg(debug_assertions)]
-//! const _GRAMMAR: &'static str = include_str!("path/to/my_grammar.pest"); // relative to this file
-//!
 //! #[derive(Parser)]
 //! #[grammar = "path/to/my_grammar.pest"] // relative to src
 //! struct MyParser;


Test patch: