Repository: pest-parser/pest (Rust)

Issue:
Grammar file not found on workspace
I use pest in a crate of a workspace, so this crate has no `src` folder. I was forced to create an empty `src` subfolder in my crate so that pest could find my file, it's not clean.

It's easy to reproduce this bug, create a new workspace project and try to import pest grammar file from a crate of workspace.


Discussion:
The question is, how does one solve this? It makes sense for the grammar file to be in `src` since that will make sure it's included in the crate.
I've been using pest in a workspace, and this hasn't been a problem for me.

What I might see being a problem is a _binary_ crate using `bin` instead of `src`. But for my own workspace, I've had no issues; the `src` path is the `src` of the crate I'm working in.

Could you provide an exact reproduction steps? (Also, OS may play a role here, unfortunately.)
I think he means that it will not work where no `src` is present within a crate?
I have created an example repository that reproduces the bug.
If you clone the repository it does not compile. If you create an empty folder named `src` in `my-usb-crate` then it compiles.

https://github.com/librelois/workspace_use_pest
> The question is, how does one solve this?

My opinion is that the path should start from the root folder of the crate, and not from the sub-folder `src` :)
Oh, that project is set up in a very unconventional manner.

Most workspaces are set up where each subcrate has a `src` directory containing the sources, rather than redefining where the entry point is in `Cargo.toml`. (They also can have other "magic" directories, such as `tests` and `examples`.)

Your problem isn't that you're in a workspace, but that you've defined a different `lib.path`.

<details><summary>A common workspace layout</summary>

```
/Cargo.toml
/derive/Cargo.toml
/derive/src/lib.rs
/derive/tests/test001.rs
/traits/Cargo.toml
/traits/src/lib.rs
```
</details>

---

Any change to where the `#[grammar = ""]` path starts from is technically a breaking change. The path is currently defined to be relative to the `src` directory, so for full strict backwards compatibility, that needs to be the default for a relative path.

One solution is to change the path to instead of being relative from `src`, to be relative from the directory of `lib.path` (`bin.path`?).

Another is to take a page from the JS world and use "anchored" paths: `#[grammar = "@/src/pest/grammar.pest"]`, where `@` is basically a "path macro" for `%CARGO_MANIFEST_DIR%`.

And another (poor?) solution is just to make the path `%CARGO_MANIFEST_DIR%/src/../{}` work even when `src` doesn't exist (and "just" collapsing isn't necessarily correct in the face of symlinks, thus why Rust's `canonicalize` currently uses the filesystem instead of being "simple").
It's worth mentioning that https://github.com/pest-parser/pest/pull/360 was necessary because not all users are using cargo (and are missing the env vars relied upon here). IMO what this really wants is the ability to use `include!` and have it pre-expand before proc macros run. I'd consider opening an issue on rust-lang/rust, since this seems like a pretty reasonable thing to want and AFAIK there aren't a lot of great options at the moment.
@cramertj, expending `include!`s before procedural macros are run would probably not be a stop-gap solution for our Cargo dependency. IMO the needed API is just being able to get the path of the file where the procedural macro resides. I've thought about opening an issue on Rust a while ago, but the procedural macros API was still heavily discussed.
@dragostis 
>  IMO the needed API is just being able to get the path of the file where the procedural macro resides.

Where the macro resides? What are you referring to here? The path of file that invokes the macro? The reason I suggested `include!` is that I was imagining a sort of two-stage expansion where the first expands to an `include!` invocation that pulls in the grammar source and the second processes that grammar, generating the resulting Rust code. This would remove the need for the extra `include_str!` in order to hint to cargo about the dep-tracking, but it does involve extra steps. A proc_macro API for "resolve this `Span` containing a path to a `TokenStream` from the pointed-to-file" would perhaps be a more direct way of achieving this.
Sorry, yes, I meant the invocation.

The API can be as simple as to only resolve the `Span` to string or a byte buffer. 
> string or a byte buffer.

I'd think you'd still want a `TokenStream` so that you can refer to `Span`s in the file and do error reporting using the normal API.
Just want to leave a message here: Would be really nice to be able to use a path that does not start from the src/ dir. Use case: I'm using pest from a build.rs and thus don't really want my grammar file(s) be located in src/, which should contain only runtime code.
I've hit this problem as well in a project where I'm not using the default cargo `src/` directory structure. I'd expect this file to be loaded relative to the directory the source file is contained in, similar to all the Rust builtin include-style macros.

I'm guessing its the way it is because plugins don't have access to the source location.
@brson, precisely. I've thought about making a push for a way to get that in the procedural macro world, but I haven't really had time for it. FYI, you can also inline pest grammars right inside of the macro by providing the grammar inline. #360 
It also initially surprised and confused me that relative paths in `#[grammar = "..."]` are implicitly prefixed with `src/`.

I am using Pest in a project with a non-Cargo build system: *Bazel* (with `rules_rust` to define crates and `cargo-raze` to manage dependencies). Bazel Rust builds don't need to follow Cargo directory structure conventions like the use of `src/`.

Fortunately, similar to others, I can easily work around the issue by moving my grammar file to an otherwise empty `src/` subdirectory.
Hey :)

I just stumbled upon this issue and wanted to put in my 2 cents.

One of [my projects](https://github.com/Nukesor/pueue/issues/213) has the issue at hand.
The project doesn't have a `src` folder, as it's split into two different binaries/namespaces that shouldn't have any knowledge about each other. However, they should still belong to the same crate, and thereby shouldn't be put into separate crates in a workspace.

Cargo also fully supports this scenario and using `src` is just a rough guideline.

I would really like to have the possibility to set the path to the parser file, without having to restructure the whole project.

In case you don't want to change the root of the path to `CARGO_MANIFEST_DIR` due to backward compatibility issues, I have an alternative proposal.

What do you think about adding a new parameter which discovers from the root path: `gramar_from_root` (it's a stupid name, but just to give an idea)




Patch:
diff --git a/generator/src/lib.rs b/generator/src/lib.rs
index 27b4d816..a2a54df4 100644
--- a/generator/src/lib.rs
+++ b/generator/src/lib.rs
@@ -41,7 +41,21 @@ pub fn derive_parser(input: TokenStream, include_grammar: bool) -> TokenStream {
     let (data, path) = match content {
         GrammarSource::File(ref path) => {
             let root = env::var("CARGO_MANIFEST_DIR").unwrap_or_else(|_| ".".into());
-            let path = Path::new(&root).join("src/").join(&path);
+
+            // Check whether we can find a file at the path relative to the CARGO_MANIFEST_DIR
+            // first.
+            //
+            // If we cannot find the expected file over there, fallback to the
+            // `CARGO_MANIFEST_DIR/src`, which is the old default and kept for convenience
+            // reasons.
+            // TODO: This could be refactored once `std::path::absolute()` get's stabilized.
+            // https://doc.rust-lang.org/std/path/fn.absolute.html
+            let path = if Path::new(&root).join(&path).exists() {
+                Path::new(&root).join(&path)
+            } else {
+                Path::new(&root).join("src/").join(&path)
+            };
+
             let file_name = match path.file_name() {
                 Some(file_name) => file_name,
                 None => panic!("grammar attribute should point to a file"),


Test patch:
diff --git a/derive/tests/grammar.rs b/derive/tests/grammar.rs
index 57bed907..cf5a4a60 100644
--- a/derive/tests/grammar.rs
+++ b/derive/tests/grammar.rs
@@ -16,7 +16,7 @@ extern crate pest;
 extern crate pest_derive;
 
 #[derive(Parser)]
-#[grammar = "../tests/grammar.pest"]
+#[grammar = "tests/grammar.pest"]
 struct GrammarParser;
 
 #[test]