Repository: allan2/dotenvy (Rust)

Issue:
Regression: Multiline support breaks .env files with quotes in comments
Hello there,

First thanks for keeping this crate alive.
I just tried to update to the latest version, but it broke my environment.
It looks like the new multiline support also checks within comments, and that breaks everything.

It is very simple to reproduce this by modifying `dotenv/tests/common/mod.rs` to this:
```rust
pub fn make_test_dotenv() -> io::Result<TempDir> {
    tempdir_with_dotenv(
        r#"
        # Start of .env file
        # Comment line with single ' quote
        # Comment line with double " quote
        TESTKEY=test_val
        # End of .env file
        "#
    )
}
```

That will create a multiline .env file to test with. And it will break.
When  you remove the two lines having a double and single quote it will run again.

I didn't had time to look into this my self, but maybe @hoijui or @allan2 knows a quick fix for this :).

For now i will stick with the previous version of this crate.



Discussion:
Thanks for reporting this. I will look into it.
Hi @BlackDex,

Maybe I am misunderstanding the issue. I was unable to reproduce.

*.env*
```sh
# Start of .env file
# Comment line with single ' quote
# Comment line with double " quote
TESTKEY=test_val
# End of .env file
```

*main.rs*
```rust
use dotenvy::{dotenv, dotenv_iter, Error};

fn main() -> Result<(), Error> {
    dotenv()?;
    for item in dotenv_iter()? {
        let (key, val) = item?;
        println!("{key}={val}");  // prints `TESTKEY=test_val`
    }
    Ok(())
}
```

Could you provide more info or an example?

@allan2  If i use the example i made above and run `cargo test` it breaks for me
@allan2, i created a new branch on my fork to illustrate it.
https://github.com/allan2/dotenvy/compare/master...BlackDex:dotenvy:quote_issue

With that branch, within the `dotenv` directory both `cargo test` and `cargo run --example quote_issue` will fail
The issue here is that the new Multiline iter checks every line, also if it starts with a `#`.
Which for multiline variables could be needed of course, but it should exclude lines that start with a `#` (And any type of whitespace in front of it) when not within a `StrongOpen` i think.

Example of when it could be needed:
```bash
# This is a comment
CODEGEN_TEST_VAR1="hello!"
CODEGEN_TEST_VAR2="'quotes within quotes'"
# This is a comment with a quote '
CODEGEN_TEST_MULTILINE1="First Line
Second Line"
CODEGEN_TEST_MULTILINE1="# First Line Comment
Second Line
#Third Line Comment
Fourth Line
"
```

@allan2 Unfortunately this is still an issue.

It even has an other regression now.
Take this for example:
```bash
# Start of .env file
# Comment line with single ' quote
# Comment line with double " quote
 # Comment line, starts with space with double " quote
 CODEGEN_TEST_VAR1="hello!"
CODEGEN_TEST_VAR2="'quotes within quotes'"
CODEGEN_TEST_VAR3="double quoted with # hash in value"
CODEGEN_TEST_VAR4='single quoted with # hash in value'
CODEGEN_TEST_VAR5=not_quoted_with_#_hash_in_value
CODEGEN_TEST_VAR5=not_quoted_with_comment_beheind #_hash_in_value
# End of .env file
```

This worked in the `0.15.1` version just fine. Now it breaks.

And the following example also breaks with the multiline feature enabled:
```bash
# Start of .env file
# Comment line with single ' quote
# Comment line with double " quote
 # Comment line, starts with space with double " quote
 CODEGEN_TEST_VAR1="hello!"
CODEGEN_TEST_VAR2="'quotes within quotes'"
CODEGEN_TEST_VAR3="double quoted with # hash in value"
CODEGEN_TEST_VAR4='single quoted with # hash in value'
CODEGEN_TEST_VAR5=not_quoted_with_#_hash_in_value
CODEGEN_TEST_VAR5=not_quoted_with_comment_beheind #_hash_in_value
CODEGEN_TEST_MULTILINE1="First Line
Second Line"
CODEGEN_TEST_MULTILINE2="# First Line Comment
Second Line
#Third Line Comment
Fourth Line
"
# End of .env file
```

If i just leave the `CODEGEN_TEST_MULTILINE1` there, and remove the rest that will work.
Now, i don't know if you want to support Lines starting with a `#` within a multiline quoted string, but besides lines starting with a `#` string, they could be anywhere within the quoted string of course, which still breaks.



I am also having an issue with `KEY='ABC#DEF'`:

**What's happening:**

The line iterator, via `eval_end_state()`, in the `iter` module returns the line as `KEY='ABC`, truncating at the `#`.

Despite the strong quotes `'`, which should require no escaping, it is necessary to escape the `#` as `\#` to get the full line. 

However, this creates problems down the line, as the line iterator is wrong, but the line parsing later on is correct.

**What I tried:**

So I escaped the source line to read `KEY='ABC\#DEF'`. (Added `\`.)

However, then `parse_value()` in the `parse` module dumps the `\` into the final result (extracting `\#` rather than `#` into the final  value).
This happens because `parse_value()` _correctly_ does not expect any escaping (and never sets `escaped = true`) while within `strong_quote`.

**The only two possible outcomes are therefore:**

- **If `#` is** (correctly) **not escaped** within strong quotes: an incorrect value `ABC` instead of the correct value `ABC#DEF`.
- **If `#` is** (incorrectly) **escaped as `\#`** despite strong quotes: an incorrect value `ABC\#DEF` instead of the correct value `ABC#DEF`. 


**This is a regression:**

As mentioned by BlackDex above, this is a `0.15.3` regression.
The issue is introduced in https://github.com/allan2/dotenvy/pull/16/files#diff-2630ab209b8952cf63a6f52f41cef74ba4a5ab035d861895016911ecde5c94a1
@allan2 @LeoniePhiline i might have a fix for this: https://github.com/allan2/dotenvy/compare/master...BlackDex:dotenvy:quote_issue
Hmm, doesn't seem like a very good fix though.
I'm encoutering some other issues with other test .env files I'm currently trying out.


@allan2 i think i have a fix for this, please verify the PR #18 if i missed something.




Patch:
diff --git a/dotenv/src/iter.rs b/dotenv/src/iter.rs
index bf68cf91..6fc60245 100644
--- a/dotenv/src/iter.rs
+++ b/dotenv/src/iter.rs
@@ -38,39 +38,45 @@ struct QuotedLines<B> {
     buf: B,
 }
 
-enum QuoteState {
+enum ParseState {
     Complete,
     Escape,
     StrongOpen,
     StrongOpenEscape,
     WeakOpen,
     WeakOpenEscape,
+    Comment,
 }
 
-fn eval_end_state(prev_state: QuoteState, buf: &str) -> QuoteState {
+fn eval_end_state(prev_state: ParseState, buf: &str) -> ParseState {
     let mut cur_state = prev_state;
 
     for c in buf.chars() {
         cur_state = match cur_state {
-            QuoteState::Escape => QuoteState::Complete,
-            QuoteState::Complete => match c {
-                '\\' => QuoteState::Escape,
-                '"' => QuoteState::WeakOpen,
-                '\'' => QuoteState::StrongOpen,
-                _ => QuoteState::Complete,
+            ParseState::Escape => ParseState::Complete,
+            ParseState::Complete => match c {
+                '#' => return ParseState::Comment,
+                '\\' => ParseState::Escape,
+                '"' => ParseState::WeakOpen,
+                '\'' => ParseState::StrongOpen,
+                _ => ParseState::Complete,
             },
-            QuoteState::WeakOpen => match c {
-                '\\' => QuoteState::WeakOpenEscape,
-                '"' => QuoteState::Complete,
-                _ => QuoteState::WeakOpen,
+            ParseState::WeakOpen => match c {
+                '#' => return ParseState::Comment,
+                '\\' => ParseState::WeakOpenEscape,
+                '"' => ParseState::Complete,
+                _ => ParseState::WeakOpen,
             },
-            QuoteState::WeakOpenEscape => QuoteState::WeakOpen,
-            QuoteState::StrongOpen => match c {
-                '\\' => QuoteState::StrongOpenEscape,
-                '\'' => QuoteState::Complete,
-                _ => QuoteState::StrongOpen,
+            ParseState::WeakOpenEscape => ParseState::WeakOpen,
+            ParseState::StrongOpen => match c {
+                '#' => return ParseState::Comment,
+                '\\' => ParseState::StrongOpenEscape,
+                '\'' => ParseState::Complete,
+                _ => ParseState::StrongOpen,
             },
-            QuoteState::StrongOpenEscape => QuoteState::StrongOpen,
+            ParseState::StrongOpenEscape => ParseState::StrongOpen,
+            // Comments last the entire line.
+            ParseState::Comment => panic!("should have returned early"),
         };
     }
     cur_state
@@ -81,13 +87,13 @@ impl<B: BufRead> Iterator for QuotedLines<B> {
 
     fn next(&mut self) -> Option<Result<String>> {
         let mut buf = String::new();
-        let mut cur_state = QuoteState::Complete;
+        let mut cur_state = ParseState::Complete;
         let mut buf_pos;
         loop {
             buf_pos = buf.len();
             match self.buf.read_line(&mut buf) {
                 Ok(0) => match cur_state {
-                    QuoteState::Complete => return None,
+                    ParseState::Complete => return None,
                     _ => {
                         let len = buf.len();
                         return Some(Err(Error::LineParse(buf, len)));
@@ -95,14 +101,29 @@ impl<B: BufRead> Iterator for QuotedLines<B> {
                 },
                 Ok(_n) => {
                     cur_state = eval_end_state(cur_state, &buf[buf_pos..]);
-                    if let QuoteState::Complete = cur_state {
-                        if buf.ends_with('\n') {
-                            buf.pop();
-                            if buf.ends_with('\r') {
+
+                    match cur_state {
+                        ParseState::Complete => {
+                            if buf.ends_with('\n') {
                                 buf.pop();
+                                if buf.ends_with('\r') {
+                                    buf.pop();
+                                }
                             }
+                            return Some(Ok(buf));
+                        }
+                        ParseState::Escape => {}
+                        ParseState::StrongOpen => {}
+                        ParseState::StrongOpenEscape => {}
+                        ParseState::WeakOpen => {}
+                        ParseState::WeakOpenEscape => {}
+                        ParseState::Comment => {
+                            // Find the start of the comment
+                            let idx = buf.find(|c| c == '#').unwrap();
+                            // Drop the trailing comment text
+                            buf.truncate(idx);
+                            return Some(Ok(buf));
                         }
-                        return Some(Ok(buf));
                     }
                 }
                 Err(e) => return Some(Err(Error::Io(e))),


Test patch:
diff --git a/dotenv/tests/test-multiline-comment.rs b/dotenv/tests/test-multiline-comment.rs
new file mode 100644
index 00000000..7b316780
--- /dev/null
+++ b/dotenv/tests/test-multiline-comment.rs
@@ -0,0 +1,26 @@
+mod common;
+use std::env;
+
+use common::tempdir_with_dotenv;
+use dotenvy::dotenv;
+
+#[test]
+fn test_issue_12() {
+    let _f = tempdir_with_dotenv(
+        r#"
+# Start of .env file
+# Comment line with single ' quote
+# Comment line with double " quote
+ # Comment line with double " quote and starts with a space
+TESTKEY=test_val # A '" comment
+# End of .env file
+"#,
+    )
+    .expect("should write test env");
+
+    dotenv().expect("should succeed");
+    assert_eq!(
+        env::var("TESTKEY").expect("test env key not set"),
+        "test_val"
+    );
+}