Repository: joke2k/django-environ

Issue:
Hash sign inside double quotes is interpreted as the start of a comment
Ran into this problem when updating to `0.11.2` for a project with an `.env` file like this:

```
SECRET_KEY="abc#def"
```

The `SECRET_KEY` used to be `abc#def` but now is read as `"abc`.

Apparently the `#` is interpreted as the start of a comment now, except in *singly*-quoted strings.
Looking at the tests for this new feature only the single-quoted case is tested, but the documentation contains examples with double quotes. It seems they are not treated the same in this case; is this intended behaviour?

Comments:

Patch:
diff --git a/.gitignore b/.gitignore
index 5862901..0b4fa31 100644
--- a/.gitignore
+++ b/.gitignore
@@ -1,6 +1,6 @@
 # This file is part of the django-environ.
 #
-# Copyright (c) 2021, Serghei Iakovlev <egrep@protonmail.ch>
+# Copyright (c) 2021-2023, Serghei Iakovlev <egrep@protonmail.ch>
 # Copyright (c) 2013-2021, Daniele Faraglia <daniele.faraglia@gmail.com>
 #
 # For the full copyright and license information, please view
diff --git a/CHANGELOG.rst b/CHANGELOG.rst
index 57c4c8d..ef00d40 100644
--- a/CHANGELOG.rst
+++ b/CHANGELOG.rst
@@ -5,8 +5,19 @@ All notable changes to this project will be documented in this file.
 The format is inspired by `Keep a Changelog <https://keepachangelog.com/en/1.0.0/>`_
 and this project adheres to `Semantic Versioning <https://semver.org/spec/v2.0.0.html>`_.
 
+`v0.11.3`_ - 0-Undefined-2023
+-----------------------------
+Changed
++++++++
+- Disabled inline comments handling by default due to potential side effects.
+  While the feature itself is useful, the project's philosophy dictates that
+  it should not be enabled by default for all users
+  `#499 <https://github.com/joke2k/django-environ/issues/499>`_.
+
+
+
 `v0.11.2`_ - 1-September-2023
--------------------------------
+-----------------------------
 Fixed
 +++++
 - Revert "Add variable expansion." feature
@@ -31,7 +42,7 @@ Added
   `#463 <https://github.com/joke2k/django-environ/pull/463>`_.
 - Added variable expansion
   `#468 <https://github.com/joke2k/django-environ/pull/468>`_.
-- Added capability to handle comments after #, after quoted values,
+- Added capability to handle comments after ``#``, after quoted values,
   like ``KEY= 'part1 # part2' # comment``
   `#475 <https://github.com/joke2k/django-environ/pull/475>`_.
 - Added support for ``interpolate`` parameter
@@ -388,6 +399,7 @@ Added
 - Initial release.
 
 
+.. _v0.11.3: https://github.com/joke2k/django-environ/compare/v0.11.2...v0.11.3
 .. _v0.11.2: https://github.com/joke2k/django-environ/compare/v0.11.1...v0.11.2
 .. _v0.11.1: https://github.com/joke2k/django-environ/compare/v0.11.0...v0.11.1
 .. _v0.11.0: https://github.com/joke2k/django-environ/compare/v0.10.0...v0.11.0
diff --git a/LICENSE.txt b/LICENSE.txt
index 0bd208d..8737f75 100644
--- a/LICENSE.txt
+++ b/LICENSE.txt
@@ -1,4 +1,4 @@
-Copyright (c) 2021, Serghei Iakovlev <egrep@protonmail.ch>
+Copyright (c) 2021-2023, Serghei Iakovlev <egrep@protonmail.ch>
 Copyright (c) 2013-2021, Daniele Faraglia <daniele.faraglia@gmail.com>
 
 Permission is hereby granted, free of charge, to any person obtaining a copy
diff --git a/docs/tips.rst b/docs/tips.rst
index 20b8c3e..66538c4 100644
--- a/docs/tips.rst
+++ b/docs/tips.rst
@@ -2,6 +2,71 @@
 Tips
 ====
 
+Handling Inline Comments in .env Files
+======================================
+
+``django-environ`` provides an optional feature to parse inline comments in ``.env``
+files. This is controlled by the ``parse_comments`` parameter in the ``read_env``
+method.
+
+Modes
+-----
+
+- **Enabled (``parse_comments=True``)**: Inline comments starting with ``#`` will be ignored.
+- **Disabled (``parse_comments=False``)**: The entire line, including comments, will be read as the value.
+- **Default**: The behavior is the same as when ``parse_comments=False``.
+
+Side Effects
+------------
+
+While this feature can be useful for adding context to your ``.env`` files,
+it can introduce unexpected behavior. For example, if your value includes
+a ``#`` symbol, it will be truncated when ``parse_comments=True``.
+
+Why Disabled by Default?
+------------------------
+
+In line with the project's philosophy of being explicit and avoiding unexpected behavior,
+this feature is disabled by default. If you understand the implications and find the feature
+useful, you can enable it explicitly.
+
+Example
+-------
+
+Here is an example demonstrating the different modes of handling inline comments.
+
+**.env file contents**:
+
+.. code-block:: shell
+
+   # .env file contents
+   BOOL_TRUE_WITH_COMMENT=True # This is a comment
+   STR_WITH_HASH=foo#bar # This is also a comment
+
+**Python code**:
+
+.. code-block:: python
+
+   import environ
+
+   # Using parse_comments=True
+   env = environ.Env()
+   env.read_env(parse_comments=True)
+   print(env('BOOL_TRUE_WITH_COMMENT'))  # Output: True
+   print(env('STR_WITH_HASH'))  # Output: foo
+
+   # Using parse_comments=False
+   env = environ.Env()
+   env.read_env(parse_comments=False)
+   print(env('BOOL_TRUE_WITH_COMMENT'))  # Output: True # This is a comment
+   print(env('STR_WITH_HASH'))  # Output: foo#bar # This is also a comment
+
+   # Using default behavior
+   env = environ.Env()
+   env.read_env()
+   print(env('BOOL_TRUE_WITH_COMMENT'))  # Output: True # This is a comment
+   print(env('STR_WITH_HASH'))  # Output: foo#bar # This is also a comment
+
 
 Docker-style file based variables
 =================================
diff --git a/environ/__init__.py b/environ/__init__.py
index 28d4a5a..ddf05f9 100644
--- a/environ/__init__.py
+++ b/environ/__init__.py
@@ -21,7 +21,7 @@ from .environ import *
 __copyright__ = 'Copyright (C) 2013-2023 Daniele Faraglia'
 """The copyright notice of the package."""
 
-__version__ = '0.11.2'
+__version__ = '0.11.3'
 """The version of the package."""
 
 __license__ = 'MIT'
diff --git a/environ/environ.py b/environ/environ.py
index f74884b..a3d64f2 100644
--- a/environ/environ.py
+++ b/environ/environ.py
@@ -1,6 +1,6 @@
 # This file is part of the django-environ.
 #
-# Copyright (c) 2021-2022, Serghei Iakovlev <egrep@protonmail.ch>
+# Copyright (c) 2021-2023, Serghei Iakovlev <egrep@protonmail.ch>
 # Copyright (c) 2013-2021, Daniele Faraglia <daniele.faraglia@gmail.com>
 #
 # For the full copyright and license information, please view
@@ -862,8 +862,8 @@ class Env:
         return config
 
     @classmethod
-    def read_env(cls, env_file=None, overwrite=False, encoding='utf8',
-                 **overrides):
+    def read_env(cls, env_file=None, overwrite=False, parse_comments=False,
+                 encoding='utf8', **overrides):
         r"""Read a .env file into os.environ.
 
         If not given a path to a dotenv path, does filthy magic stack
@@ -883,6 +883,8 @@ class Env:
             the Django settings module from the Django project root.
         :param overwrite: ``overwrite=True`` will force an overwrite of
             existing environment variables.
+        :param parse_comments: Determines whether to recognize and ignore
+           inline comments in the .env file. Default is False.
         :param encoding: The encoding to use when reading the environment file.
         :param \**overrides: Any additional keyword arguments provided directly
             to read_env will be added to the environment. If the key matches an
@@ -927,22 +929,40 @@ class Env:
         for line in content.splitlines():
             m1 = re.match(r'\A(?:export )?([A-Za-z_0-9]+)=(.*)\Z', line)
             if m1:
+
+                # Example:
+                #
+                # line: KEY_499=abc#def
+                # key:  KEY_499
+                # val:  abc#def
                 key, val = m1.group(1), m1.group(2)
-                # Look for value in quotes, ignore post-# comments
-                # (outside quotes)
-                m2 = re.match(r"\A\s*'(?<!\\)(.*)'\s*(#.*\s*)?\Z", val)
-                if m2:
-                    val = m2.group(1)
+
+                if not parse_comments:
+                    # Default behavior
+                    #
+                    # Look for value in single quotes
+                    m2 = re.match(r"\A'(.*)'\Z", val)
+                    if m2:
+                        val = m2.group(1)
                 else:
-                    # For no quotes, find value, ignore comments
-                    # after the first #
-                    m2a = re.match(r"\A(.*?)(#.*\s*)?\Z", val)
-                    if m2a:
-                        val = m2a.group(1)
+                    # Ignore post-# comments (outside quotes).
+                    # Something like ['val'  # comment] becomes ['val'].
+                    m2 = re.match(r"\A\s*'(?<!\\)(.*)'\s*(#.*\s*)?\Z", val)
+                    if m2:
+                        val = m2.group(1)
+                    else:
+                        # For no quotes, find value, ignore comments
+                        # after the first #
+                        m2a = re.match(r"\A(.*?)(#.*\s*)?\Z", val)
+                        if m2a:
+                            val = m2a.group(1)
+
+                # Look for value in double quotes
                 m3 = re.match(r'\A"(.*)"\Z', val)
                 if m3:
                     val = re.sub(r'\\(.)', _keep_escaped_format_characters,
                                  m3.group(1))
+
                 overrides[key] = str(val)
             elif not line or line.startswith('#'):
                 # ignore warnings for empty line-breaks or comments


Test patch:
diff --git a/tests/test_env.py b/tests/test_env.py
index 0b105f5..85e0499 100644
--- a/tests/test_env.py
+++ b/tests/test_env.py
@@ -7,6 +7,7 @@
 # the LICENSE.txt file that was distributed with this source code.
 
 import os
+import tempfile
 from urllib.parse import quote
 
 import pytest
@@ -21,6 +22,59 @@ from .asserts import assert_type_and_value
 from .fixtures import FakeEnv
 
 
+@pytest.mark.parametrize(
+        'variable,value,raw_value,parse_comments',
+        [
+            # parse_comments=True
+            ('BOOL_TRUE_STRING_LIKE_BOOL_WITH_COMMENT', 'True', "'True' # comment\n", True),
+            ('BOOL_TRUE_BOOL_WITH_COMMENT', 'True ', "True # comment\n", True),
+            ('STR_QUOTED_IGNORE_COMMENT', 'foo', " 'foo' # comment\n", True),
+            ('STR_QUOTED_INCLUDE_HASH', 'foo # with hash', "'foo # with hash' # not comment\n", True),
+            ('SECRET_KEY_1', '"abc', '"abc#def"\n', True),
+            ('SECRET_KEY_2', 'abc', 'abc#def\n', True),
+            ('SECRET_KEY_3', 'abc#def', "'abc#def'\n",  True),
+
+            # parse_comments=False
+            ('BOOL_TRUE_STRING_LIKE_BOOL_WITH_COMMENT', "'True' # comment", "'True' # comment\n", False),
+            ('BOOL_TRUE_BOOL_WITH_COMMENT', 'True # comment', "True # comment\n", False),
+            ('STR_QUOTED_IGNORE_COMMENT', " 'foo' # comment", " 'foo' # comment\n", False),
+            ('STR_QUOTED_INCLUDE_HASH', "'foo # with hash' # not comment", "'foo # with hash' # not comment\n", False),
+            ('SECRET_KEY_1', 'abc#def', '"abc#def"\n', False),
+            ('SECRET_KEY_2', 'abc#def', 'abc#def\n', False),
+            ('SECRET_KEY_3', 'abc#def', "'abc#def'\n",  False),
+
+            # parse_comments is not defined (default behavior)
+            ('BOOL_TRUE_STRING_LIKE_BOOL_WITH_COMMENT', "'True' # comment", "'True' # comment\n", None),
+            ('BOOL_TRUE_BOOL_WITH_COMMENT', 'True # comment', "True # comment\n", None),
+            ('STR_QUOTED_IGNORE_COMMENT', " 'foo' # comment", " 'foo' # comment\n", None),
+            ('STR_QUOTED_INCLUDE_HASH', "'foo # with hash' # not comment", "'foo # with hash' # not comment\n", None),
+            ('SECRET_KEY_1', 'abc#def', '"abc#def"\n', None),
+            ('SECRET_KEY_2', 'abc#def', 'abc#def\n', None),
+            ('SECRET_KEY_3', 'abc#def', "'abc#def'\n",  None),
+        ],
+    )
+def test_parse_comments(variable, value, raw_value, parse_comments):
+    old_environ = os.environ
+
+    with tempfile.TemporaryDirectory() as temp_dir:
+        env_path = os.path.join(temp_dir, '.env')
+
+        with open(env_path, 'w') as f:
+            f.write(f'{variable}={raw_value}\n')
+            f.flush()
+
+            env = Env()
+            Env.ENVIRON = {}
+            if parse_comments is None:
+                env.read_env(env_path)
+            else:
+                env.read_env(env_path, parse_comments=parse_comments)
+
+            assert env(variable) == value
+
+    os.environ = old_environ
+
+
 class TestEnv:
     def setup_method(self, method):
         """
@@ -112,10 +166,8 @@ class TestEnv:
         [
             (True, 'BOOL_TRUE_STRING_LIKE_INT'),
             (True, 'BOOL_TRUE_STRING_LIKE_BOOL'),
-            (True, 'BOOL_TRUE_STRING_LIKE_BOOL_WITH_COMMENT'),
             (True, 'BOOL_TRUE_INT'),
             (True, 'BOOL_TRUE_BOOL'),
-            (True, 'BOOL_TRUE_BOOL_WITH_COMMENT'),
             (True, 'BOOL_TRUE_STRING_1'),
             (True, 'BOOL_TRUE_STRING_2'),
             (True, 'BOOL_TRUE_STRING_3'),
@@ -341,8 +393,6 @@ class TestEnv:
 
     def test_smart_cast(self):
         assert self.env.get_value('STR_VAR', default='string') == 'bar'
-        assert self.env.get_value('STR_QUOTED_IGNORE_COMMENT', default='string') == 'foo'
-        assert self.env.get_value('STR_QUOTED_INCLUDE_HASH', default='string') == 'foo # with hash'
         assert self.env.get_value('BOOL_TRUE_STRING_LIKE_INT', default=True)
         assert not self.env.get_value(
             'BOOL_FALSE_STRING_LIKE_INT',
diff --git a/tests/test_env.txt b/tests/test_env.txt
index d5480bf..39ab896 100644
--- a/tests/test_env.txt
+++ b/tests/test_env.txt
@@ -25,8 +25,6 @@ BOOL_TRUE_STRING_3='yes'
 BOOL_TRUE_STRING_4='y'
 BOOL_TRUE_STRING_5='true'
 BOOL_TRUE_BOOL=True
-BOOL_TRUE_STRING_LIKE_BOOL_WITH_COMMENT='True' # comment
-BOOL_TRUE_BOOL_WITH_COMMENT=True # comment
 BOOL_FALSE_STRING_LIKE_INT='0'
 BOOL_FALSE_INT=0
 BOOL_FALSE_STRING_LIKE_BOOL='False'
@@ -47,8 +45,6 @@ INT_VAR=42
 STR_LIST_WITH_SPACES= foo,  spaces
 STR_LIST_WITH_SPACES_QUOTED=' foo','  quoted'
 STR_VAR=bar
-STR_QUOTED_IGNORE_COMMENT= 'foo' # comment
-STR_QUOTED_INCLUDE_HASH='foo # with hash' # not comment
 MULTILINE_STR_VAR=foo\nbar
 MULTILINE_QUOTED_STR_VAR="---BEGIN---\r\n---END---"
 MULTILINE_ESCAPED_STR_VAR=---BEGIN---\\n---END---