Repository: conan-io/conan
Pull request: [profiles] Raising an error if any profile sections are duplicated
Changelog: Feature: Raising a ConanException if any section is duplicated in the same Conan profile file.
Docs: omit
Closes: https://github.com/conan-io/conan/issues/16448


Problem statement:
[bug] profile include() does not work with os=Linux
### Describe the bug

(conan 2.4.0, on linux)


I have a base profile, `base-linux`:
```
[settings]
arch=x86_64
os=Linux
```

I have another profile, `base-clang`:
```
include(base-linux)
[settings]
compiler=clang
compiler.libcxx=libstdc++11
compiler.version=18
compiler.cppstd=20
```

And I have the top level profile, `reldeb-clang`:
```
include(base-clang-18)

[settings]
build_type=RelWithDebInfo
```

When I build with this as the host profile, the resulting output is:
```
======== Input profiles ========
Profile host:
[settings]
build_type=RelWithDebInfo
compiler=clang
compiler.cppstd=20
compiler.libcxx=libstdc++11
compiler.version=18
```

Note, there is no os (and arch) listed, and the recipes complain and fail to build.

If I move the base-linux contents into base-clang, then it works.

There seems to be no documentation on the website about this behaviour (that I could see),
and seems to be a bug.
If the user can't do this, then conan should detect this problem and fail earlier with a clear error message.


### How to reproduce it

_No response_

Hints from issue discussion:
Hi @paulharris - Thanks for reporting this 😁 

I do not get to reproduce the issue. It's working well on my side. Do you have another file named `base-linux` in your Conan v2 home profile? If I put an empty `base-linux` in the Conan profiles folder: `/Users/conan/.conan2/profiles/base-linux`, then the behavior is the same as described above (you can run `conan profile list` to check it rapidly).
To discard name collisions, let's change the file name to `base-superprofile-linux` to something unique or even use a relative path instead of using only the name:

**base-clang**

```ini
include(base-superprofile-linux)
[settings]
compiler=clang
compiler.libcxx=libstdc++11
compiler.version=18
compiler.cppstd=20
```

Or:

```ini
include(./base-linux)
[settings]
compiler=clang
compiler.libcxx=libstdc++11
compiler.version=18
compiler.cppstd=20
```

Let me know if it solved the issue.

That file is the only file named "base-linux" on the filesystem.

I can confirm it is loading by putting "BREAKME" in the file, at which point it fails.
I specify a version-controlled location for my profiles.

```
$ conan version
version: 2.4.0
conan_path: /build/mx/devenv/bin/conan
python
  version: 3.10.12
  sys_version: 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0]
  sys_executable: /build/mx/devenv/bin/python
  is_frozen: False
  architecture: x86_64
system
  version: #24-Ubuntu SMP PREEMPT_DYNAMIC Tue May  7 14:26:31 UTC 2024
  platform: Linux-6.5.0-1023-oem-x86_64-with-glibc2.35
  system: Linux
  release: 6.5.0-1023-oem
  cpu: x86_64
```

```
$ ls
myprofs

$ ls myprofs/
base-clang-18  base-linux  release-clang-18

$ conan new cmake_exe -d name=testprofile -d version=1.0
File saved: CMakeLists.txt
File saved: conanfile.py
File saved: src/main.cpp
File saved: src/testprofile.cpp
File saved: src/testprofile.h
File saved: test_package/conanfile.py
```


```
$ cat myprofs/release-clang-18 
include(base-clang-18)

[settings]
build_type=Release
```
```
$ cat myprofs/base-clang-18 
include(./base-linux)

[settings]
# arch=x86_64
# os=Linux
compiler=clang
compiler.libcxx=libstdc++11
compiler.version=18
compiler.cppstd=20

[conf]
tools.build:compiler_executables={'c': '/usr/bin/clang-18', 'cpp': '/usr/bin/clang++-18'}
```
```
$ cat myprofs/base-linux 
[settings]
arch=x86_64
os=Linux
```

```
$ conan build . --profile:host myprofs/release-clang-18 

======== Input profiles ========
Profile host:
[settings]
arch=x86_64
build_type=Release
compiler=clang
compiler.cppstd=20
compiler.libcxx=libstdc++11
compiler.version=18
os=Linux
[conf]
tools.build:compiler_executables={'c': '/usr/bin/clang-18', 'cpp': '/usr/bin/clang++-18'}

Profile build:
[settings]
[conf]



======== Computing dependency graph ========
Graph root
    conanfile.py (testprofile/1.0): /build/mx/testprofile/conanfile.py

======== Computing necessary packages ========

======== Installing packages ========

======== Installing packages ========

======== Finalizing install (deploy, generators) ========
conanfile.py (testprofile/1.0): Calling generate()
conanfile.py (testprofile/1.0): Generators folder: /build/mx/testprofile/build/Release/generators
ERROR: conanfile.py (testprofile/1.0): Error in generate() method, line 30
	tc.generate()
	ConanException: The 'build' profile must have a 'os' declared
```

```
$ echo "BREAKME" >> myprofs/base-linux 
$ conan build . --profile:host myprofs/release-clang-18 
ERROR: Error reading 'myprofs/release-clang-18' profile: Error reading 'base-clang-18' profile: Error reading './base-linux' profile: Invalid setting line 'BREAKME'
```

I just copied and pasted everything from your comment above (even the commented lines in the clang profile). Neither macOS nor Linux fails 😓 

```shell
$ conan build . --profile:host profs/release-clang-18

======== Input profiles ========
Profile host:
[settings]
arch=x86_64
build_type=Release
compiler=clang
compiler.cppstd=20
compiler.libcxx=libstdc++11
compiler.version=18
os=Linux
[conf]
tools.build:compiler_executables={'c': '/usr/bin/clang-18', 'cpp': '/usr/bin/clang++-18'}

Profile build:
[settings]
arch=x86_64
build_type=Release
compiler=gcc
compiler.cppstd=gnu17
compiler.libcxx=libstdc++11
compiler.version=11
os=Linux

======== Computing dependency graph ========
Graph root
    conanfile.py (testprofile/1.0): /home/develop/conan/xxxxxxx/conanfile.py

======== Computing necessary packages ========

======== Installing packages ========

======== Installing packages ========

======== Finalizing install (deploy, generators) ========
conanfile.py (testprofile/1.0): Calling generate()
conanfile.py (testprofile/1.0): Generators folder: /home/develop/conan/xxxxxxx/build/Release/generators
conanfile.py (testprofile/1.0): CMakeToolchain generated: conan_toolchain.cmake
conanfile.py (testprofile/1.0): CMakeToolchain: Preset 'conan-release' added to CMakePresets.json.
    (cmake>=3.23) cmake --preset conan-release
    (cmake<3.23) cmake <path> -G "Unix Makefiles" -DCMAKE_TOOLCHAIN_FILE=generators/conan_toolchain.cmake  -DCMAKE_POLICY_DEFAULT_CMP0091=NEW -DCMAKE_BUILD_TYPE=Release
conanfile.py (testprofile/1.0): CMakeToolchain generated: /home/develop/conan/xxxxxxx/build/Release/generators/CMakePresets.json
conanfile.py (testprofile/1.0): CMakeToolchain generated: /home/develop/conan/xxxxxxx/CMakeUserPresets.json
conanfile.py (testprofile/1.0): Generating aggregated env files
conanfile.py (testprofile/1.0): Generated aggregated env files: ['conanbuild.sh', 'conanrun.sh']

======== Calling build() ========
conanfile.py (testprofile/1.0): Calling build()
conanfile.py (testprofile/1.0): Running CMake.configure()
conanfile.py (testprofile/1.0): RUN: cmake -G "Unix Makefiles" -DCMAKE_TOOLCHAIN_FILE="generators/conan_toolchain.cmake" -DCMAKE_INSTALL_PREFIX="/home/develop/conan/xxxxxxx" -DCMAKE_POLICY_DEFAULT_CMP0091="NEW" -DCMAKE_BUILD_TYPE="Release" "/home/develop/conan/xxxxxxx"
.....
```

I just opened a PR to simulate this too: https://github.com/conan-io/conan/pull/16454

Please, feel free to try it locally to see if it doesn't work. If the CI does not fail, let's add all the changes that could make it fail.
Ok thanks, there was a problem with my test,
1) I have an empty default profile, so I needed to specify --profile:build
2) My actual profile has a second [settings] section, which is the real problem.

I'm aware that ini file formats have limitations, but having a second group results in the first group's settings ignored.

ie I've changed the test, the profile now looks like:
```
    [settings]
    arch=x86_64
    os=Linux
    build_type=Release
    compiler=clang
    compiler.libcxx=libstdc++11
    compiler.version=18
    compiler.cppstd=20

    [conf]
    tools.build:compiler_executables={'c': '/usr/bin/clang-18', 'cpp': '/usr/bin/clang++-18'}

    [settings]
    package/*:option=Whatever
```

Note that second [settings] group
The output is now:
```
E           conans.errors.ConanInvalidConfiguration: conanfile.py (hello/1.0): Invalid ID: Invalid: 'settings.os' value not defined
```

While I agree this is user-error, profile files can be long, and the cause is not obvious.
I think conan's handling of the profile files should change to either:
a) consider ALL items in all matching [group] sections
or, b) error if there is a repeated [group]

ie the user won't be aware that some of the profile settings will be completely ignored.

I've pushed an update to the test, not sure how to append to your PR, I'll mention it there

Thanks for the feedback!

Now it makes sense what was the issue.

I agree that it would be better to have a clear error report of this situation. I think going with ``b) error if there is a repeated [group]`` would be the way to go, as having repeated sections sounds in general more like an unintended thing, and even if things are merged in sections the final behavior can be confusing to the user.

Lets try to add that error.

Commit message: Added extra warning if sections are duplicated in the same profile file

Patch to review:
diff --git a/conans/client/profile_loader.py b/conans/client/profile_loader.py
index cc4763fba8e..8ac2617bdc1 100644
--- a/conans/client/profile_loader.py
+++ b/conans/client/profile_loader.py
@@ -263,7 +263,10 @@ def get_profile(profile_text, base_profile=None):
                                                          "options", "conf", "buildenv", "runenv",
                                                          "replace_requires", "replace_tool_requires",
                                                          "runner"])
-
+        if doc.duplicated_sections:
+            sections = ', '.join(sorted([f'[{s}]' for s in doc.duplicated_sections]))
+            ConanOutput().warning(f"Profile file has duplicated sections: {sections}. It could cause"
+                                  f" some unexpected behavior.")
         # Parse doc sections into Conan model, Settings, Options, etc
         settings, package_settings = _ProfileValueParser._parse_settings(doc)
         options = Options.loads(doc.options) if doc.options else None
diff --git a/conans/util/config_parser.py b/conans/util/config_parser.py
index 75d4ad93a74..2284a977679 100644
--- a/conans/util/config_parser.py
+++ b/conans/util/config_parser.py
@@ -29,6 +29,7 @@ class ConfigParser(object):
     """
     def __init__(self, text, allowed_fields=None, strip_comments=False):
         self._sections = {}
+        self._duplicated_sections = set()
         self._allowed_fields = allowed_fields or []
         pattern = re.compile(r"^\[([a-z_]{2,50})]")
         current_lines = None
@@ -44,6 +45,9 @@ def __init__(self, text, allowed_fields=None, strip_comments=False):
                 if self._allowed_fields and field not in self._allowed_fields:
                     raise ConanException("ConfigParser: Unrecognized field '%s'" % field)
                 current_lines = []
+                if field in self._sections:
+                    # WARNING: This section is already defined. It'll override the existing one.
+                    self._duplicated_sections.add(field)
                 self._sections[field] = current_lines
             else:
                 if current_lines is None:
@@ -54,6 +58,10 @@ def __init__(self, text, allowed_fields=None, strip_comments=False):
                     line = line.strip()
                 current_lines.append(line)
 
+    @property
+    def duplicated_sections(self):
+        return self._duplicated_sections
+
     def line_items(self):
         # Used atm by load_binary_info()
         return self._sections.items()
diff --git a/test/integration/configuration/profile_test.py b/test/integration/configuration/profile_test.py
index d47dba871a3..30faff6eb28 100644
--- a/test/integration/configuration/profile_test.py
+++ b/test/integration/configuration/profile_test.py
@@ -574,3 +574,42 @@ def configure(self):
     client.save({"conanfile.py": conanfile, "my_profile.txt": profile})
     client.run("install . --profile my_profile.txt")
     assert "I'm hello and my build type is Debug" in client.out
+
+
+def test_consumer_invalid_profile_multiple_groups():
+    """
+    Issue related: https://github.com/conan-io/conan/issues/16448
+    """
+    client = TestClient()
+    conanfile = GenConanfile(name="hello", version="1.0").with_settings("os", "arch",
+                                                                        "build_type", "compiler")
+    prof1 = textwrap.dedent("""\
+    [settings]
+    arch=x86_64
+    os=Linux
+    build_type=Release
+    compiler=clang
+    compiler.libcxx=libstdc++11
+    compiler.version=18
+    compiler.cppstd=20
+
+    [conf]
+    tools.build:compiler_executables={'c': '/usr/bin/clang-18', 'cpp': '/usr/bin/clang++-18'}
+
+    [settings]
+    # Empty and duplicated section
+
+    [options]
+    package/*:option=Whatever
+
+    [conf]
+    # Another one
+    """)
+    client.save({
+        "conanfile.py": conanfile,
+        "myprofs/myprofile": prof1
+    })
+    client.run("build . --profile:host myprofs/myprofile -g CMakeToolchain",
+               assert_error=True)
+    assert "Profile file has duplicated sections: [conf], [settings]" in client.out
+    assert "Invalid: 'settings.os' value not defined" in client.out


Code review comments:
diff_hunk: @@ -263,7 +263,10 @@ def get_profile(profile_text, base_profile=None):
                                                          "options", "conf", "buildenv", "runenv",
                                                          "replace_requires", "replace_tool_requires",
                                                          "runner"])
-
+        if doc.duplicated_sections:
+            sections = ', '.join(sorted([f'[{s}]' for s in doc.duplicated_sections]))
+            ConanOutput().warning(f"Profile file has duplicated sections: {sections}. It could cause"
path: conans/client/profile_loader.py
text: @user1:
Isnt this more of an actual error? I guess it would be breaking to raise :/

@user2:
I would prefer to see it as an error, as conan is ignoring some settings in the shadowed section.  That could be causing some extremely weird behaviour that would be hard to detect until much later.

@author:
That's why I did not raise an error. I was afraid of breaking anything. IMO, a warning is enough to realize any unexpected mistake.

@author:
@user3 wdyt?

@user3:
I think warnings are often skipped and not seen at all.

As the behavior is basically destructive, adding a repeated section will destroy previous definition, this is a very bad thing, so I would raise an error if this condition is detected.