Repository: opensearch-project/opensearch-build

Issue:
Detecting and alerting of duplication keys/components/entries in YAML file
### Is your feature request related to a problem? Please describe

it was found in release 1.3.11 , a PR to update [manifest](https://github.com/opensearch-project/opensearch-build/blob/main/manifests/1.3.11/opensearch-1.3.11.yml) has duplicated components name.
It would cause the resource wasted on CI to rebuild the duplicated components 

### Describe the solution you'd like

We want to have a check to detect if there is any duplication entries based on keys/components/names and probably fail the GitHub check

### Describe alternatives you've considered

Manually check for duplicate values

### Acceptance Criteria
* The manifest check should fail at CI level for components with duplicate components.name values in opensearch and opensearch-dashboard as well as test manifests. See what are [manifests](https://github.com/opensearch-project/opensearch-build/wiki/Building-an-OpenSearch-and-OpenSearch-Dashboards-Distribution#what-are-manifests)


Discussion:
One of the possible approaches with using yamllint - https://stackoverflow.com/a/35149964
https://www.redhat.com/sysadmin/check-yaml-yamllint#:~:text=The%20yamllint%20command%20is%20an,for%20any%20YAML%20you%20write.
https://mikefarah.gitbook.io/yq/

.pre-commit-config.yaml has 
```
'pipenv run yamllint .'
```
The DEVELOPER_GUIDE.md also mentions running
```
$ pipenv run pre-commit install
$ pipenv run pre-commit run --all-files
```
How do we do this at CI level?
Should we try to run the same git hook in CI?


Resolving patch:
diff --git a/src/ci_workflow/ci_manifests.py b/src/ci_workflow/ci_manifests.py
--- a/src/ci_workflow/ci_manifests.py
+++ b/src/ci_workflow/ci_manifests.py
@@ -7,9 +7,12 @@
 
 
 import re
+from collections import Counter
 from io import TextIOWrapper
 from typing import Type, Union
 
+import yaml
+
 from ci_workflow.ci_args import CiArgs
 from ci_workflow.ci_input_manifest import CiInputManifest
 from ci_workflow.ci_test_manifest import CiTestManifest
@@ -23,6 +26,29 @@ def __klass(filename: str) -> Union[Type[CiTestManifest], Type[CiInputManifest]]
         else:
             return CiInputManifest
 
+    @staticmethod
+    def __get_duplicate_component_names(count_component_names: Counter) -> list:
+        duplicate_component_names = []
+        for component_name, count in count_component_names.items():
+            if count > 1:
+                duplicate_component_names.append(component_name)
+        return duplicate_component_names
+
+    @staticmethod
+    def __check_duplicate_component_names(file: TextIOWrapper) -> None:
+        yaml_dict = yaml.safe_load(file)
+        component_names = []
+        for component in yaml_dict['components']:
+            component_names.append(component['name'])
+        count_component_names = Counter(component_names)
+
+        if set(count_component_names.values()) != set([1]):
+            duplicate_component_names = CiManifests.__get_duplicate_component_names(count_component_names)
+            duplicate_component_names_string = ', '.join(duplicate_component_names)
+            raise ValueError(f"Found {duplicate_component_names_string} as a duplicate component(s) in manifest {file.name}. ")
+        file.seek(0)
+
     @classmethod
     def from_file(cls, file: TextIOWrapper, args: CiArgs) -> Union[CiTestManifest, CiInputManifest]:
+        cls.__check_duplicate_component_names(file)
         return cls.__klass(file.name)(file, args)


Test patch:
diff --git a/tests/tests_ci_workflow/data/opensearch-1.3.11.yml b/tests/tests_ci_workflow/data/opensearch-1.3.11.yml
new file mode 100644
--- /dev/null
+++ b/tests/tests_ci_workflow/data/opensearch-1.3.11.yml
@@ -0,0 +1,194 @@
+---
+schema-version: '1.0'
+build:
+  name: OpenSearch
+  version: 1.3.11
+ci:
+  image:
+    name: opensearchstaging/ci-runner:ci-runner-centos7-opensearch-build-v2
+    args: -e JAVA_HOME=/opt/java/openjdk-11
+components:
+  - name: OpenSearch
+    repository: https://github.com/opensearch-project/OpenSearch.git
+    ref: tags/1.3.11
+    checks:
+      - gradle:publish
+      - gradle:properties:version
+  - name: common-utils
+    repository: https://github.com/opensearch-project/common-utils.git
+    ref: tags/1.3.11.0
+    checks:
+      - gradle:publish
+      - gradle:properties:version
+    platforms:
+      - linux
+      - windows
+  - name: job-scheduler
+    repository: https://github.com/opensearch-project/job-scheduler.git
+    ref: tags/1.3.11.0
+    checks:
+      - gradle:properties:version
+      - gradle:dependencies:opensearch.version
+    platforms:
+      - linux
+      - windows
+  - name: alerting
+    repository: https://github.com/opensearch-project/alerting.git
+    ref: tags/1.3.11.0
+    checks:
+      - gradle:properties:version
+      - gradle:dependencies:opensearch.version: alerting
+    platforms:
+      - linux
+      - windows
+  - name: opensearch-reports
+    repository: https://github.com/opensearch-project/reporting.git
+    ref: tags/1.3.11.0
+    checks:
+      - gradle:properties:version
+      - gradle:dependencies:opensearch.version
+    platforms:
+      - linux
+      - windows
+  - name: cross-cluster-replication
+    repository: https://github.com/opensearch-project/cross-cluster-replication.git
+    ref: tags/1.3.11.0
+    checks:
+      - gradle:properties:version
+      - gradle:dependencies:opensearch.version
+    platforms:
+      - linux
+      - windows
+  - name: security
+    repository: https://github.com/opensearch-project/security.git
+    ref: tags/1.3.11.0
+    checks:
+      - gradle:properties:version
+      - gradle:dependencies:opensearch.version
+    platforms:
+      - linux
+      - windows
+  - name: ml-commons
+    repository: https://github.com/opensearch-project/ml-commons.git
+    ref: tags/1.3.11.0
+    checks:
+      - gradle:properties:version
+      - gradle:dependencies:opensearch.version: opensearch-ml-plugin
+    platforms:
+      - linux
+      - windows
+  - name: asynchronous-search
+    repository: https://github.com/opensearch-project/asynchronous-search.git
+    ref: tags/1.3.11.0
+    checks:
+      - gradle:properties:version
+      - gradle:dependencies:opensearch.version
+    platforms:
+      - linux
+      - windows
+  - name: sql
+    repository: https://github.com/opensearch-project/sql.git
+    ref: tags/1.3.11.0
+    checks:
+      - gradle:properties:version
+      - gradle:dependencies:opensearch.version: plugin
+    platforms:
+      - linux
+      - windows
+  - name: k-NN
+    repository: https://github.com/opensearch-project/k-NN.git
+    ref: tags/1.3.11.0
+    checks:
+      - gradle:properties:version
+      - gradle:dependencies:opensearch.version
+    platforms:
+      - linux
+      - windows
+  - name: performance-analyzer
+    repository: https://github.com/opensearch-project/performance-analyzer.git
+    ref: tags/1.3.11.0
+    checks:
+      - gradle:properties:version
+      - gradle:dependencies:opensearch.version
+    platforms:
+      - linux
+  - name: opensearch-observability
+    repository: https://github.com/opensearch-project/observability.git
+    ref: tags/1.3.11.0
+    checks:
+      - gradle:properties:version
+      - gradle:dependencies:opensearch.version
+    platforms:
+      - linux
+      - windows
+  - name: anomaly-detection
+    repository: https://github.com/opensearch-project/anomaly-detection.git
+    ref: tags/1.3.11.0
+    checks:
+      - gradle:properties:version
+      - gradle:dependencies:opensearch.version
+    platforms:
+      - linux
+      - windows
+  - name: index-management
+    repository: https://github.com/opensearch-project/index-management.git
+    ref: tags/1.3.11.0
+    checks:
+      - gradle:properties:version
+      - gradle:dependencies:opensearch.version
+    platforms:
+      - linux
+      - windows
+  - name: sql
+    repository: https://github.com/opensearch-project/sql.git
+    ref: tags/1.3.11.0
+    checks:
+      - gradle:properties:version
+      - gradle:dependencies:opensearch.version: plugin
+    platforms:
+      - linux
+      - windows
+  - name: k-NN
+    repository: https://github.com/opensearch-project/k-NN.git
+    ref: tags/1.3.11.0
+    checks:
+      - gradle:properties:version
+      - gradle:dependencies:opensearch.version
+    platforms:
+      - linux
+      - windows
+  - name: performance-analyzer
+    repository: https://github.com/opensearch-project/performance-analyzer.git
+    ref: tags/1.3.11.0
+    checks:
+      - gradle:properties:version
+      - gradle:dependencies:opensearch.version
+    platforms:
+      - linux
+  - name: opensearch-observability
+    repository: https://github.com/opensearch-project/observability.git
+    ref: tags/1.3.11.0
+    checks:
+      - gradle:properties:version
+      - gradle:dependencies:opensearch.version
+    platforms:
+      - linux
+      - windows
+  - name: anomaly-detection
+    repository: https://github.com/opensearch-project/anomaly-detection.git
+    ref: tags/1.3.11.0
+    checks:
+      - gradle:properties:version
+      - gradle:dependencies:opensearch.version
+    platforms:
+      - linux
+      - windows
+  - name: index-management
+    repository: https://github.com/opensearch-project/index-management.git
+    ref: tags/1.3.11.0
+    checks:
+      - gradle:properties:version
+      - gradle:dependencies:opensearch.version
+    platforms:
+      - linux
+      - windows
diff --git a/tests/tests_ci_workflow/test_duplicate_components_in_manifest_file.py b/tests/tests_ci_workflow/test_duplicate_components_in_manifest_file.py
new file mode 100644
--- /dev/null
+++ b/tests/tests_ci_workflow/test_duplicate_components_in_manifest_file.py
@@ -0,0 +1,20 @@
+# Copyright OpenSearch Contributors
+# SPDX-License-Identifier: Apache-2.0
+#
+# The OpenSearch Contributors require contributions made to
+# this file be licensed under the Apache-2.0 license or a
+# compatible open source license.
+
+import os
+import unittest
+
+from ci_workflow.ci_manifests import CiManifests
+
+
+class TestDuplicateComponentsInManifestFile(unittest.TestCase):
+    def test_duplicate_components_in_manifest_file(self) -> None:
+        data_path = os.path.join(os.path.dirname(__file__), "data")
+        manifest_filename = os.path.join(data_path, 'opensearch-1.3.11.yml')
+        with self.assertRaises(ValueError):
+            with open(manifest_filename, 'r') as manifest_file:
+                CiManifests.from_file(manifest_file, None)