Repository: opensearch-project/opensearch-build

Issue:
Detecting and alerting of duplication keys/components/entries in YAML file
### Is your feature request related to a problem? Please describe

it was found in release 1.3.11 , a PR to update [manifest](https://github.com/opensearch-project/opensearch-build/blob/main/manifests/1.3.11/opensearch-1.3.11.yml) has duplicated components name.
It would cause the resource wasted on CI to rebuild the duplicated components 

### Describe the solution you'd like

We want to have a check to detect if there is any duplication entries based on keys/components/names and probably fail the GitHub check

### Describe alternatives you've considered

Manually check for duplicate values

### Acceptance Criteria
* The manifest check should fail at CI level for components with duplicate components.name values in opensearch and opensearch-dashboard as well as test manifests. See what are [manifests](https://github.com/opensearch-project/opensearch-build/wiki/Building-an-OpenSearch-and-OpenSearch-Dashboards-Distribution#what-are-manifests)


Discussion:
One of the possible approaches with using yamllint - https://stackoverflow.com/a/35149964
https://www.redhat.com/sysadmin/check-yaml-yamllint#:~:text=The%20yamllint%20command%20is%20an,for%20any%20YAML%20you%20write.
https://mikefarah.gitbook.io/yq/

.pre-commit-config.yaml has 
```
'pipenv run yamllint .'
```
The DEVELOPER_GUIDE.md also mentions running
```
$ pipenv run pre-commit install
$ pipenv run pre-commit run --all-files
```
How do we do this at CI level?
Should we try to run the same git hook in CI?


Resolving patch:
diff --git a/src/ci_workflow/ci_manifests.py b/src/ci_workflow/ci_manifests.py
--- a/src/ci_workflow/ci_manifests.py
+++ b/src/ci_workflow/ci_manifests.py
@@ -26,25 +26,16 @@ def __klass(filename: str) -> Union[Type[CiTestManifest], Type[CiInputManifest]]
         else:
             return CiInputManifest
 
-    @staticmethod
-    def __get_duplicate_component_names(count_component_names: Counter) -> list:
-        duplicate_component_names = []
-        for component_name, count in count_component_names.items():
-            if count > 1:
-                duplicate_component_names.append(component_name)
-        return duplicate_component_names
-
     @staticmethod
     def __check_duplicate_component_names(file: TextIOWrapper) -> None:
         yaml_dict = yaml.safe_load(file)
         component_names = []
         for component in yaml_dict['components']:
             component_names.append(component['name'])
-        count_component_names = Counter(component_names)
 
-        if set(count_component_names.values()) != set([1]):
-            duplicate_component_names = CiManifests.__get_duplicate_component_names(count_component_names)
-            duplicate_component_names_string = ', '.join(duplicate_component_names)
+        duplicate_component_names = [comp for comp, count in Counter(component_names).items() if count > 1]
+        duplicate_component_names_string = ', '.join(duplicate_component_names)
+        if len(duplicate_component_names) > 0:
             raise ValueError(f"Found {duplicate_component_names_string} as a duplicate component(s) in manifest {file.name}. ")
         file.seek(0)
 


Test patch:
diff --git a/tests/tests_ci_workflow/data/opensearch-1.3.11.yml b/tests/tests_ci_workflow/data/opensearch-1.3.11-with-duplicates.yml
similarity index 100%
rename from tests/tests_ci_workflow/data/opensearch-1.3.11.yml
rename to tests/tests_ci_workflow/data/opensearch-1.3.11-with-duplicates.yml
diff --git a/tests/tests_ci_workflow/test_duplicate_components_in_manifest_file.py b/tests/tests_ci_workflow/test_duplicate_components_in_manifest_file.py
--- a/tests/tests_ci_workflow/test_duplicate_components_in_manifest_file.py
+++ b/tests/tests_ci_workflow/test_duplicate_components_in_manifest_file.py
@@ -14,7 +14,12 @@
 class TestDuplicateComponentsInManifestFile(unittest.TestCase):
     def test_duplicate_components_in_manifest_file(self) -> None:
         data_path = os.path.join(os.path.dirname(__file__), "data")
-        manifest_filename = os.path.join(data_path, 'opensearch-1.3.11.yml')
-        with self.assertRaises(ValueError):
+        manifest_filename = os.path.join(data_path, 'opensearch-1.3.11-with-duplicates.yml')
+        with self.assertRaises(ValueError) as valueErrorException:
             with open(manifest_filename, 'r') as manifest_file:
                 CiManifests.from_file(manifest_file, None)
+        self.assertTrue(
+            "Found sql, k-NN, performance-analyzer, opensearch-observability, anomaly-detection, "
+            "index-management as a duplicate component(s) in manifest "
+            + manifest_filename
+            in str(valueErrorException.exception))