Repository: opensearch-project/opensearch-build

Issue:
[Bug]: Version workflow to auto update input manifests is failing
### Describe the bug

Our version workflow to auto raise PR for input manifest update is currently failing for almost 10 days. https://github.com/opensearch-project/opensearch-build/actions/workflows/versions.yml
The context of error:
```
BUILD FAILED in 11s
2022-11-04 00:49:45 WARNING  Error getting version of performance-analyzer: Command './gradlew properties -Dopensearch.version=3.0.0 -Dbuild.snapshot=false' returned non-zero exit status 1., ignored
2022-11-04 00:49:45 INFO     Checking out opensearch-observability#main
2022-11-04 00:49:45 INFO     Executing "git init" in /tmp/tmpco2c3n05/opensearch-observability
2022-11-04 00:49:45 INFO     Executing "git remote add origin https://github.com/opensearch-project/opensearch-observability.git" in /tmp/tmpco2c3n05/opensearch-observability
2022-11-04 00:49:45 INFO     Executing "git fetch --depth 1 origin main" in /tmp/tmpco2c3n05/opensearch-observability
2022-11-04 00:49:45 INFO     Removing /tmp/tmpco2c3n05
```
The problem might be the workflow trying to clone `https://github.com/opensearch-project/opensearch-observability.git` but the correct URL should be `https://github.com/opensearch-project/observability.git`
### To reproduce

https://github.com/opensearch-project/opensearch-build/actions/workflows/versions.yml

### Expected behavior

_No response_

### Screenshots

If applicable, add screenshots to help explain your problem.

### Host / Environment

_No response_

### Additional context

_No response_

### Relevant log output

_No response_


Discussion:
I don't think these maven issues are the reason for the wf failure. 

The [latest](https://github.com/opensearch-project/opensearch-build/actions/runs/3390231242/jobs/5634121899) failure has a separate issue at the bottom.  Where it is trying to fetch `https://github.com/opensearch-project/opensearch-observability` where the repo name is actually https://github.com/opensearch-project/observability.

The latest passing wf run still has those maven errors in it.  So those maven issues are a separate problem (looks like its trying to fetch a "released" version of 3.0 when 3.0 has not yet been released).
> I don't think these maven issues are the reason for the wf failure.
> 
> The [latest](https://github.com/opensearch-project/opensearch-build/actions/runs/3390231242/jobs/5634121899) failure has a separate issue at the bottom. Where it is trying to fetch `https://github.com/opensearch-project/opensearch-observability` where the repo name is actually https://github.com/opensearch-project/observability.
> 
> The latest passing wf run still has those maven errors in it. So those maven issues are a separate problem (looks like its trying to fetch a "released" version of 3.0 when 3.0 has not yet been released).

Yes. I just updated the issue description. Thanks for the input here!
Looks like the repo name being cloned is tied to the name field in the manifest. This [PR](https://github.com/opensearch-project/opensearch-build/commit/20dc58b38c38157aa63dbe88a59038aa433e5556) has a name of opensearch-observability when it needs to be observability.  This feels like a bug where name should not be driving the repo path.
> Looks like the repo name being cloned is tied to the name field in the manifest. This [PR](https://github.com/opensearch-project/opensearch-build/commit/20dc58b38c38157aa63dbe88a59038aa433e5556) has a name of opensearch-observability when it needs to be observability. This feels like a bug where name should not be driving the repo path.

But seems like we have been using `opensearch-observability` as the name for previous manifest. 
e.g. https://github.com/opensearch-project/opensearch-build/blob/a872a391df1a02802425ad200ef273d66f4c6700/manifests/2.2.0/opensearch-2.2.0.yml#L135
I see whats happening, looks to me like this a [bug](https://github.com/opensearch-project/opensearch-build/blob/main/src/manifests_workflow/component_opensearch.py#L29) and never shown up, the manifest update used to happen even before all components are loaded to the latest version manifest, here in this case its [3.0.0](https://github.com/opensearch-project/opensearch-build/tree/main/manifests/3.0.0), components like [opensearch-obervability](https://github.com/opensearch-project/opensearch-build/blob/main/manifests/3.0.0/opensearch-3.0.0.yml#L68-L74) or notifications-core, when added which has different name than the existing repo ([observability](https://github.com/opensearch-project/observability)), this workflow `./manifests.sh update` would fail.
Till now this scenario did not come up, the `./manifests.sh update` would be successfully executed even before all components are loaded to the latest manifest
Log: `2022-11-05 00:10:19 INFO     Examining components in the latest manifest of OpenSearch (3.0.0)`
Now since for 3.0.0 the components like `opensearch-obervability` are added early hence this bug has shown up.

Temporary fix:
On my local for 3.0.0 input manifest I have updated the `obervability` as ` - name: observability` and `./manifests.sh update ` works fine
```
  - name: observability
    repository: https://github.com/opensearch-project/observability.git
    ref: main
    platforms:
      - linux
      - windows
    working_directory: opensearch-observability
```


> I see whats happening, looks to me like this a [bug](https://github.com/opensearch-project/opensearch-build/blob/main/src/manifests_workflow/component_opensearch.py#L29) and never shown up, the manifest update used to happen even before all components are loaded to the latest version manifest, here in this case its [3.0.0](https://github.com/opensearch-project/opensearch-build/tree/main/manifests/3.0.0), components like [opensearch-obervability](https://github.com/opensearch-project/opensearch-build/blob/main/manifests/3.0.0/opensearch-3.0.0.yml#L68-L74) or notifications-core, when added which has different name than the existing repo ([observability](https://github.com/opensearch-project/observability)), this workflow `./manifests.sh update` would fail. Till now this scenario did not come up, the `./manifests.sh update` would be successfully executed even before all components are loaded to the latest manifest Log: `2022-11-05 00:10:19 INFO Examining components in the latest manifest of OpenSearch (3.0.0)` Now since for 3.0.0 the components like `opensearch-obervability` are added early hence this bug has shown up.
> 
> Temporary fix: On my local for 3.0.0 input manifest I have updated the `obervability` as:
> 
> ```
>   - name: observability
>     repository: https://github.com/opensearch-project/observability.git
>     ref: main
>     platforms:
>       - linux
>       - windows
>     working_directory: opensearch-observability
> ```

Yes. I think this is the cause. 
https://github.com/opensearch-project/opensearch-build/blob/a872a391df1a02802425ad200ef273d66f4c6700/src/manifests_workflow/component_opensearch.py#L29)
it takes the name to construct the repo link instead of the repo link from the manifest. 
Another option should be updating the logic here to take the repository link from the existing manifest.
Since we are already reading the manifest file, makes more sense to just use the repo field to fetch the component github url to avoid this situation in future. 
I will work on the fix. @prudhvigodithi @zelinh @mch2 

Resolving patch:
diff --git a/src/manifests/component_manifest.py b/src/manifests/component_manifest.py
--- a/src/manifests/component_manifest.py
+++ b/src/manifests/component_manifest.py
@@ -41,12 +41,16 @@ def __to_dict__(self) -> dict:
 
 class Components(Dict[str, ComponentType], Generic[ComponentType]):
     def __init__(self, data: Dict[Any, Any]) -> None:
-        create_component: Callable[[Any], Tuple[str, ComponentType]] = lambda component: (component["name"], self.__create__(component))
+        create_component: Callable[[Any], Tuple[str, ComponentType]] = lambda component: (
+            component["name"], self.__create__(component))
         super().__init__(map(create_component, data))
 
     @classmethod
     def __create__(self, data: dict) -> ComponentType:
-        return Component(data)  # type: ignore[return-value]
+        if "repository" in data:
+            return ComponentFromSource(data)  # type: ignore[return-value]
+        else:
+            return Component(data)  # type: ignore[return-value]
 
     def __to_dict__(self) -> List[Dict[Any, Any]]:
         as_dict: Callable[[ComponentType], dict] = lambda component: component.__to_dict__()
@@ -98,3 +102,14 @@ def __matches__(self, focus: List[str] = [], platform: str = None) -> bool:
             logging.info(f"Skipping {self.name}")
 
         return matches
+
+
+class ComponentFromSource(Component):
+    repository: str
+
+    def __init__(self, data: dict) -> None:
+        super().__init__(data)
+        self.repository = data["repository"]
+
+    def __to_dict__(self) -> dict:
+        return {"name": self.name, "repository": self.repository}
diff --git a/src/manifests_workflow/component_opensearch.py b/src/manifests_workflow/component_opensearch.py
--- a/src/manifests_workflow/component_opensearch.py
+++ b/src/manifests_workflow/component_opensearch.py
@@ -21,12 +21,13 @@ def checkout(
         name: str,
         path: str,
         opensearch_version: str,
+        repo_url: str,
         branch: str = "main",
         snapshot: bool = False,
         working_directory: str = None,
     ) -> 'ComponentOpenSearch':
         with GitRepository(
-            f"https://github.com/opensearch-project/{name}.git",
+            repo_url,
             branch,
             path,
             working_directory,
diff --git a/src/manifests_workflow/input_manifests.py b/src/manifests_workflow/input_manifests.py
--- a/src/manifests_workflow/input_manifests.py
+++ b/src/manifests_workflow/input_manifests.py
@@ -14,6 +14,7 @@
 
 import ruamel.yaml
 
+from manifests.component_manifest import ComponentFromSource
 from manifests.input_manifest import InputComponents, InputManifest
 from manifests.manifests import Manifests
 from manifests_workflow.component_opensearch import ComponentOpenSearch
@@ -99,21 +100,23 @@ def update(
                     if component.name == self.name:
                         continue
 
-                    logging.info(f"Checking out {component.name}#main")
-                    component = component_klass.checkout(
-                        name=component.name,
-                        path=os.path.join(work_dir.name, component.name),
-                        opensearch_version=manifest.build.version,
-                        branch="main",
-                    )
-
-                    component_version = component.version
-                    if component_version:
-                        release_version = ".".join(component_version.split(".")[:3])
-                        if release_version not in main_versions.keys():
-                            main_versions[release_version] = []
-                        main_versions[release_version].append(component)
-                        logging.info(f"{component.name}#main is version {release_version} (from {component_version})")
+                    if type(component) is ComponentFromSource:
+                        logging.info(f"Checking out {component.name}#main")
+                        component = component_klass.checkout(
+                            name=component.name,
+                            path=os.path.join(work_dir.name, component.name),
+                            opensearch_version=manifest.build.version,
+                            repo_url=component.repository,
+                            branch="main",
+                        )
+
+                        component_version = component.version
+                        if component_version:
+                            release_version = ".".join(component_version.split(".")[:3])
+                            if release_version not in main_versions.keys():
+                                main_versions[release_version] = []
+                            main_versions[release_version].append(component)
+                            logging.info(f"{component.name}#main is version {release_version} (from {component_version})")
 
             # summarize
             logging.info("Found versions on main:")


Test patch:
diff --git a/tests/tests_manifests_workflow/test_component_opensearch.py b/tests/tests_manifests_workflow/test_component_opensearch.py
--- a/tests/tests_manifests_workflow/test_component_opensearch.py
+++ b/tests/tests_manifests_workflow/test_component_opensearch.py
@@ -16,7 +16,7 @@ class TestComponentOpenSearch(unittest.TestCase):
     @patch("os.makedirs")
     @patch.object(GitRepository, "__checkout__")
     def test_checkout(self, *mocks: MagicMock) -> None:
-        component = ComponentOpenSearch.checkout("common-utils", "path", "1.1.0")
+        component = ComponentOpenSearch.checkout("common-utils", "path", "1.1.0", "git")
         self.assertEqual(component.name, "common-utils")
         self.assertFalse(component.snapshot)
 
diff --git a/tests/tests_manifests_workflow/test_input_manifests_opensearch.py b/tests/tests_manifests_workflow/test_input_manifests_opensearch.py
--- a/tests/tests_manifests_workflow/test_input_manifests_opensearch.py
+++ b/tests/tests_manifests_workflow/test_input_manifests_opensearch.py
@@ -9,7 +9,7 @@
 import unittest
 from unittest.mock import MagicMock, call, patch
 
-from manifests.input_manifest import Component
+from manifests.component_manifest import Component, ComponentFromSource
 from manifests_workflow.input_manifests_opensearch import InputManifestsOpenSearch
 
 
@@ -49,7 +49,8 @@ def test_update(self, mock_input_manifest: MagicMock, mock_component_opensearch:
         mock_component_opensearch.return_value = MagicMock(name="common-utils")
         mock_component_opensearch.checkout.return_value = MagicMock(version="0.10.0")
         mock_input_manifest_from_path.return_value.components = {
-            "common-utils": Component({"name": "common-utils", "repository": "git", "ref": "ref"})
+            "common-utils": ComponentFromSource({"name": "common-utils", "repository": "git", "ref": "ref"}),
+            "job-scheduler": Component({"name": "job-scheduler", "dist": "zip"})
         }
         manifests = InputManifestsOpenSearch()
         manifests.update()