Repository: noqdev/iambic
Issue: Race condition across multiple providers from file system changes

**Describe the bug**
During import operation, each provider tries to find existing templates that matches their template types. However, those provider are run independently. So if one of them (Google Workspace provider) delete a Google Workspace Group template file (the checked in file is removed when the corresponding cloud resource is gone); another provider like (Okta) maybe in the middle of opening a file that is removed.

This issue stems from the resource root level is pass to all the providers. 

**To Reproduce**
Steps to reproduce the behavior:

It's a little hard to reproduce because one needs to trigger a file system change within the provider import operation, and operations across two providers interleave. 

**Expected behavior**
A clear and concise description of what you expected to happen.

**Screenshots**
```
Traceback (most recent call last):
   File "/app/iambic/plugins/v0_1_0/github/github.py", line 646, in wrapped_workflow_func
     template_changes = workflow_func(repo_url, default_branch)
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   File "/app/iambic/plugins/v0_1_0/github/github.py", line 813, in _handle_import
     raise e
   File "/app/iambic/plugins/v0_1_0/github/github.py", line 803, in _handle_import
     asyncio.run(config.run_import(exe_message, repo_dir))
   File "/Python-3.11.1/Lib/asyncio/runners.py", line 190, in run
     return runner.run(main)
            ^^^^^^^^^^^^^^^^
   File "/Python-3.11.1/Lib/asyncio/runners.py", line 118, in run
     return self._loop.run_until_complete(task)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   File "/Python-3.11.1/Lib/asyncio/base_events.py", line 653, in run_until_complete
     return future.result()
            ^^^^^^^^^^^^^^^
   File "/app/iambic/config/dynamic_config.py", line 245, in run_import
     await asyncio.gather(*tasks)
   File "/app/iambic/plugins/v0_1_0/okta/handlers.py", line 78, in import_okta_resources
     await asyncio.gather(*generator_tasks)
   File "/app/iambic/plugins/v0_1_0/okta/group/template_generation.py", line 83, in generate_group_templates
     existing_template_map = await get_existing_template_map(
                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   File "/app/iambic/core/template_generation.py", line 55, in get_existing_template_map
     await gather_templates(repo_dir, template_type), template_map
     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   File "/app/iambic/core/utils.py", line 171, in gather_templates
     file_paths = await gather_limit(
                  ^^^^^^^^^^^^^^^^^^^
   File "/app/iambic/core/aio_utils/__init__.py", line 70, in gather_limit
     ret[pos[x]] = x.result()
                   ^^^^^^^^^^
   File "/app/iambic/core/utils.py", line 142, in file_regex_search
     async with aiofiles.open(file_path, mode="r") as f:
   File "/usr/local/lib/python3.11/site-packages/aiofiles/base.py", line 98, in __aenter__
     self._obj = await self._coro
                 ^^^^^^^^^^^^^^^^
   File "/usr/local/lib/python3.11/site-packages/aiofiles/threadpool/__init__.py", line 97, in _open
     f = yield from loop.run_in_executor(executor, cb)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   File "/Python-3.11.1/Lib/concurrent/futures/thread.py", line 58, in run
     result = self.fn(*self.args, **self.kwargs)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 FileNotFoundError: [Errno 2] No such file or directory: '/tmp/lambda3wfudd42/.iambic/repos/resources/google_workspace/group/iambic.org/iambic_test_group.yaml'
```

**Desktop (please complete the following information):**

- OS: [e.g. iOS]
- Browser [e.g. chrome, safari]
- Version [e.g. 22]

**Additional context**
Add any other context about the problem here.

**Community Engagement**
Your vote counts! Please support this bug report by adding a 👍 reaction to the original issue, which will aid the community and maintainers in addressing this problem.

Please refrain from adding "+1" or "me too" comments, as these create unnecessary noise for issue followers and do not help in prioritizing the issue. If you wish to contribute to solving this issue or have submitted a pull request, please leave a comment.


Changed files:
iambic/core/utils.py

diff --git a/iambic/core/utils.py b/iambic/core/utils.py
index 7e1cd779..f6a7b385 100644
--- a/iambic/core/utils.py
+++ b/iambic/core/utils.py
@@ -139,10 +139,14 @@ async def resource_file_upsert(
 async def file_regex_search(
     file_path: Union[str, Path], re_pattern: str
 ) -> Union[str, None]:
-    async with aiofiles.open(file_path, mode="r") as f:
-        file_content = await f.read()
-        if re.search(re_pattern, file_content):
-            return file_path
+    try:
+        async with aiofiles.open(file_path, mode="r") as f:
+            file_content = await f.read()
+            if re.search(re_pattern, file_content):
+                return file_path
+    except FileNotFoundError:
+        # race condition with different providers
+        return None
 
 
 async def gather_templates(repo_dir: str, template_type: str = None) -> list[str]: