Repository: bridgecrewio/checkov
Pull request: Parallelize module download
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

Took a detour - instead of modifying the existing modules, I download them beforehand and leverage a cache to have the results accessible


Problem statement:
Thread Pools
**Is your feature request related to a problem? Please describe.**

Checkov v2 does not successfully execute on our largest terraform root module. It is highly complex with many provider configurations and countable modules across multiple provider regions.

It appears that Checkov is entirely single-threaded, and I wonder if there is space in the codebase for some thread pools to parallelise the execution of some of heavier moving parts so that it might have a chance of succeeding.

**Describe the solution you'd like**
All reasonably parallelisable activities to make use of the multiprocessing library to execute in parallel threads.

**Describe alternatives you've considered**
¯\_(ツ)_/¯

Hints from issue discussion:
Interesting. I love the suggestion. What should be our benchmark for performance or a test terraform repo we can try? 
@schosterbarak - how about Cloudposse's Terraform modules? Lots of Terraform code lol and since it's public we can specify external modules which should give us a maximum slowdown scenario to measure against.  This is one repo that has a lot of TF code: https://github.com/cloudposse/terraform-aws-components
If you’ll try to scan the root of: https://github.com/cloudposse/terraform-aws-components
with the command `checkov -d`you’ll see results in 34 seconds
if you use the command
`IGNORE_HIDDEN_DIRECTORY_ENV=false checkov -d . --quiet --compact --download-external-modules True --external-modules-download-path .external_modules`
you’ll see results in 26 minutes 

I saw that most of the time checkov’s is halted on polling the files from git which makes sense.
2 reactors we need to do to improve runtime on modules:

1. If module is being used multiple times - We are downloading the same module multiple times, 1 per caller. We should put them in a common directory so we won’t download the same module several times.
2. We can parallelize the download of modules but the code there is stateful so not a 1 hours refactor.
@schosterbarak from my point of view the logic behind downloading external modules need to be refactored in general. I started in the past #1287 and made some good progress, but hit a road block on how external modules are looked up by graph checks. I think it would be great, if I talk with @rotemavni and @nimrodkor on a good way to implement it there and then both of your points should be also incorporated.
#1919 Should improve the performance of this issue greatly. From my tests - 37 minutes turned into ~10.
Leaving this issue open as parallelizing modules is still an open issue!

Commit message: Revert unnecessary changes

Patch to review:
diff --git a/checkov/common/runners/runner_registry.py b/checkov/common/runners/runner_registry.py
index 03871554bf..7fda6be3bc 100644
--- a/checkov/common/runners/runner_registry.py
+++ b/checkov/common/runners/runner_registry.py
@@ -55,10 +55,14 @@ def run(
         repo_root_for_plan_enrichment: Optional[List[Union[str, os.PathLike]]] = None,
     ) -> List[Report]:
         integration_feature_registry.run_pre_runner()
-        reports = parallel_runner.run_function(
-            lambda runner: runner.run(root_folder, external_checks_dir=external_checks_dir, files=files,
-                                      runner_filter=self.runner_filter, collect_skip_comments=collect_skip_comments),
-            self.runners, 1)
+        if len(self.runners) == 1:
+            reports = [self.runners[0].run(root_folder, external_checks_dir=external_checks_dir, files=files,
+                                           runner_filter=self.runner_filter, collect_skip_comments=collect_skip_comments)]
+        else:
+            reports = parallel_runner.run_function(
+                lambda runner: runner.run(root_folder, external_checks_dir=external_checks_dir, files=files,
+                                          runner_filter=self.runner_filter, collect_skip_comments=collect_skip_comments),
+                self.runners, 1)
 
         for scan_report in reports:
             self._handle_report(scan_report, guidelines, repo_root_for_plan_enrichment)
diff --git a/checkov/terraform/module_loading/content.py b/checkov/terraform/module_loading/content.py
index fea39c328f..a5fe03d36f 100644
--- a/checkov/terraform/module_loading/content.py
+++ b/checkov/terraform/module_loading/content.py
@@ -38,14 +38,3 @@ def cleanup(self) -> None:
 
     def __repr__(self) -> str:
         return self.path() or ""
-
-    def __enter__(self) -> "ModuleContent":
-        return self
-
-    def __exit__(
-        self,
-        exc_type: Optional[Type[BaseException]],
-        exc_value: Optional[BaseException],
-        traceback: Optional[TracebackType],
-    ) -> None:
-        self.cleanup()
diff --git a/checkov/terraform/module_loading/loaders/git_loader.py b/checkov/terraform/module_loading/loaders/git_loader.py
index 1fe735f16e..0591034959 100644
--- a/checkov/terraform/module_loading/loaders/git_loader.py
+++ b/checkov/terraform/module_loading/loaders/git_loader.py
@@ -30,14 +30,14 @@ def _load_module(self) -> ModuleContent:
             git_getter = GitGetter(module_source, create_clone_and_result_dirs=False)
             git_getter.temp_dir = self.dest_dir
             git_getter.do_get()
-
-            return_dir = self.dest_dir
-            if self.inner_module:
-                return_dir = os.path.join(self.dest_dir, self.inner_module)
-            return ModuleContent(dir=return_dir)
         except Exception as e:
-            self.logger.error(f"failed to get {self.module_source} because of {e}")
-            return ModuleContent(dir=None, failed_url=self.module_source)
+            if 'File exists' not in str(e) and 'already exists and is not an empty directory' not in str(e):
+                self.logger.error(f"failed to get {self.module_source} because of {e}")
+                return ModuleContent(dir=None, failed_url=self.module_source)
+        return_dir = self.dest_dir
+        if self.inner_module:
+            return_dir = os.path.join(self.dest_dir, self.inner_module)
+        return ModuleContent(dir=return_dir)
 
     def _find_module_path(self) -> str:
         module_source = self._parse_module_source()
diff --git a/checkov/terraform/module_loading/loaders/registry_loader.py b/checkov/terraform/module_loading/loaders/registry_loader.py
index 86dc01c427..8ec11d7f5b 100644
--- a/checkov/terraform/module_loading/loaders/registry_loader.py
+++ b/checkov/terraform/module_loading/loaders/registry_loader.py
@@ -50,7 +50,7 @@ def _is_matching_loader(self) -> bool:
 
     def _load_module(self) -> ModuleContent:
         if os.path.exists(self.dest_dir):
-            return ModuleContent(dir=None)
+            return ModuleContent(dir=self.dest_dir)
 
         best_version = self._find_best_version()
 
diff --git a/checkov/terraform/module_loading/module_finder.py b/checkov/terraform/module_loading/module_finder.py
new file mode 100644
index 0000000000..26410f79d2
--- /dev/null
+++ b/checkov/terraform/module_loading/module_finder.py
@@ -0,0 +1,89 @@
+import logging
+import os
+import re
+from typing import List, Callable
+
+from checkov.common.parallelizer.parallel_runner import parallel_runner
+from checkov.terraform.module_loading.registry import module_loader_registry
+
+
+class ModuleDownload:
+    source_dir: str
+    module_link: str = None
+    version: str = None
+
+    def __init__(self, source_dir):
+        self.source_dir = source_dir
+
+    def __str__(self):
+        return f"{self.source_dir} -> {self.module_link} ({self.version})"
+
+    @property
+    def address(self):
+        return f'{self.module_link}:{self.version}'
+
+
+def find_modules(path: str) -> List[ModuleDownload]:
+    modules_found = []
+    for root, dir_names, full_file_names in os.walk(path):
+        for file_name in full_file_names:
+            if not file_name.endswith('.tf'):
+                continue
+            with open(os.path.join(path, root, file_name)) as f:
+                try:
+                    in_module = False
+                    curr_md = None
+                    for line in f:
+                        if not in_module:
+                            if line.startswith('module'):
+                                in_module = True
+                                curr_md = ModuleDownload(os.path.dirname(os.path.join(root, file_name)))
+                                continue
+                        if in_module:
+                            if line.startswith('}'):
+                                in_module = False
+                                if curr_md.module_link is None:
+                                    logging.warning(f'A module at {curr_md.source_dir} had no source, skipping')
+                                else:
+                                    modules_found.append(curr_md)
+                                curr_md = None
+                                continue
+
+                            match = re.match('.*\\bsource\\s*=\\s*"(?P<LINK>.*)"', line)
+                            if match:
+                                curr_md.module_link = match.group('LINK')
+                                continue
+
+                            match = re.match('.*\\bversion\\s*=\\s*"[^\\d]*(?P<VERSION>.*)"', line)
+                            if match:
+                                curr_md.version = match.group('VERSION')
+                except (UnicodeDecodeError, FileNotFoundError) as e:
+                    logging.warning(f"Skipping {os.path.join(path, root, file_name)} because of {str(e)}")
+                    continue
+
+    return modules_found
+
+
+def should_download(path: str) -> bool:
+    return not (path.startswith('./') or path.startswith('../') or path.startswith('/'))
+
+
+def load_tf_modules(path: str, should_download_module: Callable[[str], bool] = should_download):
+    module_loader_registry.root_dir = path
+    modules_to_load = find_modules(path)
+
+    def _download_module(m):
+        if should_download_module(m.module_link):
+            logging.info(f'Downloading module {m.address}')
+            try:
+                content = module_loader_registry.load(m.source_dir, m.module_link,
+                                                      "latest" if not m.version else m.version)
+                if content is None or not content.loaded():
+                    logging.warning(f'Failed to download module {m.address}')
+            except Exception as e:
+                logging.warning("Unable to load module (%s): %s", m.address, e)
+
+    # To avoid duplicate work, we need to get the distinct module sources
+    distinct_modules = {m.address: m for m in modules_to_load}.values()
+
+    parallel_runner.run_function(_download_module, distinct_modules)
diff --git a/checkov/terraform/module_loading/registry.py b/checkov/terraform/module_loading/registry.py
index ab1536df00..fb0bde1e9d 100644
--- a/checkov/terraform/module_loading/registry.py
+++ b/checkov/terraform/module_loading/registry.py
@@ -1,7 +1,7 @@
 import logging
 import os
 import hashlib
-from typing import Optional, List, TYPE_CHECKING, Set
+from typing import Optional, List, TYPE_CHECKING, Set, Dict
 
 from checkov.common.util.consts import DEFAULT_EXTERNAL_MODULES_DIR
 from checkov.terraform.module_loading.content import ModuleContent
@@ -12,6 +12,7 @@
 
 class ModuleLoaderRegistry:
     loaders: List["ModuleLoader"] = []
+    module_content_cache: Dict[str, Optional[ModuleContent]] = {}
 
     def __init__(
         self, download_external_modules: bool = False, external_modules_folder_name: str = DEFAULT_EXTERNAL_MODULES_DIR
@@ -27,6 +28,13 @@ def load(self, current_dir: str, source: str, source_version: Optional[str]) ->
 Search all registered loaders for the first one which is able to load the module source type. For more
 information, see `loader.ModuleLoader.load`.
         """
+        module_address = f'{source}:{source_version}'
+        if module_address in self.module_content_cache:
+            logging.info(f'Used the cache for module {module_address}')
+            return self.module_content_cache[module_address]
+        else:
+            logging.debug(f'Cache miss for {module_address}')
+
         if os.name == 'nt':
             # For windows, due to limitations in the allowed characters for path names, the hash of the source is used.
             # https://docs.microsoft.com/en-us/windows/win32/fileio/naming-a-file#naming-conventions
@@ -37,6 +45,7 @@ def load(self, current_dir: str, source: str, source_version: Optional[str]) ->
         inner_module = ""
         next_url = source
         last_exception = None
+        content = ModuleContent(None)
         while next_url:
             source = next_url
             next_url = ""
@@ -56,6 +65,7 @@ def load(self, current_dir: str, source: str, source_version: Optional[str]) ->
                         inner_module=inner_module,
                     )
                 except Exception as e:
+                    logging.warning(f'Module {module_address} failed to load via {loader.__class__}')
                     last_exception = e
                     continue
                 if content.next_url:
@@ -69,14 +79,17 @@ def load(self, current_dir: str, source: str, source_version: Optional[str]) ->
                 elif not content.loaded():
                     if content.failed_url:
                         self.failed_urls_cache.add(content.failed_url)
-                    content.cleanup()
+                    self.module_content_cache[module_address] = ModuleContent(None)
                     continue
                 else:
+                    self.module_content_cache[module_address] = content
                     return content
 
         if last_exception is not None:
             raise last_exception
-        return ModuleContent(None)
+
+        self.module_content_cache[module_address] = content
+        return content
 
     def register(self, loader: "ModuleLoader") -> None:
         self.loaders.append(loader)
diff --git a/checkov/terraform/parser.py b/checkov/terraform/parser.py
index 8de2714784..b3a619cc89 100644
--- a/checkov/terraform/parser.py
+++ b/checkov/terraform/parser.py
@@ -21,6 +21,7 @@
 from checkov.terraform.graph_builder.graph_components.block_types import BlockType
 from checkov.terraform.graph_builder.graph_components.module import Module
 from checkov.terraform.graph_builder.utils import remove_module_dependency_in_path
+from checkov.terraform.module_loading.module_finder import load_tf_modules
 from checkov.terraform.module_loading.registry import module_loader_registry as default_ml_registry, \
     ModuleLoaderRegistry
 from checkov.terraform.parser_utils import eval_string, find_var_blocks
@@ -105,6 +106,7 @@ def parse_directory(self, directory: str, out_definitions: Optional[Dict],
         default_ml_registry.root_dir = directory
         default_ml_registry.download_external_modules = download_external_modules
         default_ml_registry.external_modules_folder_name = external_modules_download_path
+        load_tf_modules(directory)
         self._parse_directory(dir_filter=lambda d: self._check_process_dir(d), vars_files=vars_files)
 
     def parse_file(self, file: str, parsing_errors: Dict[str, Exception] = None, scan_hcl = False) -> Optional[Dict]:
@@ -421,64 +423,65 @@ def _load_modules(self, root_dir: str, module_loader_registry: ModuleLoaderRegis
                     if version and isinstance(version, list):
                         version = version[0]
                     try:
-                        with module_loader_registry.load(root_dir, source, version) as content:
-                            if not content.loaded():
-                                continue
+                        content = module_loader_registry.load(root_dir, source, version)
+                        if not content.loaded():
+                            logging.info(f'Got no content for {source}:{version}')
+                            continue
 
-                            self._internal_dir_load(directory=content.path(),
-                                                    module_loader_registry=module_loader_registry,
-                                                    dir_filter=dir_filter, specified_vars=specified_vars,
-                                                    module_load_context=module_load_context,
-                                                    keys_referenced_as_modules=keys_referenced_as_modules)
+                        self._internal_dir_load(directory=content.path(),
+                                                module_loader_registry=module_loader_registry,
+                                                dir_filter=dir_filter, specified_vars=specified_vars,
+                                                module_load_context=module_load_context,
+                                                keys_referenced_as_modules=keys_referenced_as_modules)
 
-                            module_definitions = {path: self.out_definitions[path] for path in
-                                                  list(self.out_definitions.keys()) if
-                                                  os.path.dirname(path) == content.path()}
+                        module_definitions = {path: self.out_definitions[path] for path in
+                                              list(self.out_definitions.keys()) if
+                                              os.path.dirname(path) == content.path()}
 
-                            if not module_definitions:
-                                continue
+                        if not module_definitions:
+                            continue
 
-                            # NOTE: Modules are put into the main TF definitions structure "as normal" with the
-                            #       notable exception of the file name. For loaded modules referrer information is
-                            #       appended to the file name to create this format:
-                            #         <file_name>[<referred_file>#<referrer_index>]
-                            #       For example:
-                            #         /the/path/module/my_module.tf[/the/path/main.tf#0]
-                            #       The referrer and index allow a module allow a module to be loaded multiple
-                            #       times with differing data.
-                            #
-                            #       In addition, the referring block will have a "__resolved__" key added with a
-                            #       list pointing to the location of the module data that was resolved. For example:
-                            #         "__resolved__": ["/the/path/module/my_module.tf[/the/path/main.tf#0]"]
-
-                            resolved_loc_list = module_call_data.get(RESOLVED_MODULE_ENTRY_NAME)
-                            if resolved_loc_list is None:
-                                resolved_loc_list = []
-                                module_call_data[RESOLVED_MODULE_ENTRY_NAME] = resolved_loc_list
-
-                            # NOTE: Modules can load other modules, so only append referrer information where it
-                            #       has not already been added.
-                            keys = list(module_definitions.keys())
-                            for key in keys:
-                                if key.endswith("]") or file.endswith("]"):
-                                    continue
-                                keys_referenced_as_modules.add(key)
-                                new_key = f"{key}[{file}#{module_index}]"
-                                module_definitions[new_key] = module_definitions[key]
-                                del module_definitions[key]
-                                del self.out_definitions[key]
-                                if new_key not in resolved_loc_list:
-                                    resolved_loc_list.append(new_key)
-                                if (file, module_call_name) not in self.module_address_map:
-                                    self.module_address_map[(file, module_call_name)] = str(module_index)
-                            resolved_loc_list.sort()  # For testing, need predictable ordering
-
-                            if all_module_definitions:
-                                deep_merge.merge(all_module_definitions, module_definitions)
-                            else:
-                                all_module_definitions = module_definitions
-
-                            self.external_modules_source_map[(source, version)] = content.path()
+                        # NOTE: Modules are put into the main TF definitions structure "as normal" with the
+                        #       notable exception of the file name. For loaded modules referrer information is
+                        #       appended to the file name to create this format:
+                        #         <file_name>[<referred_file>#<referrer_index>]
+                        #       For example:
+                        #         /the/path/module/my_module.tf[/the/path/main.tf#0]
+                        #       The referrer and index allow a module allow a module to be loaded multiple
+                        #       times with differing data.
+                        #
+                        #       In addition, the referring block will have a "__resolved__" key added with a
+                        #       list pointing to the location of the module data that was resolved. For example:
+                        #         "__resolved__": ["/the/path/module/my_module.tf[/the/path/main.tf#0]"]
+
+                        resolved_loc_list = module_call_data.get(RESOLVED_MODULE_ENTRY_NAME)
+                        if resolved_loc_list is None:
+                            resolved_loc_list = []
+                            module_call_data[RESOLVED_MODULE_ENTRY_NAME] = resolved_loc_list
+
+                        # NOTE: Modules can load other modules, so only append referrer information where it
+                        #       has not already been added.
+                        keys = list(module_definitions.keys())
+                        for key in keys:
+                            if key.endswith("]") or file.endswith("]"):
+                                continue
+                            keys_referenced_as_modules.add(key)
+                            new_key = f"{key}[{file}#{module_index}]"
+                            module_definitions[new_key] = module_definitions[key]
+                            del module_definitions[key]
+                            del self.out_definitions[key]
+                            if new_key not in resolved_loc_list:
+                                resolved_loc_list.append(new_key)
+                            if (file, module_call_name) not in self.module_address_map:
+                                self.module_address_map[(file, module_call_name)] = str(module_index)
+                        resolved_loc_list.sort()  # For testing, need predictable ordering
+
+                        if all_module_definitions:
+                            deep_merge.merge(all_module_definitions, module_definitions)
+                        else:
+                            all_module_definitions = module_definitions
+
+                        self.external_modules_source_map[(source, version)] = content.path()
                     except Exception as e:
                         logging.warning("Unable to load module (source=\"%s\" version=\"%s\"): %s",
                                         source, version, e)
diff --git a/tests/terraform/module_loading/loaders/test_local_path_loader.py b/tests/terraform/module_loading/loaders/test_local_path_loader.py
index 58ba7d55dc..d96d66f83e 100644
--- a/tests/terraform/module_loading/loaders/test_local_path_loader.py
+++ b/tests/terraform/module_loading/loaders/test_local_path_loader.py
@@ -7,13 +7,13 @@
 class TestLocalPathLoader(unittest.TestCase):
     def test_child_dir(self):
         current_dir = os.path.dirname(os.path.realpath(__file__))
-        with loader.load(current_dir, current_dir, "./resources", None, "", "") as content:
-            assert content.loaded()
-            assert content.path() == os.path.join(current_dir, "resources")
+        content = loader.load(current_dir, current_dir, "./resources", None, "", "")
+        assert content.loaded()
+        assert content.path() == os.path.join(current_dir, "resources")
 
     def test_unhandled_source(self):
-        with loader.load("current_dir", "current_dir", "hashicorp/consul/aws", None, "", "") as content:
-            assert not content.loaded()
+        content = loader.load("current_dir", "current_dir", "hashicorp/consul/aws", None, "", "")
+        assert not content.loaded()
 
     def test_bad_source(self):
         current_dir = os.path.dirname(os.path.realpath(__file__))
@@ -24,6 +24,6 @@ def test_absolute_path(self):
         # Generate absolute path dynamically so test is OS agnostic
         current_dir = os.path.dirname(os.path.realpath(__file__))
         target_dir = os.path.join(current_dir, "resources")
-        with loader.load(current_dir, current_dir, target_dir, None, '', '') as content:
-            assert content.loaded()
-            assert content.path() == os.path.join(current_dir, "resources")
+        content = loader.load(current_dir, current_dir, target_dir, None, '', '')
+        assert content.loaded()
+        assert content.path() == os.path.join(current_dir, "resources")
diff --git a/tests/terraform/module_loading/test_registry.py b/tests/terraform/module_loading/test_registry.py
index bf0f717c2a..23fdfebe02 100644
--- a/tests/terraform/module_loading/test_registry.py
+++ b/tests/terraform/module_loading/test_registry.py
@@ -26,14 +26,14 @@ def test_load_terraform_registry(self):
         registry = ModuleLoaderRegistry(True, DEFAULT_EXTERNAL_MODULES_DIR)
         registry.root_dir = self.current_dir
         source = "terraform-aws-modules/security-group/aws"
-        with registry.load(current_dir=self.current_dir, source=source, source_version="~> 3.0") as content:
-            assert content.loaded()
-            expected_content_path = os.path.join(
-                self.current_dir,
-                DEFAULT_EXTERNAL_MODULES_DIR,
-                "github.com/terraform-aws-modules/terraform-aws-security-group",
-            )
-            self.assertRegex(content.path(), f"^{expected_content_path}/v3.*")
+        content = registry.load(current_dir=self.current_dir, source=source, source_version="~> 3.0")
+        assert content.loaded()
+        expected_content_path = os.path.join(
+            self.current_dir,
+            DEFAULT_EXTERNAL_MODULES_DIR,
+            "github.com/terraform-aws-modules/terraform-aws-security-group",
+        )
+        self.assertRegex(content.path(), f"^{expected_content_path}/v3.*")
 
     def test_load_terraform_registry_check_cache(self):
         registry = ModuleLoaderRegistry(download_external_modules=True)


Code review comments:
diff_hunk: @@ -30,14 +30,14 @@ def _load_module(self) -> ModuleContent:
             git_getter = GitGetter(module_source, create_clone_and_result_dirs=False)
             git_getter.temp_dir = self.dest_dir
             git_getter.do_get()
-
-            return_dir = self.dest_dir
-            if self.inner_module:
-                return_dir = os.path.join(self.dest_dir, self.inner_module)
-            return ModuleContent(dir=return_dir)
         except Exception as e:
-            self.logger.error(f"failed to get {self.module_source} because of {e}")
-            return ModuleContent(dir=None, failed_url=self.module_source)
+            if 'File exists' not in str(e) and 'already exists and is not an empty directory' not in str(e):
path: checkov/terraform/module_loading/loaders/git_loader.py
text: @user1:
I think it would good to create a variable for `str(e)` then you only need to cast it once, you can also use it then in the error logging

diff_hunk: @@ -0,0 +1,89 @@
+import logging
+import os
+import re
+from typing import List, Callable
+
+from checkov.common.parallelizer.parallel_runner import parallel_runner
+from checkov.terraform.module_loading.registry import module_loader_registry
+
+
+class ModuleDownload:
+    source_dir: str
+    module_link: str = None
+    version: str = None
+
+    def __init__(self, source_dir):
+        self.source_dir = source_dir
+
+    def __str__(self):
+        return f"{self.source_dir} -> {self.module_link} ({self.version})"
+
+    @property
+    def address(self):
+        return f'{self.module_link}:{self.version}'
+
+
+def find_modules(path: str) -> List[ModuleDownload]:
+    modules_found = []
+    for root, dir_names, full_file_names in os.walk(path):
+        for file_name in full_file_names:
+            if not file_name.endswith('.tf'):
+                continue
+            with open(os.path.join(path, root, file_name)) as f:
+                try:
+                    in_module = False
+                    curr_md = None
+                    for line in f:
+                        if not in_module:
+                            if line.startswith('module'):
+                                in_module = True
+                                curr_md = ModuleDownload(os.path.dirname(os.path.join(root, file_name)))
+                                continue
+                        if in_module:
+                            if line.startswith('}'):
+                                in_module = False
+                                if curr_md.module_link is None:
+                                    logging.warning(f'A module at {curr_md.source_dir} had no source, skipping')
+                                else:
+                                    modules_found.append(curr_md)
+                                curr_md = None
+                                continue
+
+                            match = re.match('.*\\bsource\\s*=\\s*"(?P<LINK>.*)"', line)
+                            if match:
+                                curr_md.module_link = match.group('LINK')
+                                continue
+
+                            match = re.match('.*\\bversion\\s*=\\s*"[^\\d]*(?P<VERSION>.*)"', line)
+                            if match:
+                                curr_md.version = match.group('VERSION')
+                except (UnicodeDecodeError, FileNotFoundError) as e:
+                    logging.warning(f"Skipping {os.path.join(path, root, file_name)} because of {str(e)}")
+                    continue
+
+    return modules_found
+
+
+def should_download(path: str) -> bool:
+    return not (path.startswith('./') or path.startswith('../') or path.startswith('/'))
+
+
+def load_tf_modules(path: str, should_download_module: Callable[[str], bool] = should_download):
+    module_loader_registry.root_dir = path
+    modules_to_load = find_modules(path)
+
+    def _download_module(m):
+        if should_download_module(m.module_link):
+            logging.info(f'Downloading module {m.address}')
+            try:
+                content = module_loader_registry.load(m.source_dir, m.module_link,
+                                                      "latest" if not m.version else m.version)
+                if content is None or not content.loaded():
+                    logging.warning(f'Failed to download module {m.address}')
+            except Exception as e:
+                logging.warning("Unable to load module (%s): %s", m.address, e)
path: checkov/terraform/module_loading/module_finder.py
text: @user1:
```suggestion
                logging.warning(f"Unable to load module ({m.address}): {e}")
```

diff_hunk: @@ -0,0 +1,89 @@
+import logging
+import os
+import re
+from typing import List, Callable
+
+from checkov.common.parallelizer.parallel_runner import parallel_runner
+from checkov.terraform.module_loading.registry import module_loader_registry
+
+
+class ModuleDownload:
+    source_dir: str
+    module_link: str = None
+    version: str = None
+
+    def __init__(self, source_dir):
+        self.source_dir = source_dir
+
+    def __str__(self):
+        return f"{self.source_dir} -> {self.module_link} ({self.version})"
+
+    @property
+    def address(self):
+        return f'{self.module_link}:{self.version}'
+
+
+def find_modules(path: str) -> List[ModuleDownload]:
+    modules_found = []
+    for root, dir_names, full_file_names in os.walk(path):
+        for file_name in full_file_names:
+            if not file_name.endswith('.tf'):
+                continue
+            with open(os.path.join(path, root, file_name)) as f:
+                try:
+                    in_module = False
+                    curr_md = None
+                    for line in f:
+                        if not in_module:
+                            if line.startswith('module'):
+                                in_module = True
+                                curr_md = ModuleDownload(os.path.dirname(os.path.join(root, file_name)))
+                                continue
+                        if in_module:
+                            if line.startswith('}'):
+                                in_module = False
+                                if curr_md.module_link is None:
+                                    logging.warning(f'A module at {curr_md.source_dir} had no source, skipping')
+                                else:
+                                    modules_found.append(curr_md)
+                                curr_md = None
+                                continue
+
+                            match = re.match('.*\\bsource\\s*=\\s*"(?P<LINK>.*)"', line)
+                            if match:
+                                curr_md.module_link = match.group('LINK')
+                                continue
+
+                            match = re.match('.*\\bversion\\s*=\\s*"[^\\d]*(?P<VERSION>.*)"', line)
+                            if match:
+                                curr_md.version = match.group('VERSION')
+                except (UnicodeDecodeError, FileNotFoundError) as e:
+                    logging.warning(f"Skipping {os.path.join(path, root, file_name)} because of {str(e)}")
path: checkov/terraform/module_loading/module_finder.py
text: @user1:
```suggestion
                    logging.warning(f"Skipping {os.path.join(path, root, file_name)} because of {e}")
```
no need for `str()` cast, will be done by the f-string literal anyway