Repository: kubeflow/pipelines

Issue:
[sdk] Containerized Python Component module not found error 
There is a bug when building a containerized Python component that happens (at least) in the case when the longest path of the import graph ending at the component involves >2 modules. 

### Environment
KFP SDK 2.0.0-beta.6

### Steps to reproduce
For example:

```python
# component.py
from module_one import one
from kfp import dsl

@dsl.component
def comp(): ...
```

```python
# module_one.py
from module_two import two
one = 1
```

```python
# module_two.py
two = 2
```

Then: `kfp component build .`

You get a `No module named` error.

### Expected result

Should build without an error.

### Materials and Reference
Related: https://github.com/kubeflow/pipelines/issues/8353


Discussion:
Hi @connor-mccarthy I am trying to follow the docs and I ended up with the same error as #8353 how can I help to close this issue?
I can reproduce it with even one just module:

working directory:

```bash
├── main.py
├── poetry.lock
├── poetry.toml
├── pyproject.toml
└── utils.py
```

Where:

## main.py

```
from kfp import dsl
from .utils import return_date

@dsl.component(
    base_image='python:3.7',
    target_image='gcr.io/my-project/my-component:v1',
)
def main():
    x = return_date()
```

## utils.py

```
from datetime import datetime

def return_date():
    return datetime.now().strftime('%Y-%m-%d')
```


## Results:

```bash
╰─>$ kfp component build .
Building component using KFP package path: kfp==2.0.0-beta.11
attempted relative import with no known parent package
```

Alternatively: 

```bash
─>$ kfp component build bug/
Building component using KFP package path: kfp==2.0.0-beta.11
attempted relative import with no known parent package
```

Adding a `__init__.py` which is not in the docs:

 have the same results
 
 Moving the files to a `src` folder and trying to replicate the [docs](https://www.kubeflow.org/docs/components/pipelines/v2/author-a-pipeline/components/#2-containerized-python-components)  I cannot seem to have the same results as the docs
 
 ## Update:
 
Even with changing the `utils.py` as:

```
def return_date():
    return "today"
```

still get the same error.

My poetry just have the package installation `kfp = {version = "2.0.0b11", extras = ["all"]}`




@Davidnet, thank you for the detailed reproduction notes. This is on our backlog.

> how can I help to close this issue?

If you're interested, you're welcome to address this bug and submit a PR!
@connor-mccarthy Awesome, thanks for the response any ideas on where could I start looking?
Thanks, @Davidnet. Some links:

* [CLI entrypoint](https://github.com/kubeflow/pipelines/blob/master/sdk/python/kfp/cli/cli.py#L96)
* [`kfp component build` command](https://github.com/kubeflow/pipelines/blob/master/sdk/python/kfp/cli/component.py)
* [test code](https://github.com/kubeflow/pipelines/blob/master/sdk/python/kfp/cli/component_test.py)
* [docs source](https://github.com/kubeflow/website/blob/master/content/en/docs/components/pipelines/v2/author-a-pipeline/components.md)
Hey guys,
I also had a look on this issue as it is quite annoying and I think I might have found the cause for `No module named` problem.
When running `kfp` command, this resolves (`which kfp`) to a script in (my case, depending on your python installation it will be different) mamba environment bin directory `~/mambaforge/envs/kfp-env/bin/kfp`.
The file looks like this:
```
#!/home/b4sus/mambaforge/envs/kfp-env/bin/python3.11
# -*- coding: utf-8 -*-
import re
import sys
from kfp.cli.__main__ import main
if __name__ == '__main__':
    sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])
    sys.exit(main())
```
I patched the file by adding `print(sys.path)` to see the path where python will be checking for modules. The thing is that current directory (where the `kfp` is executed from) is not there. That one is added only when executing python directly (do not quote me on this, but see [here](https://docs.python.org/3/library/sys.html?#sys.path)) and not through this shell approach.

Now comes the funny part, bare with me here :)
The sdk will check the provided directory in search for components. It will list python files by using Path.glob and try to load each module ([here](https://github.com/kubeflow/pipelines/blob/master/sdk/python/kfp/cli/component.py#L173)).
This may or may not yield the error. It all depends on the order of files returned by the glob method.
For example, following the files from @connor-mccarthy setup, if the glob would return files in order `[module_two.py, module_one.py, component.py]`, the whole thing would work. Loading `module_two.py` would be no problem (no imports), loading `module_one.py` would be no problem (import of `module_two` would work because it was already loaded in), loading of `component.py` would be no problem (import of `module_one` would work because it was already loaded in).

In other words, python would never need to check `sys.path` for loading the modules.

However, if the returned order from glob would be for example `[module_one.py, module_two.py, component.py]`, loading would fail during load of `module_one.py` which is importing `module_two`. This time python doesn't have `module_two` cached and needs to check `sys.path` and it cannot find it -> `No module named` error.

As a quickfix, I added following to the kfp file (`~/mambaforge/envs/kfp-env/bin/kfp`):
```
    import os
    sys.path.append(os.getcwd())
```
This should work when all the component python files are in the current directory where you execute `kfp`.

It also works when using slightly more of a structure, eg:
```
my_project
    app
        components
            __init__.py
            my_component.py (having absolute import app.utils.helper
        utils
            __init__.py
            helper.py
```
Then from `my_project` directory execute:
```
kfp component build . --component-filepattern app/components/my_comp*
```
Thanks for this thorough investigation, @b4sus. This makes sense based on the errors we've observed.

> As a quickfix, I added following to the kfp file (~/mambaforge/envs/kfp-env/bin/kfp):

Is this quickfix a final fix? Or is there a more robust fix you have in mind? If final, are you interested in submitting a PR? I would be happy to review promptly.
Hey @connor-mccarthy,
some more considerations could be necessary, including having some recommended structure for the project. Additionally, I am no expert on python import system, so that's something as well :)

Let's assume standard (I think) python project (let's call it `myproject`) layout:
```
project_dir/
    myproject/
        __init__.py
        components/
            __init__py.py
            my_comp.py
        pipelines/
            __init__py.py
            my_pipelines.py
    pyproject.toml
    .gitignore
```
Now (considering my fix) you have to run `kfp` from `project_dir` like
```
kfp component build .
```
This will work, but there are caveats:
1. you have to use absolute imports (generally recommended afaik)
2. you have to use `.` as input for `kfp` - it will be recursive - checking subdirectories as well - desirable I'd say
3. if somewhere you have also lightweight components it will fail (because of missing base_image) - so you have to separate those to different packages - eg `myproject/components/lightweight` and `myproject/components/cont` - and then use `myproject/components/cont` and then use `--component-filepattern` like `kfp component build . --component-filepattern "myproject/components/cont/*.py"`
4. complete content of `project_dir` will end up in docker image
5. input for `kfp component build` (the `.` above) is basically usesless - you cannot use anything else - otherwise Dockerfile and other files will be generated elsewhere and the docker image will not have correct structure copied into
6. possibly something else

All this is fine for me, but needs to be considered.
@b4sus 
> including having some recommended structure for the project

I agree with this. I'm working on a v2 docs refresh currently and will keep this in mind.

> but there are caveats

Thank you for laying out these considerations. In general, as long as there are no regressions (all existing user code that uses Containerized Python Components will still work), I'm happy to eagerly merge a better but not perfect fix. It sounds like some of these constraints may be been introduced by the proposed fix, however (such as the requirement to use the cwd `.` as the path). If that's the case, perhaps we ought to wait.

If we do wait, I'll take a stab at this soon and certainly leverage your investigation. Thank you again for the thorough writeup -- this helps a lot.

Resolving patch:
diff --git a/sdk/python/kfp/components/utils.py b/sdk/python/kfp/components/utils.py
--- a/sdk/python/kfp/components/utils.py
+++ b/sdk/python/kfp/components/utils.py
@@ -44,6 +44,7 @@ def load_module(module_name: str, module_directory: str) -> types.ModuleType:
         location=os.path.join(module_directory, f'{module_name}.py'))
     module = importlib.util.module_from_spec(module_spec)
     sys.modules[module_spec.name] = module
+    sys.path.insert(0, str(module_directory))
     module_spec.loader.exec_module(module)
     return module
 


Test patch:
diff --git a/sdk/python/kfp/cli/component_test.py b/sdk/python/kfp/cli/component_test.py
--- a/sdk/python/kfp/cli/component_test.py
+++ b/sdk/python/kfp/cli/component_test.py
@@ -265,6 +265,24 @@ def test_component_filepattern_can_be_used_to_restrict_discovery(self):
 
             '''))
 
+    def test_nested_module_imports(self):
+        module_two = 'two = 2'
+        _write_file('module_two.py', module_two)
+
+        module_one = 'from module_two import two\none = 1'
+        _write_file('module_one.py', module_one)
+
+        component = _make_component(
+            func_name='comp', target_image='custom-image')
+        component = 'from module_one import one\n' + component
+        _write_components('component.py', component)
+
+        result = self.runner.invoke(
+            self.cli,
+            ['build', str(self._working_dir)],
+        )
+        self.assertEqual(result.exit_code, 0)
+
     def test_emptry_requirements_txt_file_is_generated(self):
         component = _make_component(
             func_name='train', target_image='custom-image')