Repository: espressif/pytest-embedded (language: python)

Issue:
ImportError When Enable Services `idf, qemu` (RDT-414)
https://github.com/espressif/pytest-embedded/blob/main/pytest-embedded-idf/pytest_embedded_idf/__init__.py#L4

https://github.com/espressif/pytest-embedded/blob/main/pytest-embedded-idf/pytest_embedded_idf/serial.py#L9

we need some lazy load method?


Discussion / hints:

Fix patch:
diff --git a/.gitignore b/.gitignore
index e8fd7e52..494031cb 100644
--- a/.gitignore
+++ b/.gitignore
@@ -65,6 +65,7 @@ target/
 
 # venv
 venv/
+.venv/
 
 # IDE
 .idea/
diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
index adf3859f..878075e7 100644
--- a/.pre-commit-config.yaml
+++ b/.pre-commit-config.yaml
@@ -11,31 +11,19 @@ repos:
     rev: v4.4.0
     hooks:
       - id: trailing-whitespace
-        exclude: &whitespace_excludes |
-          (?x)^(
-            .+\.(md|rst|map|bin)|
-            .+test.*\/.*expected.*|
-          )$
       - id: end-of-file-fixer
-        exclude: *whitespace_excludes
       - id: mixed-line-ending
         args: [ '-f=lf' ]
       - id: double-quote-string-fixer
-  - repo: https://github.com/ikamensh/flynt/
-    rev: '0.77'
+  - repo: https://github.com/pycqa/isort
+    rev: 5.12.0
     hooks:
-      - id: flynt
+      - id: isort
   - repo: https://github.com/psf/black
-    rev: 22.12.0
+    rev: 23.1.0
     hooks:
       - id: black
-        args: [ '--skip-string-normalization' ]
-  - repo: https://github.com/pycqa/flake8
-    rev: 6.0.0
+  - repo: https://github.com/charliermarsh/ruff-pre-commit
+    rev: 'v0.0.255'
     hooks:
-      - id: flake8
-  - repo: https://github.com/pycqa/isort
-    rev: 5.12.0
-    hooks:
-      - id: isort
-        name: isort (python)
+      - id: ruff
diff --git a/CHANGELOG.md b/CHANGELOG.md
index ba3a183d..cc359bad 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -101,7 +101,7 @@
 - `live_print_call` could accept kwargs `msg_queue` and `expect_returncode`
 - **idf**: Support panic handler output for RISC-V targets
 - **serial**: support bind ports by USB device location
-- **esp**: `EspSerial.use_esptool` could accept kwargs `hard_reset_after` and `no_stub` 
+- **esp**: `EspSerial.use_esptool` could accept kwargs `hard_reset_after` and `no_stub`
 - **esp**: add `EsptoolArgs`
 - **esp**: support bind ports by `esptool` `read_mac`
 - **esp**: support loadable elf
diff --git a/README.md b/README.md
index 33424cf7..b6b655a7 100644
--- a/README.md
+++ b/README.md
@@ -25,7 +25,7 @@ from pytest_embedded import Dut
 def test_basic_expect(redirect, dut: Dut):
     with redirect():
         print('this would be redirected')
-  
+
     dut.expect('this')
     dut.expect_exact('would')
     dut.expect('[be]{2}')
@@ -45,7 +45,7 @@ test_basic.py .                                                        [100%]
 - if run with `pytest -s`, the output would be as follows:
 
 ```shell
-collected 1 item                                                                                                  
+collected 1 item
 
 test_basic.py 2022-01-01 12:34:56 this would be redirected
 .
diff --git a/docs/expecting.md b/docs/expecting.md
index 63e5db77..f8e42878 100644
--- a/docs/expecting.md
+++ b/docs/expecting.md
@@ -37,7 +37,7 @@ If expecting success, the return value would be a `re.Match` object.
         # here we use fixture `redirect` to write the sys.stdout to dut
         with redirect():
             print('this would be redirected')
-    
+
         res = dut.expect('this (would) be ([cdeirt]+)')
         assert res.group() == b'this would be redirected'
         assert res.group(1) == b'would'
@@ -58,10 +58,10 @@ You can get the bytes read before timeout by expecting a `pexpect.TIMEOUT` objec
             for _ in range(5):
                 dut.write('1')
                 time.sleep(2)
-    
+
         write_thread = threading.Thread(target=write_bytes, daemon=True)
         write_thread.start()
-    
+
         res = dut.expect(pexpect.TIMEOUT, timeout=3)
         assert res == b'11'
     ```
@@ -76,10 +76,10 @@ You can also get all bytes in the pexpect process buffer by expecting a `pexpect
     def test_expect_from_eof_current_buffer(dut):
         dut.write('this would be redirected')
         dut.expect('this')
-    
+
         # close the pexpect process to generate an EOF
         dut.pexpect_proc.terminate()
-    
+
         res = dut.expect(pexpect.EOF, timeout=None)
         assert res == b' would be redirected'
     ```
@@ -90,16 +90,16 @@ You can also get all bytes in the pexpect process buffer by expecting a `pexpect
     If you're expecting `pexpect.EOF` as the first statement, it would return an empty byte string
 
     !!! example
-    
+
         ```python
         import pexpect
 
         def test_expect_from_eof_at_first(dut):
             dut.write('this would be redirected')
-        
+
             # close the pexpect process to generate an EOF
             dut.pexpect_proc.terminate()
-        
+
             res = dut.expect(pexpect.EOF, timeout=None)
             assert res == b''
         ```
@@ -113,14 +113,14 @@ What's more, argument `pattern` could be a list of all supported types.
 
     def test_expect_from_list(dut):
         dut.write("this would be redirected")
-    
+
         pattern_list = [
             "this",
             b"would",
             "[be]+",
             re.compile(b"redirected"),
         ]
-    
+
         for _ in range(4):
             dut.expect(pattern_list)
     ```
@@ -138,7 +138,7 @@ If the pattern is `str`, would convert to `bytes` and then run the function.
     ```python
     def test_expect_exact(dut):
         dut.write('this would be redirected')
-    
+
         dut.expect_exact('this would')
         dut.expect_exact(b'be redirected')
     ```
@@ -150,12 +150,12 @@ Same as [`expect(pattern, **kwargs)`][expectpattern-kwargs], argument `pattern`
     ```python
     def test_expect_exact_from_list(dut):
         dut.write('this would be redirected')
-    
+
         pattern_list = [
             'this would',
             b'be redirected',
         ]
-    
+
         for _ in range(2):
             dut.expect_exact(pattern_list)
     ```
@@ -168,15 +168,14 @@ This function would parse the output as the unity output. The default value of `
 
 When the test script ends, the DUT object would raise `AssertionError` if any unity test case's result is "FAIL".
 
-What's more, it would dump the junit report under a temp folder and would combine the junit report into the main one 
-if you use `pytest --junitxml` feature.
+What's more, it would dump the junit report under a temp folder and would combine the junit report into the main one if you use `pytest --junitxml` feature.
 
 !!! example
 
     ```python
     import inspect
     import pytest
-    
+
     def test_expect_unity_test_output_basic(dut):
         dut.write(inspect.cleandoc('''
             foo.c:100:test_case:FAIL:Expected 2 was 1
@@ -187,7 +186,7 @@ if you use `pytest --junitxml` feature.
         '''))
         with pytest.raises(AssertionError):
             dut.expect_unity_test_output()
-    
+
         assert len(dut.testsuite.testcases) == 2
         assert dut.testsuite.attrs['failures'] == 2
         assert dut.testsuite.testcases[0].attrs['message'] == 'Expected 2 was 1'
@@ -201,7 +200,7 @@ It also supports [unity fixtures](https://github.com/ThrowTheSwitch/Unity/tree/m
     ```python
     import inspect
     import pytest
-    
+
     def test_expect_unity_test_output_fixture(dut):
         dut.write(inspect.cleandoc('''
             TEST(group, test_case)foo.c:100::FAIL:Expected 2 was 1
@@ -212,7 +211,7 @@ It also supports [unity fixtures](https://github.com/ThrowTheSwitch/Unity/tree/m
         '''))
         with pytest.raises(AssertionError):
             dut.expect_unity_test_output()
-    
+
         assert len(dut.testsuite.testcases) == 2
         assert dut.testsuite.attrs['failures'] == 2
         assert dut.testsuite.testcases[0].attrs['message'] == 'Expected 2 was 1'
diff --git a/docs/key_concepts.md b/docs/key_concepts.md
index 1fd40ca2..074b1033 100644
--- a/docs/key_concepts.md
+++ b/docs/key_concepts.md
@@ -95,23 +95,23 @@ Here are a few examples of how to enable this. For detailed information, please
 After you enabled the multi-dut mode, all the fixtures would be a tuple with instances. Each instance inside the tuple would be independent. For parametrization, each configuration will use `|` as a separator for each instance.
 
 !!! example
-    
+
     ```shell
     pytest \
     --embedded-services serial|serial \
     --count 2 \
     --app-path <master_bin>|<slave_bin>
     ```
-  
+
     In this example, `app` would be a tuple of 2 `App` instances, and `dut` would be a tuple of 2 `Dut` Instances.
-  
+
     You can test with:
-  
+
     ```python
     def test(dut):
         master = dut[0]
         slave = dut[1]
-  
+
         master.expect('sent')
         slave.expect('received')
     ```
@@ -121,7 +121,7 @@ After you enabled the multi-dut mode, all the fixtures would be a tuple with ins
 If all DUTs share the same configuration value, you can specify only once.
 
 !!! example
-      
+
     ```shell
     pytest \
     --embedded-services serial \
@@ -136,7 +136,7 @@ If all DUTs share the same configuration value, you can specify only once.
 Sometimes one option is only useful when enabling specific services. You can set a vacant value if this config is only useful for certain DUTs.
 
 !!! example
-      
+
     ```shell
     pytest \
     --embedded-services qemu|serial \
@@ -145,7 +145,7 @@ Sometimes one option is only useful when enabling specific services. You can set
     --qemu-cli-args "<args>|" \
     --port "|<port>" \
     ```
-  
+
     `--qemu-cli-args` would apply to the first DUT with `qemu` service and `--port` would apply to the second DUT with `serial` service.
 
 ## Logging
diff --git a/docs/services.md b/docs/services.md
index 0d357958..8d9a1280 100644
--- a/docs/services.md
+++ b/docs/services.md
@@ -2,11 +2,11 @@
 
 ## Fixtures
 
-Please refer to instructions under the "fixtures defined from pytest_embedded.plugin" chapter of the output of `pytest --fixtures`. 
+Please refer to instructions under the "fixtures defined from pytest_embedded.plugin" chapter of the output of `pytest --fixtures`.
 
 ## CLI Options
 
-Please refer to the instructions under the "embedded" or "embedded-[SERVICE]" groups of the output of `pytest --help`. 
+Please refer to the instructions under the "embedded" or "embedded-[SERVICE]" groups of the output of `pytest --help`.
 
 ## Services
 
diff --git a/examples/arduino/README.md b/examples/arduino/README.md
index 74def0f0..876b4785 100644
--- a/examples/arduino/README.md
+++ b/examples/arduino/README.md
@@ -33,4 +33,3 @@ $ pytest examples/arduino -k test_hello_arduino
 
 This will parse the `build` directory created earlier, flash the chip and
 expect the `Hello Arduino!` text to be printed.
-
diff --git a/pyproject.toml b/pyproject.toml
index ed229a5f..39ede622 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -1,3 +1,8 @@
+[tool.commitizen]
+name = "cz_conventional_commits"
+version = "1.2.4"
+tag_format = "v$version"
+
 [tool.isort]
 profile = 'black'
 
@@ -5,13 +10,60 @@ profile = 'black'
 line-length = 120
 target-version = ['py37']
 force-exclude = '/tests/'
-skip-string-normalization = 'True'
+skip-string-normalization = true
+
+[tool.ruff]
+select = [
+    'F',  # Pyflakes
+    'E',  # pycodestyle
+    'W',  # pycodestyle
+#    'C90',  # mccabe
+#    'I',  # isort
+#    'N',  # pep8-naming
+#    'D',  # pydocstyle
+#    'UP',  #  pyupgrade
+#    'YTT',  # flake8-2020
+#    'ANN',  # flake8-annotations
+#    'S',  # flake8-bandit
+#    'BLE',  # flake8-blind-except
+#    'FBT',  # flake8-boolean-trap
+#    'B',  # flake8-bugbear
+#    'A',  # flake8-builtins
+#    'COM',  # flake8-commas
+#    'C4',  #  flake8-comprehensions
+#    'DTZ',  # flake8-datetimez
+#    'T10',  # flake8-debugger
+#    'DJ',  #  flake8-django
+#    'EM',  #  flake8-errmsg
+#    'EXE',  # flake8-executable
+#    'ISC',  # flake8-implicit-str-concat
+#    'ICN',  # flake8-import-conventions
+#    'G',  # flake8-logging-format
+#    'INP',  # flake8-no-pep420
+#    'PIE',  # flake8-pie
+#    'T20',  # flake8-print
+#    'PYI',  # flake8-pyi
+#    'PT',  #  flake8-pytest-style
+#    'Q',  # flake8-quotes
+#    'RSE',  # flake8-raise
+#    'RET',  # flake8-return
+#    'SLF',  # flake8-self
+#    'SIM',  # flake8-simplify
+#    'TID',  # flake8-tidy-imports
+#    'TCH',  # flake8-type-checking
+#    'ARG',  # flake8-unused-arguments
+#    'PTH',  # flake8-use-pathlib
+#    'ERA',  # eradicate
+#    'PD',  #  pandas-vet
+#    'PGH',  # pygrep-hooks
+#    'PL',  #  Pylint
+#    'TRY',  # tryceratops
+#    'NPY',  # NumPy-specific rules
+#    'RUF',  # Ruff-specific rules
+]
+line-length = 120
+target-version = "py37"
 
 [tool.pytest.ini_options]
 norecursedirs = 'examples/*'
 addopts = "-s"
-
-[tool.commitizen]
-name = "cz_conventional_commits"
-version = "1.2.4"
-tag_format = "v$version"


Test patch:
diff --git a/pytest-embedded-arduino/pytest_embedded_arduino/serial.py b/pytest-embedded-arduino/pytest_embedded_arduino/serial.py
index 8878d002..22658943 100644
--- a/pytest-embedded-arduino/pytest_embedded_arduino/serial.py
+++ b/pytest-embedded-arduino/pytest_embedded_arduino/serial.py
@@ -68,5 +68,5 @@ def flash(self) -> None:
         except Exception:
             raise
         finally:
-            for (_, f) in flash_files:
+            for _, f in flash_files:
                 f.close()
diff --git a/pytest-embedded-idf/pytest_embedded_idf/__init__.py b/pytest-embedded-idf/pytest_embedded_idf/__init__.py
index 8ff30696..05bdc6e6 100644
--- a/pytest-embedded-idf/pytest_embedded_idf/__init__.py
+++ b/pytest-embedded-idf/pytest_embedded_idf/__init__.py
@@ -1,9 +1,26 @@
+import importlib
+
+from pytest_embedded.utils import lazy_load
+
 from .app import IdfApp
-from .dut import IdfDut
 from .linux import LinuxDut, LinuxSerial
-from .serial import IdfSerial
 from .unity_tester import CaseTester
 
+__getattr__ = lazy_load(
+    importlib.import_module(__name__),
+    {
+        'IdfApp': IdfApp,
+        'LinuxDut': LinuxDut,
+        'LinuxSerial': LinuxSerial,
+        'CaseTester': CaseTester,
+    },
+    {
+        'IdfSerial': '.serial',
+        'IdfDut': '.dut',
+    },
+)
+
+
 __all__ = [
     'IdfApp',
     'IdfSerial',
diff --git a/pytest-embedded-idf/pytest_embedded_idf/app.py b/pytest-embedded-idf/pytest_embedded_idf/app.py
index 7c035481..55d5abf7 100644
--- a/pytest-embedded-idf/pytest_embedded_idf/app.py
+++ b/pytest-embedded-idf/pytest_embedded_idf/app.py
@@ -211,7 +211,7 @@ def _is_encrypted(_flash_args: Dict[str, Any], _offset: int, _file_path: str):
             return False
 
         flash_files = []
-        for (offset, file_path) in flash_args['flash_files'].items():
+        for offset, file_path in flash_args['flash_files'].items():
             flash_files.append(
                 FlashFile(
                     int(offset, 0),
diff --git a/pytest-embedded-idf/pytest_embedded_idf/dut.py b/pytest-embedded-idf/pytest_embedded_idf/dut.py
index 0963b5f4..b5400e4e 100644
--- a/pytest-embedded-idf/pytest_embedded_idf/dut.py
+++ b/pytest-embedded-idf/pytest_embedded_idf/dut.py
@@ -91,6 +91,10 @@ def panic_output_decode_script(self) -> t.Optional[str]:
         return os.path.realpath(script_filepath)
 
     def _check_panic_decode_trigger(self):  # type: () -> None
+        if not self.app.elf_file:
+            logging.warning('No elf file found. Skipping decode panic output...')
+            return
+
         with open(self.logfile, 'rb') as output_file:
             output = output_file.read()
         # get the panic output by looking for the indexes
@@ -101,6 +105,7 @@ def _check_panic_decode_trigger(self):  # type: () -> None
         panic_output = panic_output_res if panic_output_res else None
         if panic_output is None:
             return
+
         with tempfile.NamedTemporaryFile(mode='wb', delete=False) as panic_output_file:
             panic_output_file.write(panic_output)
             panic_output_file.flush()
@@ -219,7 +224,7 @@ def close(self) -> None:
         if not self.skip_check_coredump:
             try:
                 self._check_coredump()
-            except ValueError as e:
+            except Exception as e:
                 logging.debug(e)
         super().close()
 
diff --git a/pytest-embedded-idf/pytest_embedded_idf/serial.py b/pytest-embedded-idf/pytest_embedded_idf/serial.py
index ff5ea4c3..59e6e948 100644
--- a/pytest-embedded-idf/pytest_embedded_idf/serial.py
+++ b/pytest-embedded-idf/pytest_embedded_idf/serial.py
@@ -171,9 +171,9 @@ def flash(self) -> None:
                     os.remove(nvs_file.name)
                 except OSError:
                     pass
-            for (_, f) in flash_files:
+            for _, f in flash_files:
                 f.close()
-            for (_, f) in encrypt_files:
+            for _, f in encrypt_files:
                 f.close()
 
     @EspSerial.use_esptool()
diff --git a/pytest-embedded/pytest_embedded/unity.py b/pytest-embedded/pytest_embedded/unity.py
index eebe235b..b4f13961 100644
--- a/pytest-embedded/pytest_embedded/unity.py
+++ b/pytest-embedded/pytest_embedded/unity.py
@@ -54,7 +54,9 @@
     (0xFFFFE, 0xFFFFF),
     (0x10FFFE, 0x10FFFF),
 ]
-ILLEGAL_XML_CHAR_REGEX = re.compile(f"[{''.join([f'{chr(l)}-{chr(r)}' for l, r in _avoid_compatibility_chars])}]")
+ILLEGAL_XML_CHAR_REGEX = re.compile(
+    f"[{''.join([f'{chr(lower)}-{chr(upper)}' for lower, upper in _avoid_compatibility_chars])}]"
+)
 
 
 def escape_illegal_xml_chars(s: str) -> str:
diff --git a/pytest-embedded/pytest_embedded/utils.py b/pytest-embedded/pytest_embedded/utils.py
index ab87c148..9480c6d2 100644
--- a/pytest-embedded/pytest_embedded/utils.py
+++ b/pytest-embedded/pytest_embedded/utils.py
@@ -1,10 +1,12 @@
 import dataclasses
+import importlib
 import logging
 import os
 import re
 import typing as t
 
-from . import App
+if t.TYPE_CHECKING:
+    from . import App
 
 #############
 # Constants #
@@ -176,14 +178,14 @@ def drop_port_target_cache(self, port: str) -> None:
         except KeyError:
             logging.warning('no port-target cache with port %s', port)
 
-    def hit_port_app_cache(self, port: str, app: App) -> bool:
+    def hit_port_app_cache(self, port: str, app: 'App') -> bool:
         if self.port_app_cache.get(port, None) == app.binary_path:
             logging.debug('hit port-app cache: %s - %s', port, app.binary_path)
             return True
 
         return False
 
-    def set_port_app_cache(self, port: str, app: App) -> None:
+    def set_port_app_cache(self, port: str, app: 'App') -> None:
         self.port_app_cache[port] = app.binary_path
         logging.debug('set port-app cache: %s - %s', port, app.binary_path)
 
@@ -193,3 +195,53 @@ def drop_port_app_cache(self, port: str) -> None:
             logging.debug('drop port-app cache with port %s', port)
         except KeyError:
             logging.warning('no port-app cache with port %s', port)
+
+
+_ModuleType = type(importlib)
+
+
+def lazy_load(
+    base_module: _ModuleType, name_obj_dict: t.Dict[str, t.Any], obj_module_dict: t.Dict[str, str]
+) -> t.Callable[[str], t.Any]:
+    """
+    use __getattr__ in the __init__.py file to lazy load some objects
+
+    Args:
+        base_module (ModuleType): base package module
+        name_obj_dict (dict[str, any]): name, real object dict, used to store real objects,
+            no need to add lazy-load objects
+        obj_module_dict (dict[str, str]): dict of object name and module name
+
+    Returns:
+        __getattr__ function
+
+    Examples:
+        ```python
+        __getattr__ = lazy_load(
+            importlib.import_module(__name__),
+            {
+                'IdfApp': IdfApp,
+                'LinuxDut': LinuxDut,
+                'LinuxSerial': LinuxSerial,
+                'CaseTester': CaseTester,
+            },
+            {
+                'IdfSerial': '.serial',
+                'IdfDut': '.dut',
+            },
+        )
+        ```
+    """
+
+    def __getattr__(object_name):
+        if object_name in name_obj_dict:
+            return name_obj_dict[object_name]
+        elif object_name in obj_module_dict:
+            module = importlib.import_module(obj_module_dict[object_name], base_module.__name__)
+            imported = getattr(module, object_name)
+            name_obj_dict[object_name] = imported
+            return imported
+        else:
+            raise AttributeError('Attribute %s not found in module %s', object_name, base_module.__name__)
+
+    return __getattr__