Repository: dlt-hub/dlt

Issue:
dlt.pipeline destination bug found
### dlt version

1.6.1

### Describe the problem

I upgraded from dlt 1.4.1 to 1.6.1 and i believe some behaviour has changed in the way dlt is auto-loading the destination credentials when dlt.pipeline is invoked.

For context, I dockerise all of my pipelines after they work locally. This issue was only presented to me **after** the pipeline had been dockerised. I have no issues doing this with 5 other pipelines that have been constructed the same way under previous versions of dlt.

My destination and other credentials are formatted correctly in my locally .dlt/secrets.toml file and have no problem running in outside of a container (just virtual conda env), and everything works as it should when I change my dlt version to 1.4.1 when i build the docker image.

Here is the error traceback:


```  File "//mysql_database_pipeline.py", line 19, in load_refresh_tables
    pipeline = dlt.pipeline(pipeline_name="mysql_refresh_tables", destination='postgres', dataset_name="public", dev_mode=False)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/dlt/common/configuration/inject.py", line 259, in _wrap
    return f(*bound_args.args, **bound_args.kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/dlt/pipeline/__init__.py", line 148, in pipeline
    destination = Destination.from_reference(
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/dlt/common/destination/reference.py", line 241, in from_reference
    return DestinationReference.from_reference(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/dlt/common/destination/reference.py", line 343, in from_reference
    factory = cls.find(ref)
              ^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/dlt/common/destination/reference.py", line 298, in find
    refs = cls.to_fully_qualified_refs(ref)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/dlt/common/destination/reference.py", line 285, in to_fully_qualified_refs
    for ref_prefix in get_plugin_modules():
                      ^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/dlt/common/runtime/run_context.py", line 176, in get_plugin_modules
    ctx_module = current().module
                 ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/dlt/common/runtime/run_context.py", line 64, in module
    return self.import_run_dir_module(self.run_dir)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/dlt/common/runtime/run_context.py", line 98, in import_run_dir_module
    m_ = importlib.import_module(os.path.basename(run_dir))
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1201, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1114, in _sanity_check
ValueError: Empty module name```

### Expected behavior

_No response_

### Steps to reproduce

Build simple mysql -> postgres sync replicator. 
Build a slim docker image with just the following:

```
FROM python:3.11-slim-buster 
COPY /mysql/mysql_requirements.txt /mysql/mysql_requirements.txt
RUN pip install --upgrade pip
RUN pip install -r /mysql/mysql_requirements.txt

COPY . .
ENTRYPOINT ["python", "-u", "mysql_database_pipeline.py"]
```

Where the `mysql_database_pipeline.py` should look something like:
```
    # Create a pipeline
    pipeline = dlt.pipeline(pipeline_name="mysql_refresh_tables", destination='postgres', dataset_name="public", dev_mode=False)
    credentials = dlt.secrets.get('sources.sql_database.credentials')

    # Load some other tables with replace write disposition. This overwrites the existing tables in destination
    refresh_sources = []
    table = 'table_name'
    refresh_source = sql_table(credentials, table=table)
    refresh_sources.append(refresh_source)
    info = pipeline.run(refresh_sources, write_disposition="replace")
```

The mysql_requirements.txt file that **works**:
(change dlt==1.4.1 to dlt=1.6.1 to get the bug)

```certifi==2025.1.31
chardet==5.2.0
charset-normalizer==3.4.1
click==8.1.8
**dlt==1.4.1**
fsspec==2025.2.0
gitdb==4.0.12
GitPython==3.1.44
giturlparse==0.12.0
greenlet==3.1.1
hexbytes==1.3.0
humanize==4.11.0
idna==3.10
jsonpath-ng==1.7.0
makefun==1.15.6
markdown-it-py==3.0.0
mdurl==0.1.2
orjson==3.10.15
packaging==24.2
pathvalidate==3.2.3
pendulum==3.0.0
pluggy==1.5.0
ply==3.11
psycopg2-binary==2.9.10
Pygments==2.19.1
PyMySQL==1.1.1
python-dateutil==2.9.0.post0
pytz==2025.1
PyYAML==6.0.2
requests==2.32.3
requirements-parser==0.11.0
rich==13.9.4
rich-argparse==1.6.0
semver==3.0.4
simplejson==3.19.3
six==1.17.0
smmap==5.0.2
SQLAlchemy==2.0.38
tenacity==9.0.0
time-machine==2.16.0
tomlkit==0.13.2
types-setuptools==75.8.0.20250110
typing_extensions==4.12.2
tzdata==2025.1
urllib3==2.3.0
```

### Operating system

Linux

### Runtime environment

Docker, Docker Compose

### Python version

3.11

### dlt data source

mysql

### dlt destination

Postgres

### Other deployment details

_No response_

### Additional information

_No response_

Discussion:
@rudolfix: @damiah this is a bug, thanks! we'll release a fix tomorrow in an alpha if you want to check it

Solution patch:
diff --git a/dlt/common/runtime/run_context.py b/dlt/common/runtime/run_context.py
index 488dc7a4..80690e69 100644
--- a/dlt/common/runtime/run_context.py
+++ b/dlt/common/runtime/run_context.py
@@ -98,8 +98,12 @@ class RunContext(SupportsRunContext):
         """Returns a top Python module of the project (if importable)"""
         import importlib
 
+        # trailing separator will be removed by abspath
         run_dir = os.path.abspath(run_dir)
-        m_ = importlib.import_module(os.path.basename(run_dir))
+        base_dir = os.path.basename(run_dir)
+        if not base_dir:
+            raise ImportError(f"run dir {run_dir} looks like filesystem root")
+        m_ = importlib.import_module(base_dir)
         if m_.__file__ and m_.__file__.startswith(run_dir):
             return m_
         else:
@@ -172,7 +176,7 @@ def is_folder_writable(path: str) -> bool:
 def get_plugin_modules() -> List[str]:
     """Return top level module names of all discovered plugins, including `dlt`.
 
-    If current run context is a top levle module it is also included, otherwise empty string.
+    If current run context is a top level module it is also included, otherwise empty string.
     """
     from dlt.common.configuration.plugins import PluginContext
 
diff --git a/docs/website/docs/general-usage/http/rest-client.md b/docs/website/docs/general-usage/http/rest-client.md
index 53e1a2b7..1bbb3894 100644
--- a/docs/website/docs/general-usage/http/rest-client.md
+++ b/docs/website/docs/general-usage/http/rest-client.md
@@ -1,7 +1,7 @@
 ---
 title: RESTClient
 description: Learn how to use the RESTClient class to interact with RESTful APIs
-keywords: [api, http, rest, request, extract, restclient, client, pagination, json, response, data_selector, session, auth, paginator, JSONLinkPaginator, headerlinkpaginator, offsetpaginator, jsonresponsecursorpaginator, queryparampaginator, bearer, token, authentication]
+keywords: [api, http, rest, request, extract, restclient, client, pagination, json, response, data_selector, session, auth, paginator, JSONLinkPaginator, headerlinkpaginator, offsetpaginator, jsonresponsecursorpaginator, queryparampaginator, bearer, token, authentication, headercursorpaginator]
 ---
 
 The `RESTClient` class offers an interface for interacting with RESTful APIs, including features like:
@@ -114,6 +114,7 @@ Paginators are used to handle paginated responses. The `RESTClient` class comes
 - [OffsetPaginator](#offsetpaginator) - pagination based on offset and limit query parameters.
 - [PageNumberPaginator](#pagenumberpaginator) - pagination based on page numbers.
 - [JSONResponseCursorPaginator](#jsonresponsecursorpaginator) - pagination based on a cursor in the JSON response.
+- [HeaderCursorPaginator](#headercursorpaginator) - pagination based on a cursor in the response headers.
 
 If the API uses a non-standard pagination, you can [implement a custom paginator](#implementing-a-custom-paginator) by subclassing the `BasePaginator` class.
 
@@ -327,6 +328,39 @@ client = RESTClient(
 )
 ```
 
+#### HeaderCursorPaginator
+
+`HeaderCursorPaginator` handles pagination based on a cursor in the response headers.
+
+**Parameters:**
+
+- `cursor_key`: The key in the response headers that contains the cursor value. Defaults to `"next"`.
+- `cursor_param`: The query parameter used to send the cursor value in the next request. Defaults to `"cursor"`.
+
+**Example:**
+
+Consider an API endpoint `https://api.example.com/items` returning a response with a `NextPageToken` header containing the cursor for the next page:
+
+```text
+Content-Type: application/json
+NextPageToken: n3xtp4g3
+
+[
+    {"id": 1, "name": "item1"},
+    {"id": 2, "name": "item2"},
+    ...
+]
+```
+
+To paginate through responses from this API, use `HeaderCursorPaginator` with `cursor_key` set to `"NextPageToken"`:
+
+```py
+client = RESTClient(
+    base_url="https://api.example.com",
+    paginator=HeaderCursorPaginator(cursor_key="NextPageToken")
+)
+```
+
 ### Implementing a custom paginator
 
 When working with APIs that use non-standard pagination schemes, or when you need more control over the pagination process, you can implement a custom paginator by subclassing the `BasePaginator` class and implementing the methods `init_request`, `update_state`, and `update_request`.
diff --git a/docs/website/docs/reference/performance.md b/docs/website/docs/reference/performance.md
index 436cce76..dffa1ae4 100644
--- a/docs/website/docs/reference/performance.md
+++ b/docs/website/docs/reference/performance.md
@@ -319,3 +319,24 @@ volumes {
   }
 }
 ```
+## Handling storage limits
+
+If your storage reaches its limit, you are likely running dlt in a cloud environment with restricted disk space. To prevent issues, mount an external cloud storage location and set the `DLT_DATA_DIR` environment variable to point to it. This ensures that dlt uses the mounted storage as its data directory instead of local disk space.
+
+
+### Setting `DLT_DATA_DIR`
+
+You can configure `DLT_DATA_DIR` in your environment setup as follows:
+
+```py
+import os
+
+# Define the path to your mounted external storage
+data_dir = "/path/to/mounted/bucket/dlt_pipeline_data"
+
+# Set the DLT_DATA_DIR environment variable
+os.environ["DLT_DATA_DIR"] = data_dir
+
+# Rest of your pipeline code
+```
+This directs dlt to use the specified external storage for all data operations, preventing local storage constraints.
\ No newline at end of file


Test patch:
diff --git a/tests/common/runtime/test_run_context.py b/tests/common/runtime/test_run_context.py
index 21621fc6..cdfee590 100644
--- a/tests/common/runtime/test_run_context.py
+++ b/tests/common/runtime/test_run_context.py
@@ -10,6 +10,7 @@ from dlt.common.runtime.init import _INITIALIZED, apply_runtime_config, restore_
 from dlt.common.runtime.run_context import RunContext, get_plugin_modules, is_folder_writable
 from dlt.common.utils import set_working_dir
 
+import tests
 from tests.utils import MockableRunContext, TEST_STORAGE_ROOT
 
 
@@ -144,6 +145,14 @@ def test_context_switch_restores_logger() -> None:
         assert logger.LOGGER.name == "dlt-tests"
 
 
+def test_run_dir_module_import() -> None:
+    with pytest.raises(ImportError, match="filesystem root"):
+        RunContext.import_run_dir_module(os.sep)
+    with pytest.raises(ImportError):
+        RunContext.import_run_dir_module(os.path.join("tests", "no_such_module"))
+    assert RunContext.import_run_dir_module("tests") is tests
+
+
 def test_tmp_folder_writable() -> None:
     import tempfile