Repository: epoch8/datapipe (language: python)

Issue:
Move cli to datapipe-core



Discussion / hints:

Fix patch:
diff --git a/.gitattributes b/.gitattributes
index 5cb11f35..a140a9d6 100644
--- a/.gitattributes
+++ b/.gitattributes
@@ -1,3 +1,2 @@
-*.ipynb filter=lfs diff=lfs merge=lfs -text
 *.xml filter=lfs diff=lfs merge=lfs -text
 *.jp*g filter=lfs diff=lfs merge=lfs -text
diff --git a/.readthedocs.yml b/.readthedocs.yml
new file mode 100644
index 00000000..68f9b178
--- /dev/null
+++ b/.readthedocs.yml
@@ -0,0 +1,30 @@
+# .readthedocs.yaml
+# Read the Docs configuration file
+# See https://docs.readthedocs.io/en/stable/config-file/v2.html for details
+
+# Required
+version: 2
+
+build:
+  os: "ubuntu-22.04"
+  tools:
+    python: "3.10"
+  jobs:
+    post_create_environment:
+      # Install poetry
+      # https://python-poetry.org/docs/#installing-manually
+      - pip install poetry
+      # Tell poetry to not use a virtual environment
+      - poetry config virtualenvs.create false
+    post_install:
+      # Install dependencies with 'docs' dependency group
+      # https://python-poetry.org/docs/managing-dependencies/#dependency-groups
+      - poetry install --all-extras
+
+# Build documentation in the docs/ directory with Sphinx
+sphinx:
+   configuration: docs/source/conf.py
+
+# If using Sphinx, optionally build your docs in additional formats such as PDF
+# formats:
+#    - pdf
diff --git a/.vscode/settings.json b/.vscode/settings.json
index 47de3105..58c67a2e 100644
--- a/.vscode/settings.json
+++ b/.vscode/settings.json
@@ -24,8 +24,15 @@
     ],
     "[python]": {
         "editor.defaultFormatter": "ms-python.black-formatter",
-        "editor.formatOnSave": true
+        "editor.formatOnSave": true,
+        "editor.codeActionsOnSave": {
+            "source.organizeImports": true
+        },
     },
+    "isort.args": [
+        "--profile",
+        "black"
+    ],
     "githubIssues.queries": [
         {
             "label": "My Issues",
@@ -43,5 +50,5 @@
             "label": "Backlog",
             "query": "state:open repo:${owner}/${repository} sort:created-desc no:milestone"
         },
-    ]
+    ],
 }
\ No newline at end of file
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 60e28070..4fbf599c 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1,8 +1,22 @@
-# WIP 0.11.12
+# WIP 0.12.0
 
+## Breaking changes
+* Move cli from `datapipe-app` to `datapipe`
+* Remove separate `datapipe step status` command, now it's a flag: `datapipe
+  step list --status`
+* `DatatableTransform` moved from `datapipe.compute` to `datapipe.core_steps`
+* Remove `datapipe.metastore.MetaTableData` (nobody used it anyway)
+
+## New features
+
+* Add command `step run_changelist` to CLI
+
+## Refactorings
 * Add `labels` arg and property to `ComputeStep` base class
 * Add `labels` arg to `BatchTransform` and `BatchTransformStep`
 * Add `labels` arg to `BatchGenerate` and `DatatableTransformStep`
+* Large refactoring, `ComputeStep` now contains pieces of overridable functions
+  for `run_full` and `run_changelist`
 * Add prototype events logging for steps, add
   `event_logger.log_step_full_complete`, add table `datapipe_step_events`
 
diff --git a/TODO b/TODO
deleted file mode 100644
index 7b320a91..00000000
--- a/TODO
+++ /dev/null
@@ -1,78 +0,0 @@
-v0.10:
-    ☐ Переписать `get_process_idx` #136 показывает, что там точно есть проблемы
-    ☐ Решить, точно ли `read_data` должно быть внешним параметром в `TableStoreFiledir` #132
-    ✔ Работающий Transform без пересекающихся ключей @done(21-11-22 19:36)
-    ✔ Аккуратный рефакторинг core_steps (функции run в Steps) @done(21-11-22 23:58)
-    ✔ BatchGenerate через DatatableTransform @done(21-11-22 23:21)
-    ✔ BatchTransform через DatatableTransform @done(21-11-22 23:58)
-    ✔ UpdateExternalTableStep через DatatableTransform @done(21-11-23 00:10)
-    ✔ Обобщить Transform и Generate через DTTransform @done(21-11-22 23:59)
-    ✔ Объединить DatatableTransformStep с ComputeStep @done(21-11-23 00:24)
-    ☐ Решить что делать с подавлением исключений в BatchGenerateStep
-    ✔ Вынести интеграцию с Labelstudio в отдельную библиотеку datapipe-moderation @done(22-03-20 23:04)
-    ✔ Убрать обязательные inputs, outputs у PipelineStep @done(21-11-20 19:06)
-    ✔ Убрать input_dts у BatchGenerateStep @done(21-11-20 19:06)
-    ☐ Реализовать автоименование у шагов
-    ✔ Добавить шаг, который работает на уровне всей DataTable, а не батчей изменений @done(21-11-22 20:37)
-    ✔ Ускорить обновление метадаты: заменить удаление/создание на апдейт @done
-
-v0.11:
-    ☐ Объединить шаг трансформации и объявление схем результирующих таблиц (сделать каталог опциональным)
-    ☐ Добавить схему данных как класс
-    ☐ Нормальное управление схемами (полная схема в Table, а не частичная в Store)
-    ☐ Причесать работу с primary_schema
-    ☐ Поддержка сквозной сущности по схеме идентификации
-
-Backlog:
-    ☐ Режим демона
-        ☐ datapipe-exporter для экспорта состояния таблиц
-    ☐ Внести интеграцию с Prefect в основную библиотеку
-    ☐ Реализовать компиляцию из пайплайна в Prefect flow
-    ☐ Считать хеш-сумму в Store
-    ☐ Сделать MetaTable декоратором над транзакцией (или DataTable?)
-    ☐ Сделать прогрессбар для gen_* шага
-    ☐ Сделать возможность передавать fsspec urls в TableStoreFiledir, ex: zip://{label}/{id}_test.jpg::./CATALOG_DIR/archive.zip
-    ☐ Сделать "контекст записи"/"контекст чтения" для TableDataStore чтобы можно было кешировать операции чтения
-    ☐ Сделать синхронизацию DataStore с метадатой, как отдельный шаг
-    ☐ Добавить возможность передавать аргументы в *Generate и *Transform
-
-Ветка Label Studio:
-    ☐ Починить баг с авторизацией в 1.0.2+ лейбел студии (в 1.0.1 работает)
-        UPD: Баг на их стороне, починить пока не смогу, см. https://github.com/heartexlabs/label-studio/issues/1049
-    ☐ Подумать об обработке Run в лейбел студии в случае, когда сервис не запущен
-        Сценарий: если LS будет уничтожен и пересоздан, то наша синхронизация не зальет туда задачи заново (возможно, с разметкой)
-    ☐ Идемпотентная запись задач в LS, если задача уже есть, то она должна обновляться, а не дублироваться
-
-Archive:
-  ✔ Перенести логику создания подшагов пайплайна в PipelineStep @done(21-11-20 16:22) @project(build-compute)
-  ✔ Слить dsl, step, compute @done(21-11-20 16:21) @project(build-compute)
-  ✔ Перенести RunConfig в run_config @done(21-11-20 16:22) @project(build-compute)
-  ✔ Добавить список фильтров как тип @done(21-11-20 16:44) @project(Filter)
-  ✔ Many-to-one, one-to-many @done(21-11-20 16:44) @project(TODO)
-  ✔ Переехать на SQLAlchemy 1.4 https://www.sqlalchemy.org/blog/2021/03/15/sqlalchemy-1.4.0-released/ @done(21-11-20 16:44) @project(TODO)
-  ✔ Добавить фильтрацию для gen_process_many @done(21-10-26 23:27) @project(Filter)
-  ✔ Объединить DataTable и MetaTable @done(21-10-03 19:07) @project(One-to-many)
-  ✔ Перенести создание MetaTable в DataStore @done(21-09-26 21:03) @project(One-to-many)
-  ✔ Перенести логику работы со списком таблиц из dsl.Catalog в DataStore @done(21-09-26 21:03) @project(One-to-many)
-  ✔ Передвинуть глобальные функции inc_process, gen_process вниз datatable @done(21-09-26 21:05) @project(One-to-many)
-  ✔ get_metadata @done(21-10-03 19:07) @project(One-to-many)
-  ✔ get_existing_idx @done(21-10-03 19:07) @project(One-to-many)
-  ✔ get_table_debug_info @done(21-10-03 19:07) @project(One-to-many)
-  ✔ store_chunk @done(21-10-03 19:07) @project(One-to-many)
-  ✔ Реализовать общие тесты для всех table_store @done(21-09-26 16:48) @project(One-to-many)
-  ✔ Реализовать честный апдейт данных в MetaTable @done(21-07-31 01:03) @project(One-to-many)
-  ✔ Вынести работу с БД в MetaTable отдельно от DBStore @done(21-07-24 21:40) @project(One-to-many)
-  ✔ Добавить поддержку дополнительных полей в MetaTable @done(21-09-26 14:46) @project(One-to-many)
-  ✔ Добавить индексацию Store на основе индексного датафрейма (а не индекса) @done(21-09-26 14:46) @project(One-to-many)
-  ✔ Перейти с id индекса, на произвольный индекс в DataTable @done(21-09-26 14:46) @project(One-to-many)
-  ✔ Удалить ConstIdx из Store @done(21-09-26 14:46) @project(TODO)
-  ✔ Не удалять метаданные при удалении строк @done(21-09-26 14:47) @project(TODO)
-  ✔ Поддержка внешних хранилищ для данных (папка с файлами) @project(TODO)
-  ✔ Полноценные тесты @project(TODO)
-  ✔ Использование конструктора запросов SQLAlchemy вместо сырого SQL @project(TODO)
-  ✔ Поддержка SQLite @project(TODO)
-  ✔ Добавить тест на sign up @project(Ветка Label Studio)
-  ✔ Унести Project Setting в Лейбел Студию внутрь @project(Ветка Label Studio)
-  ✔ Унести добавление unique_id в Лейбел Студию внутрь @project(Ветка Label Studio)
-  ✔ Переписать получение текущей разметки через yield @project(Ветка Label Studio)
-  ✔ Добавить параметры и тесты на data, annotations и predictions @project(Ветка Label Studio)
diff --git a/datapipe/cli.py b/datapipe/cli.py
new file mode 100644
index 00000000..92d165fd
--- /dev/null
+++ b/datapipe/cli.py
@@ -0,0 +1,423 @@
+import importlib.metadata as metadata
+import os.path
+import sys
+import time
+from typing import Dict, List, Optional, cast
+
+import click
+import rich
+from datapipe_app import DatapipeApp
+from opentelemetry import trace
+from opentelemetry.sdk.resources import SERVICE_NAME, Resource
+from opentelemetry.sdk.trace import TracerProvider
+from opentelemetry.sdk.trace.export import BatchSpanProcessor, ConsoleSpanExporter
+from rich import print as rprint
+
+from datapipe.compute import ComputeStep, run_steps, run_steps_changelist
+from datapipe.types import ChangeList, IndexDF, Labels
+
+tracer = trace.get_tracer("datapipe_app")
+
+rich.reconfigure(highlight=False)
+
+
+def load_pipeline(pipeline_name: str) -> DatapipeApp:
+    pipeline_split = pipeline_name.split(":")
+
+    if len(pipeline_split) == 1:
+        module_name = pipeline_split[0]
+        app_name = "app"
+    elif len(pipeline_split) == 2:
+        module_name, app_name = pipeline_split
+    else:
+        raise Exception(
+            f"Expected PIPELINE in format 'module:app' got '{pipeline_name}'"
+        )
+
+    from importlib import import_module
+
+    sys.path.append(os.getcwd())
+
+    pipeline_mod = import_module(module_name)
+    app = getattr(pipeline_mod, app_name)
+
+    assert isinstance(app, DatapipeApp)
+
+    return app
+
+
+def parse_labels(labels: str) -> Labels:
+    if labels == "":
+        return []
+
+    labels_list = []
+
+    for kv in labels.split(","):
+        if "=" not in kv:
+            raise Exception(
+                f"Expected labels in format 'key=value,key2=value2' got '{labels}'"
+            )
+
+        k, v = kv.split("=")
+        labels_list.append((k, v))
+
+    return labels_list
+
+
+def filter_steps_by_labels_and_name(
+    app: DatapipeApp, labels: Labels = [], name_prefix: str = ""
+) -> List[ComputeStep]:
+    res = []
+
+    for step in app.steps:
+        for k, v in labels:
+            if (k, v) not in step.labels:
+                break
+        else:
+            if step.name.startswith(name_prefix):
+                res.append(step)
+
+    return res
+
+
+@click.group()
+@click.option("--debug", is_flag=True, help="Log debug output")
+@click.option("--debug-sql", is_flag=True, help="Log SQL queries VERY VERBOSE")
+@click.option("--trace-stdout", is_flag=True, help="Log traces to console")
+@click.option("--trace-jaeger", is_flag=True, help="Enable tracing to Jaeger")
+@click.option(
+    "--trace-jaeger-host", type=click.STRING, default="localhost", help="Jaeger host"
+)
+@click.option("--trace-jaeger-port", type=click.INT, default=14268, help="Jaeger port")
+@click.option("--trace-gcp", is_flag=True, help="Enable tracing to Google Cloud Trace")
+@click.option("--pipeline", type=click.STRING, default="app")
+@click.pass_context
+def cli(
+    ctx: click.Context,
+    debug: bool,
+    debug_sql: bool,
+    trace_stdout: bool,
+    trace_jaeger: bool,
+    trace_jaeger_host: str,
+    trace_jaeger_port: int,
+    trace_gcp: bool,
+    pipeline: str,
+) -> None:
+    import logging
+
+    if debug:
+        datapipe_logger = logging.getLogger("datapipe")
+        datapipe_logger.setLevel(logging.DEBUG)
+
+        datapipe_core_steps_logger = logging.getLogger("datapipe.core_steps")
+        datapipe_core_steps_logger.setLevel(logging.DEBUG)
+
+        logging.basicConfig(level=logging.DEBUG)
+    else:
+        logging.basicConfig(level=logging.INFO)
+
+    if debug_sql:
+        logging.getLogger("sqlalchemy.engine").setLevel(logging.INFO)
+
+    trace.set_tracer_provider(
+        TracerProvider(resource=Resource.create({SERVICE_NAME: "datapipe"}))
+    )
+
+    if trace_stdout:
+        processor = BatchSpanProcessor(ConsoleSpanExporter())
+        trace.get_tracer_provider().add_span_processor(processor)  # type: ignore
+
+    if trace_jaeger:
+        from opentelemetry.exporter.jaeger.thrift import JaegerExporter  # type: ignore
+
+        # create a JaegerExporter
+        jaeger_exporter = JaegerExporter(
+            # configure agent
+            # agent_host_name='localhost',
+            # agent_port=6831,
+            # optional: configure also collector
+            collector_endpoint=f"http://{trace_jaeger_host}:{trace_jaeger_port}/api/traces?format=jaeger.thrift",
+            # username=xxxx, # optional
+            # password=xxxx, # optional
+            # max_tag_value_length=None # optional
+        )
+
+        # Create a BatchSpanProcessor and add the exporter to it
+        span_processor = BatchSpanProcessor(jaeger_exporter)
+
+        # add to the tracer
+        trace.get_tracer_provider().add_span_processor(span_processor)  # type: ignore
+
+    if trace_gcp:
+        from opentelemetry.exporter.cloud_trace import CloudTraceSpanExporter
+
+        cloud_trace_exporter = CloudTraceSpanExporter(
+            resource_regex=r".*",
+        )
+        trace.get_tracer_provider().add_span_processor(BatchSpanProcessor(cloud_trace_exporter))  # type: ignore
+
+    ctx.ensure_object(dict)
+    with tracer.start_as_current_span("init"):
+        ctx.obj["pipeline"] = load_pipeline(pipeline)
+
+
+@cli.group()
+def table():
+    pass
+
+
+@table.command()
+@click.pass_context
+def list(ctx: click.Context) -> None:
+    app: DatapipeApp = ctx.obj["pipeline"]
+
+    for table in sorted(app.catalog.catalog.keys()):
+        print(table)
+
+
+@table.command()
+@click.argument("table")
+@click.pass_context
+def reset_metadata(ctx: click.Context, table: str) -> None:
+    app: DatapipeApp = ctx.obj["pipeline"]
+
+    dt = app.catalog.get_datatable(app.ds, table)
+    dt.reset_metadata()
+
+
+@cli.command()
+@click.pass_context
+def run(ctx: click.Context) -> None:
+    app: DatapipeApp = ctx.obj["pipeline"]
+
+    with tracer.start_as_current_span("run"):
+        from datapipe.compute import run_steps
+
+        run_steps(app.ds, app.steps)
+
+
+@cli.group()
+def db():
+    pass
+
+
+@db.command()
+@click.pass_context
+def create_all(ctx: click.Context) -> None:
+    app: DatapipeApp = ctx.obj["pipeline"]
+
+    app.ds.meta_dbconn.sqla_metadata.create_all(app.ds.meta_dbconn.con)
+
+
+@cli.command()
+@click.option("--tables", type=click.STRING, default="*")
+@click.option("--fix", is_flag=True, type=click.BOOL, default=False)
+@click.pass_context
+def lint(ctx: click.Context, tables: str, fix: bool) -> None:
+    app: DatapipeApp = ctx.obj["pipeline"]
+
+    from . import lints
+
+    checks = [
+        lints.LintDeleteTSIsNewerThanUpdateOrProcess(),
+        lints.LintDataWOMeta(),
+    ]
+
+    tables_from_catalog = app.catalog.catalog.keys()
+    print(f"Pipeline contains {len(tables_from_catalog)} tables")
+
+    if tables == "*":
+        tables_to_process = tables_from_catalog
+    else:
+        tables_to_process = tables.split(",")
+
+    for table_name in sorted(tables_to_process):
+        print(f"Checking '{table_name}': ", end="")
+
+        dt = app.catalog.get_datatable(app.ds, table_name)
+
+        errors = []
+
+        for check in checks:
+            (status, msg) = check.check(dt)
+
+            if status == lints.LintStatus.OK:
+                print(".", end="")
+            elif status == lints.LintStatus.SKIP:
+                print("S", end="")
+            elif status == lints.LintStatus.FAIL:
+                rprint("[red]F[/red]", end="")
+                errors.append((check, msg))
+
+        if len(errors) == 0:
+            rprint("[green] ok[/green]")
+        else:
+            rprint("[red] FAIL[/red]")
+            for check, msg in errors:
+                print(f" * {check.desc}: {msg}", end="")
+
+                if fix:
+                    try:
+                        (fix_status, fix_msg) = check.fix(dt)
+                        if fix_status == lints.LintStatus.OK:
+                            rprint("[green]... FIXED[/green]", end="")
+                        elif fix_status == lints.LintStatus.SKIP:
+                            rprint("[yellow]... SKIPPED[/yellow]", end="")
+                        else:
+                            rprint("[red]... FAILED TO FIX[/red]", end="")
+
+                            if fix_msg:
+                                print(fix_msg, end="")
+                    except:  # noqa
+                        rprint("[red]... FAILED TO FIX[/red]", end="")
+
+                print()
+            print()
+
+
+@cli.group()
+@click.option("--labels", type=click.STRING, default="")
+@click.option("--name", type=click.STRING, default="")
+@click.pass_context
+def step(ctx: click.Context, labels: str, name: str):
+    app: DatapipeApp = ctx.obj["pipeline"]
+
+    labels_dict = parse_labels(labels)
+    steps = filter_steps_by_labels_and_name(app, labels=labels_dict, name_prefix=name)
+
+    ctx.obj["steps"] = steps
+
+
+def to_human_repr(step: ComputeStep, extra_args: Optional[Dict] = None) -> str:
+    res = []
+
+    res.append(f"[green][bold]{step.name}[/bold][/green] ({step.__class__.__name__})")
+
+    if step.labels:
+        labels = " ".join([f"[magenta]{k}={v}[/magenta]" for (k, v) in step.labels])
+        res.append(f"  labels: {labels}")
+
+    if inputs_arr := [i.name for i in step.input_dts]:
+        inputs = ", ".join(inputs_arr)
+        res.append(f"  inputs: {inputs}")
+
+    if outputs_arr := [i.name for i in step.output_dts]:
+        outputs = ", ".join(outputs_arr)
+        res.append(f"  outputs: {outputs}")
+
+    if extra_args is not None:
+        for k, v in extra_args.items():
+            res.append(f"  {k}: {v}")
+
+    return "\n".join(res)
+
+
+@step.command()  # type: ignore
+@click.option("--status", is_flag=True, type=click.BOOL, default=False)
+@click.pass_context
+def list(ctx: click.Context, status: bool) -> None:  # noqa
+    app: DatapipeApp = ctx.obj["pipeline"]
+    steps: List[ComputeStep] = ctx.obj["steps"]
+
+    for step in steps:
+        extra_args = {}
+
+        if status:
+            if len(step.input_dts) > 0:
+                try:
+                    changed_idx_count = app.ds.get_changed_idx_count(
+                        inputs=step.input_dts,
+                        outputs=step.output_dts,
+                    )
+
+                    if changed_idx_count > 0:
+                        extra_args[
+                            "changed_idx_count"
+                        ] = f"[red]{changed_idx_count}[/red]"
+
+                except NotImplementedError:
+                    # Currently we do not support empty join_keys
+                    extra_args["changed_idx_count"] = "[red]N/A[/red]"
+
+        rprint(to_human_repr(step, extra_args=extra_args))
+        rprint("")
+
+
+@step.command()  # type: ignore
+@click.option("--loop", is_flag=True, default=False, help="Run continuosly in a loop")
+@click.option(
+    "--loop-delay", type=click.INT, default=30, help="Delay between loops in seconds"
+)
+@click.pass_context
+def run(ctx: click.Context, loop: bool, loop_delay: int) -> None:  # noqa
+    app: DatapipeApp = ctx.obj["pipeline"]
+    steps_to_run: List[ComputeStep] = ctx.obj["steps"]
+    steps_to_run_names = [f"'{i.name}'" for i in steps_to_run]
+    print(f"Running following steps: {', '.join(steps_to_run_names)}")
+
+    while True:
+        if len(steps_to_run) > 0:
+            run_steps(app.ds, steps_to_run)
+
+        if not loop:
+            break
+        else:
+            print(f"Loop ended, sleeping {loop_delay}s...")
+            time.sleep(loop_delay)
+            print("\n\n")
+
+
+@step.command()  # type: ignore
+@click.option("--loop", is_flag=True, default=False, help="Run continuosly in a loop")
+@click.option(
+    "--loop-delay", type=click.INT, default=1, help="Delay between loops in seconds"
+)
+@click.pass_context
+@click.option("--chunk-size", type=click.INT, default=1000)
+def run_changelist(
+    ctx: click.Context, loop: bool, loop_delay: int, chunk_size: int
+) -> None:
+    app: DatapipeApp = ctx.obj["pipeline"]
+    steps_to_run: List[ComputeStep] = ctx.obj["steps"]
+    steps_to_run_names = [f"'{i.name}'" for i in steps_to_run]
+    print(
+        f"Running following steps: {', '.join(steps_to_run_names)} with {chunk_size=}"
+    )
+    idx_gen, cnt = None, None
+    while True:
+        if len(steps_to_run) > 0:
+            if idx_gen is None:
+                idx_count, idx_gen = app.ds.get_full_process_ids(
+                    inputs=steps_to_run[0].input_dts,
+                    outputs=steps_to_run[0].output_dts,
+                    chunk_size=chunk_size,
+                )
+                cnt = 0
+            try:
+                idx = next(idx_gen)
+                cl = ChangeList()
+                for input_dt in steps_to_run[0].input_dts:
+                    cl.append(input_dt.name, cast(IndexDF, input_dt.get_data(idx=idx)))
+                run_steps_changelist(app.ds, steps_to_run, cl)
+            except StopIteration:
+                idx_gen = None
+                cnt = 0
+        if idx_gen is not None and cnt is not None:
+            print(f"Chunk {cnt}/{idx_count} ended")
+            cnt += 1
+        else:
+            if not loop:
+                break
+            else:
+                print(f"All chunks ended, sleeping {loop_delay}s...")
+                time.sleep(loop_delay)
+                print("\n\n")
+
+
+for entry_point in metadata.entry_points().get("datapipe.cli", []):
+    register_commands = entry_point.load()
+    register_commands(cli)
+
+
+def main():
+    cli(auto_envvar_prefix="DATAPIPE")
diff --git a/datapipe/compute.py b/datapipe/compute.py
index 41f18600..5f7bce0a 100644
--- a/datapipe/compute.py
+++ b/datapipe/compute.py
@@ -1,15 +1,17 @@
 import hashlib
 import logging
+import time
 from abc import ABC, abstractmethod
 from dataclasses import dataclass
-from typing import Any, Dict, List, Optional, Protocol
+from typing import Dict, Iterable, List, Optional, Tuple
 
+import tqdm
 from opentelemetry import trace
 
 from datapipe.datatable import DataStore, DataTable
 from datapipe.run_config import RunConfig
 from datapipe.store.table_store import TableStore
-from datapipe.types import ChangeList
+from datapipe.types import ChangeList, DataDF, IndexDF, Labels, TransformResult
 
 logger = logging.getLogger("datapipe.compute")
 tracer = trace.get_tracer("datapipe.compute")
@@ -39,68 +41,7 @@ def get_datatable(self, ds: DataStore, name: str) -> DataTable:
         return ds.get_or_create_table(name=name, table_store=self.catalog[name].store)
 
 
-@dataclass
-class Pipeline:
-    steps: List["PipelineStep"]
-
-
-class PipelineStep(ABC):
-    """
-    Пайплайн описывается через PipelineStep.
-
-    Для выполнения у каждого pipeline_step выполняется build_compute на
-    основании которых создается граф вычислений.
-    """
-
-    @abstractmethod
-    def build_compute(self, ds: DataStore, catalog: Catalog) -> List["ComputeStep"]:
-        raise NotImplementedError
-
-
-class DatatableTransformFunc(Protocol):
-    __name__: str
-
-    def __call__(
-        self,
-        ds: DataStore,
-        input_dts: List[DataTable],
-        output_dts: List[DataTable],
-        run_config: Optional[RunConfig],
-        # Возможно, лучше передавать как переменную, а не  **
-        **kwargs,
-    ) -> None:
-        ...
-
-
-class DatatableTransform(PipelineStep):
-    def __init__(
-        self,
-        func: DatatableTransformFunc,
-        inputs: List[str],
-        outputs: List[str],
-        check_for_changes: bool = True,
-        kwargs: Optional[Dict[str, Any]] = None,
-    ) -> None:
-        self.func = func
-        self.inputs = inputs
-        self.outputs = outputs
-        self.check_for_changes = check_for_changes
-        self.kwargs = kwargs
-
-    def build_compute(self, ds: DataStore, catalog: Catalog) -> List["ComputeStep"]:
-        return [
-            DatatableTransformStep(
-                name=self.func.__name__,  # type: ignore # mypy bug: https://github.com/python/mypy/issues/10976
-                input_dts=[catalog.get_datatable(ds, i) for i in self.inputs],
-                output_dts=[catalog.get_datatable(ds, i) for i in self.outputs],
-                func=self.func,
-                kwargs=self.kwargs,
-                check_for_changes=self.check_for_changes,
-            )
-        ]
-
-
-class ComputeStep(ABC):
+class ComputeStep:
     """
     Шаг вычислений в графе вычислений.
 
@@ -116,16 +57,24 @@ class ComputeStep(ABC):
     количество батчей, которые покрывают все измененные индексы.
     """
 
-    def __init__(self, name: str, labels: Optional[Dict[str, str]] = None) -> None:
+    def __init__(
+        self,
+        name: str,
+        input_dts: List[DataTable],
+        output_dts: List[DataTable],
+        labels: Optional[Labels] = None,
+    ) -> None:
         self._name = name
+        self.input_dts = input_dts
+        self.output_dts = output_dts
         self._labels = labels
 
     def get_name(self) -> str:
         ss = [
             self.__class__.__name__,
             self._name,
-            *[i.name for i in self.get_input_dts()],
-            *[o.name for o in self.get_output_dts()],
+            *[i.name for i in self.input_dts],
+            *[o.name for o in self.output_dts],
         ]
 
         m = hashlib.shake_128()
@@ -138,20 +87,13 @@ def name(self) -> str:
         return self.get_name()
 
     @property
-    def labels(self) -> Dict[str, str]:
-        return self._labels if self._labels else {}
-
-    @abstractmethod
-    def get_input_dts(self) -> List[DataTable]:
-        pass
-
-    @abstractmethod
-    def get_output_dts(self) -> List[DataTable]:
-        pass
+    def labels(self) -> Labels:
+        return self._labels if self._labels else []
 
-    def validate(self):
-        inp_p_keys_arr = [set(inp.primary_keys) for inp in self.get_input_dts() if inp]
-        out_p_keys_arr = [set(out.primary_keys) for out in self.get_output_dts() if out]
+    # TODO: move to lints
+    def validate(self) -> None:
+        inp_p_keys_arr = [set(inp.primary_keys) for inp in self.input_dts if inp]
+        out_p_keys_arr = [set(out.primary_keys) for out in self.output_dts if out]
 
         inp_p_keys = set.intersection(*inp_p_keys_arr) if len(inp_p_keys_arr) else set()
         out_p_keys = set.intersection(*out_p_keys_arr) if len(out_p_keys_arr) else set()
@@ -159,13 +101,13 @@ def validate(self):
 
         key_to_column_type_inp = {
             column.name: type(column.type)
-            for inp in self.get_input_dts()
+            for inp in self.input_dts
             for column in inp.primary_schema
             if column.name in join_keys
         }
         key_to_column_type_out = {
             column.name: type(column.type)
-            for inp in self.get_output_dts()
+            for inp in self.output_dts
             for column in inp.primary_schema
             if column.name in join_keys
         }
@@ -177,79 +119,221 @@ def validate(self):
                     f"{key_to_column_type_inp[key]} != {key_to_column_type_out[key]}"
                 )
 
-    def run_full(self, ds: DataStore, run_config: Optional[RunConfig] = None) -> None:
-        pass
+    def get_full_process_ids(
+        self,
+        ds: DataStore,
+        run_config: Optional[RunConfig] = None,
+    ) -> Tuple[int, Iterable[IndexDF]]:
+        raise NotImplementedError()
 
-    def run_changelist(
+    def get_change_list_process_ids(
         self,
         ds: DataStore,
-        changelist: ChangeList,
+        change_list: ChangeList,
         run_config: Optional[RunConfig] = None,
-    ) -> ChangeList:
-        return ChangeList()
+    ) -> Tuple[int, Iterable[IndexDF]]:
+        raise NotImplementedError()
 
+    def get_batch_input_dfs(
+        self,
+        ds: DataStore,
+        idx: IndexDF,
+        run_config: Optional[RunConfig] = None,
+    ) -> List[DataDF]:
+        return [inp.get_data(idx) for inp in self.input_dts]
 
-# TODO move to `core_steps`
-class DatatableTransformStep(ComputeStep):
-    def __init__(
+    def process_batch_dfs(
         self,
-        name: str,
-        input_dts: List[DataTable],
-        output_dts: List[DataTable],
-        func: DatatableTransformFunc,
-        kwargs: Optional[Dict[str, Any]] = None,
-        check_for_changes: bool = True,
-        labels: Optional[Dict[str, str]] = None,
-    ) -> None:
-        ComputeStep.__init__(self, name, labels=labels)
+        ds: DataStore,
+        idx: IndexDF,
+        input_dfs: List[DataDF],
+        run_config: Optional[RunConfig] = None,
+    ) -> TransformResult:
+        raise NotImplementedError()
 
-        self.input_dts = input_dts
-        self.output_dts = output_dts
+    def process_batch_dts(
+        self,
+        ds: DataStore,
+        idx: IndexDF,
+        run_config: Optional[RunConfig] = None,
+    ) -> Optional[TransformResult]:
+        with tracer.start_as_current_span("get input data"):
+            input_dfs = self.get_batch_input_dfs(ds, idx, run_config)
+
+        if sum(len(j) for j in input_dfs) == 0:
+            return None
+
+        with tracer.start_as_current_span("run transform"):
+            output_dfs = self.process_batch_dfs(
+                ds=ds,
+                idx=idx,
+                input_dfs=input_dfs,
+                run_config=run_config,
+            )
+
+        return output_dfs
 
-        self.func = func
-        self.kwargs = kwargs or {}
-        self.check_for_changes = check_for_changes
+    def store_batch_result(
+        self,
+        ds: DataStore,
+        idx: IndexDF,
+        output_dfs: Optional[TransformResult],
+        process_ts: float,
+        run_config: Optional[RunConfig] = None,
+    ) -> ChangeList:
+        changes = ChangeList()
+
+        if output_dfs is not None:
+            with tracer.start_as_current_span("store output batch"):
+                if isinstance(output_dfs, (list, tuple)):
+                    assert len(output_dfs) == len(self.output_dts)
+                else:
+                    assert len(self.output_dts) == 1
+                    output_dfs = [output_dfs]
+
+                for k, res_dt in enumerate(self.output_dts):
+                    # Берем k-ое значение функции для k-ой таблички
+                    # Добавляем результат в результирующие чанки
+                    change_idx = res_dt.store_chunk(
+                        data_df=output_dfs[k],
+                        processed_idx=idx,
+                        now=process_ts,
+                        run_config=run_config,
+                    )
 
-    def get_input_dts(self) -> List[DataTable]:
-        return self.input_dts
+                    changes.append(res_dt.name, change_idx)
 
-    def get_output_dts(self) -> List[DataTable]:
-        return self.output_dts
+        else:
+            with tracer.start_as_current_span("delete missing data from output"):
+                for k, res_dt in enumerate(self.output_dts):
+                    del_idx = res_dt.meta_table.get_existing_idx(idx)
 
-    def run_full(self, ds: DataStore, run_config: Optional[RunConfig] = None) -> None:
-        logger.info(f"Running: {self.name}")
+                    res_dt.delete_by_idx(del_idx, run_config=run_config)
 
-        if len(self.input_dts) > 0 and self.check_for_changes:
-            with tracer.start_as_current_span("check for changes"):
-                changed_idx_count = ds.get_changed_idx_count(
-                    inputs=self.input_dts,
-                    outputs=self.output_dts,
-                    run_config=run_config,
-                )
+                    changes.append(res_dt.name, del_idx)
 
-                if changed_idx_count == 0:
-                    logger.debug(
-                        f"Skipping {self.get_name()} execution - nothing to compute"
-                    )
+        return changes
 
-                    return
+    def store_batch_err(
+        self,
+        ds: DataStore,
+        idx: IndexDF,
+        e: Exception,
+        process_ts: float,
+        run_config: Optional[RunConfig] = None,
+    ) -> None:
+        logger.error(f"Process batch failed: {str(e)}")
+        ds.event_logger.log_exception(
+            e,
+            run_config=RunConfig.add_labels(
+                run_config,
+                {"idx": idx.to_dict(orient="records"), "process_ts": process_ts},
+            ),
+        )
+
+    def process_batch(
+        self,
+        ds: DataStore,
+        idx: IndexDF,
+        run_config: Optional[RunConfig] = None,
+    ) -> ChangeList:
+        with tracer.start_as_current_span("process batch"):
+            logger.debug(f"Idx to process: {idx.to_records()}")
 
-        run_config = RunConfig.add_labels(run_config, {"step_name": self.get_name()})
+            process_ts = time.time()
 
-        with tracer.start_as_current_span(f"Run {self.func.__name__}"):
             try:
-                self.func(
-                    ds=ds,
-                    input_dts=self.input_dts,
-                    output_dts=self.output_dts,
-                    run_config=run_config,
-                    **self.kwargs,
+                output_dfs = self.process_batch_dts(ds, idx, run_config)
+
+                return self.store_batch_result(
+                    ds, idx, output_dfs, process_ts, run_config
                 )
+
             except Exception as e:
-                logger.error(
-                    f"Datatable transform ({self.func.__name__}) run failed: {str(e)}"
-                )
-                ds.event_logger.log_exception(e, run_config=run_config)
+                self.store_batch_err(ds, idx, e, process_ts, run_config)
+
+                return ChangeList()
+
+    def run_full(
+        self,
+        ds: DataStore,
+        run_config: Optional[RunConfig] = None,
+    ) -> None:
+        logger.info(f"Running: {self.name}")
+        run_config = RunConfig.add_labels(run_config, {"step_name": self.name})
+
+        (idx_count, idx_gen) = self.get_full_process_ids(ds, run_config)
+
+        logger.info(f"Batches to process {idx_count}")
+
+        if idx_count is not None and idx_count == 0:
+            return
+
+        for idx in tqdm.tqdm(idx_gen, total=idx_count):
+            self.process_batch(
+                ds=ds,
+                idx=idx,
+                run_config=run_config,
+            )
+
+        ds.event_logger.log_step_full_complete(self.name)
+
+    def run_changelist(
+        self,
+        ds: DataStore,
+        change_list: ChangeList,
+        run_config: Optional[RunConfig] = None,
+    ) -> ChangeList:
+        logger.info(f"Running: {self.name}")
+        run_config = RunConfig.add_labels(run_config, {"step_name": self.name})
+
+        (idx_count, idx_gen) = self.get_change_list_process_ids(
+            ds, change_list, run_config
+        )
+
+        logger.info(f"Batches to process {idx_count}")
+
+        if idx_count is not None and idx_count == 0:
+            return ChangeList()
+
+        res_changelist = ChangeList()
+
+        for idx in tqdm.tqdm(idx_gen, total=idx_count):
+            changes = self.process_batch(
+                ds=ds,
+                idx=idx,
+                run_config=run_config,
+            )
+            res_changelist.extend(changes)
+
+        return res_changelist
+
+
+class PipelineStep(ABC):
+    """
+    Пайплайн описывается через PipelineStep.
+
+    Для выполнения у каждого pipeline_step выполняется build_compute на
+    основании которых создается граф вычислений.
+    """
+
+    @abstractmethod
+    def build_compute(self, ds: DataStore, catalog: Catalog) -> List[ComputeStep]:
+        raise NotImplementedError
+
+
+@dataclass
+class Pipeline:
+    steps: List[PipelineStep]
+
+
+class DatapipeApp:
+    def __init__(self, ds: DataStore, catalog: Catalog, pipeline: Pipeline):
+        self.ds = ds
+        self.catalog = catalog
+        self.pipeline = pipeline
+
+        self.steps = build_compute(ds, catalog, pipeline)
 
 
 def build_compute(
@@ -263,6 +347,7 @@ def build_compute(
         for step in pipeline.steps:
             compute_pipeline.extend(step.build_compute(ds, catalog))
 
+        # TODO move to lints
         for compute_step in compute_pipeline:
             compute_step.validate()
 
@@ -278,17 +363,16 @@ def print_compute(steps: List[ComputeStep]) -> None:
 def run_steps(
     ds: DataStore, steps: List[ComputeStep], run_config: Optional[RunConfig] = None
 ) -> None:
-    with tracer.start_as_current_span("run_steps"):
-        for step in steps:
-            with tracer.start_as_current_span(
-                f"{step.get_name()} {[i.name for i in step.get_input_dts()]} -> {[i.name for i in step.get_output_dts()]}"
-            ):
-                logger.info(
-                    f"Running {step.get_name()} "
-                    f"{[i.name for i in step.get_input_dts()]} -> {[i.name for i in step.get_output_dts()]}"
-                )
+    for step in steps:
+        with tracer.start_as_current_span(
+            f"{step.get_name()} {[i.name for i in step.input_dts]} -> {[i.name for i in step.output_dts]}"
+        ):
+            logger.info(
+                f"Running {step.get_name()} "
+                f"{[i.name for i in step.input_dts]} -> {[i.name for i in step.output_dts]}"
+            )
 
-                step.run_full(ds, run_config)
+        step.run_full(ds, run_config)
 
 
 def run_pipeline(
@@ -329,17 +413,21 @@ def run_steps_changelist(
                 for step in steps:
                     with tracer.start_as_current_span(
                         f"{step.get_name()} "
-                        f"{[i.name for i in step.get_input_dts()]} -> {[i.name for i in step.get_output_dts()]}"
+                        f"{[i.name for i in step.input_dts]} -> {[i.name for i in step.output_dts]}"
                     ):
                         logger.info(
                             f"Running {step.get_name()} "
-                            f"{[i.name for i in step.get_input_dts()]} -> {[i.name for i in step.get_output_dts()]}"
+                            f"{[i.name for i in step.input_dts]} -> {[i.name for i in step.output_dts]}"
                         )
 
-                        step_changes = step.run_changelist(
-                            ds, current_changes, run_config
-                        )
-                        next_changes.extend(step_changes)
+                        try:
+                            step_changes = step.run_changelist(
+                                ds, current_changes, run_config
+                            )
+                            next_changes.extend(step_changes)
+                        except NotImplementedError:
+                            # Some steps do not implement `.run_changelist`, that's ok
+                            pass
 
             current_changes = next_changes
             next_changes = ChangeList()
diff --git a/datapipe/core_steps.py b/datapipe/core_steps.py
index 9113a4f0..a1bd422f 100644
--- a/datapipe/core_steps.py
+++ b/datapipe/core_steps.py
@@ -1,300 +1,583 @@
+import itertools
 import logging
+import math
 import time
+from dataclasses import dataclass
 from typing import (
     Any,
+    Callable,
     Dict,
     Iterable,
     Iterator,
     List,
     Optional,
+    Protocol,
     Tuple,
-    Union,
-    Callable,
     cast,
 )
 
+import pandas as pd
 import tqdm
 from opentelemetry import trace
+from sqlalchemy import alias, and_, column, func, or_, select
 
-from datapipe.compute import (
-    Catalog,
-    ComputeStep,
-    DatatableTransformStep,
-    PipelineStep,
-    DatatableTransformFunc,
-)
+from datapipe.compute import Catalog, ComputeStep, PipelineStep
 from datapipe.datatable import DataStore, DataTable
-from datapipe.run_config import RunConfig
-from datapipe.types import ChangeList, DataDF, IndexDF
+from datapipe.metastore import MetaTable, TransformMetaTable
+from datapipe.run_config import LabelDict, RunConfig
+from datapipe.store.database import sql_apply_runconfig_filter
+from datapipe.types import (
+    ChangeList,
+    DataDF,
+    IndexDF,
+    Labels,
+    MetaSchema,
+    TransformResult,
+)
 
 logger = logging.getLogger("datapipe.core_steps")
 tracer = trace.get_tracer("datapipe.core_steps")
 
 
-BatchTransformFunc = Callable[..., Union[DataDF, List[DataDF], Tuple[DataDF, ...]]]
-
+class DatatableTransformFunc(Protocol):
+    __name__: str
 
-def do_batch_transform(
-    func: BatchTransformFunc,
-    ds: DataStore,
-    input_dts: List[DataTable],
-    output_dts: List[DataTable],
-    idx_gen: Iterable[IndexDF],
-    idx_count: Optional[int] = None,
-    kwargs: Optional[Dict[str, Any]] = None,
-    run_config: Optional[RunConfig] = None,
-) -> Iterator[ChangeList]:
-    """
-    Множественная инкрементальная обработка `input_dts' на основе изменяющихся индексов
-    """
-
-    logger.info(f"Batches to process {idx_count}")
-
-    if idx_count is not None and idx_count == 0:
-        # Nothing to process
-        return [ChangeList()]
-
-    for idx in tqdm.tqdm(idx_gen, total=idx_count):
-        with tracer.start_as_current_span("process batch"):
-            logger.debug(f"Idx to process: {idx.to_records()}")
-
-            with tracer.start_as_current_span("get input data"):
-                try:
-                    input_dfs = [inp.get_data(idx) for inp in input_dts]
-                except Exception as e:
-                    logger.error(f"Get input data failed: {str(e)}")
-                    ds.event_logger.log_exception(
-                        e,
-                        run_config=RunConfig.add_labels(
-                            run_config, {"idx": idx.to_dict(orient="records")}
-                        ),
-                    )
-
-                    continue
-
-            changes = ChangeList()
-
-            if sum(len(j) for j in input_dfs) > 0:
-                with tracer.start_as_current_span("run transform"):
-                    try:
-                        chunks_df = func(*input_dfs, **kwargs or {})
-                    except Exception as e:
-                        logger.error(f"Transform failed ({func.__name__}): {str(e)}")
-                        ds.event_logger.log_exception(
-                            e,
-                            run_config=RunConfig.add_labels(
-                                run_config, {"idx": idx.to_dict(orient="records")}
-                            ),
-                        )
-
-                        continue
-
-                if isinstance(chunks_df, (list, tuple)):
-                    assert len(chunks_df) == len(output_dts)
-                else:
-                    assert len(output_dts) == 1
-                    chunks_df = [chunks_df]
-
-                with tracer.start_as_current_span("store output batch"):
-                    try:
-                        for k, res_dt in enumerate(output_dts):
-                            # Берем k-ое значение функции для k-ой таблички
-                            # Добавляем результат в результирующие чанки
-                            change_idx = res_dt.store_chunk(
-                                data_df=chunks_df[k],
-                                processed_idx=idx,
-                                run_config=run_config,
-                            )
-
-                            changes.append(res_dt.name, change_idx)
-                    except Exception as e:
-                        logger.error(f"Store output batch failed: {str(e)}")
-                        ds.event_logger.log_exception(
-                            e,
-                            run_config=RunConfig.add_labels(
-                                run_config, {"idx": idx.to_dict(orient="records")}
-                            ),
-                        )
+    def __call__(
+        self,
+        ds: DataStore,
+        input_dts: List[DataTable],
+        output_dts: List[DataTable],
+        run_config: Optional[RunConfig],
+        # Возможно, лучше передавать как переменную, а не  **
+        **kwargs,
+    ) -> None:
+        ...
 
-                        continue
 
-            else:
-                with tracer.start_as_current_span("delete missing data from output"):
-                    for k, res_dt in enumerate(output_dts):
-                        del_idx = res_dt.meta_table.get_existing_idx(idx)
+# TODO подумать, может быть мы хотим дать возможность возвращать итератор TransformResult
+class DatatableBatchTransformFunc(Protocol):
+    __name__: str
 
-                        res_dt.delete_by_idx(del_idx, run_config=run_config)
+    def __call__(
+        self,
+        ds: DataStore,
+        idx: IndexDF,
+        input_dts: List[DataTable],
+        run_config: Optional[RunConfig] = None,
+        kwargs: Optional[Dict[str, Any]] = None,
+    ) -> TransformResult:
+        ...
 
-                        changes.append(res_dt.name, del_idx)
 
-            yield changes
+# TODO подумать, может быть мы хотим дать возможность возвращать итератор TransformResult
+BatchTransformFunc = Callable[..., TransformResult]
 
+BatchGenerateFunc = Callable[..., Iterator[TransformResult]]
 
-def do_full_batch_transform(
-    func: BatchTransformFunc,
-    ds: DataStore,
-    input_dts: List[DataTable],
-    output_dts: List[DataTable],
-    kwargs: Optional[Dict[str, Any]] = None,
-    chunk_size: int = 1000,
-    run_config: Optional[RunConfig] = None,
-) -> None:
-    with tracer.start_as_current_span("compute ids to process"):
-        idx_count, idx_gen = ds.get_full_process_ids(
-            inputs=input_dts,
-            outputs=output_dts,
-            chunk_size=chunk_size,
-            run_config=run_config,
-        )
 
-    gen = do_batch_transform(
-        func,
-        kwargs=kwargs,
-        ds=ds,
-        idx_count=idx_count,
-        idx_gen=idx_gen,
-        input_dts=input_dts,
-        output_dts=output_dts,
-        run_config=run_config,
-    )
+@dataclass
+class DatatableTransform(PipelineStep):
+    func: DatatableTransformFunc
+    inputs: List[str]
+    outputs: List[str]
+    check_for_changes: bool = True
+    kwargs: Optional[Dict[str, Any]] = None
+    labels: Optional[Labels] = None
 
-    for changes in gen:
-        pass
+    def build_compute(self, ds: DataStore, catalog: Catalog) -> List["ComputeStep"]:
+        return [
+            DatatableTransformStep(
+                name=self.func.__name__,
+                input_dts=[catalog.get_datatable(ds, i) for i in self.inputs],
+                output_dts=[catalog.get_datatable(ds, i) for i in self.outputs],
+                func=self.func,
+                kwargs=self.kwargs,
+                check_for_changes=self.check_for_changes,
+                labels=self.labels,
+            )
+        ]
 
 
-class BatchTransform(PipelineStep):
+class DatatableTransformStep(ComputeStep):
     def __init__(
         self,
-        func: BatchTransformFunc,
-        inputs: List[str],
-        outputs: List[str],
-        chunk_size: int = 1000,
-        kwargs: Optional[Dict[str, Any]] = None,
-        labels: Optional[Dict[str, str]] = None,
-    ):
+        name: str,
+        input_dts: List[DataTable],
+        output_dts: List[DataTable],
+        func: DatatableTransformFunc,
+        kwargs: Optional[Dict] = None,
+        check_for_changes: bool = True,
+        labels: Optional[Labels] = None,
+    ) -> None:
+        ComputeStep.__init__(self, name, input_dts, output_dts, labels)
+
         self.func = func
-        self.inputs = inputs
-        self.outputs = outputs
-        self.chunk_size = chunk_size
         self.kwargs = kwargs or {}
-        self.labels = labels
+        self.check_for_changes = check_for_changes
 
-    def build_compute(self, ds: DataStore, catalog: Catalog) -> List[ComputeStep]:
-        input_dts = [catalog.get_datatable(ds, name) for name in self.inputs]
-        output_dts = [catalog.get_datatable(ds, name) for name in self.outputs]
+    def run_full(self, ds: DataStore, run_config: Optional[RunConfig] = None) -> None:
+        logger.info(f"Running: {self.name}")
 
-        return [
-            BatchTransformStep(
-                f"{self.func.__name__}",  # type: ignore # mypy bug: https://github.com/python/mypy/issues/10976
-                input_dts=input_dts,
-                output_dts=output_dts,
-                func=self.func,
-                kwargs=self.kwargs,
-                chunk_size=self.chunk_size,
-                labels=self.labels,
-            )
-        ]
+        # TODO implement "watermark" system for tracking computation status in DatatableTransform
+        #
+        # if len(self.input_dts) > 0 and self.check_for_changes:
+        #     with tracer.start_as_current_span("check for changes"):
+        #         changed_idx_count = ds.get_changed_idx_count(
+        #             inputs=self.input_dts,
+        #             outputs=self.output_dts,
+        #             run_config=run_config,
+        #         )
+
+        #         if changed_idx_count == 0:
+        #             logger.debug(
+        #                 f"Skipping {self.get_name()} execution - nothing to compute"
+        #             )
+
+        #             return
+
+        run_config = RunConfig.add_labels(run_config, {"step_name": self.get_name()})
+
+        with tracer.start_as_current_span(f"Run {self.func}"):
+            try:
+                self.func(
+                    ds=ds,
+                    input_dts=self.input_dts,
+                    output_dts=self.output_dts,
+                    run_config=run_config,
+                    kwargs=self.kwargs,
+                )
+            except Exception as e:
+                logger.error(f"Datatable transform ({self.func}) run failed: {str(e)}")
+                ds.event_logger.log_exception(e, run_config=run_config)
 
 
-class BatchTransformStep(ComputeStep):
+def safe_func_name(func: Callable) -> str:
+    raw_name = func.__name__
+    if raw_name == "<lambda>":
+        return "lambda"
+    return raw_name
+
+
+class BaseBatchTransformStep(ComputeStep):
+    """
+    Abstract class for batch transform steps
+    """
+
     def __init__(
         self,
+        ds: DataStore,
         name: str,
         input_dts: List[DataTable],
         output_dts: List[DataTable],
-        func: BatchTransformFunc,
-        kwargs: Optional[Dict[str, Any]] = None,
+        transform_keys: Optional[List[str]] = None,
         chunk_size: int = 1000,
-        labels: Optional[Dict[str, str]] = None,
+        labels: Optional[Labels] = None,
     ) -> None:
-        ComputeStep.__init__(self, name=name, labels=labels)
+        ComputeStep.__init__(self, name, input_dts, output_dts, labels)
 
-        self.input_dts = input_dts
-        self.output_dts = output_dts
-
-        self.func = func
-        self.kwargs = kwargs or {}
         self.chunk_size = chunk_size
 
-    def get_input_dts(self) -> List[DataTable]:
-        return self.input_dts
+        self.transform_keys, self.transform_schema = self.compute_transform_schema(
+            [i.meta_table for i in input_dts],
+            [i.meta_table for i in output_dts],
+            transform_keys,
+        )
 
-    def get_output_dts(self) -> List[DataTable]:
-        return self.output_dts
+        self.meta_table = TransformMetaTable(
+            dbconn=ds.meta_dbconn,
+            name=f"{self.get_name()}_meta",
+            primary_schema=self.transform_schema,
+            create_table=ds.create_meta_table,
+        )
 
-    def run_full(self, ds: DataStore, run_config: Optional[RunConfig] = None) -> None:
-        logger.info(f"Running: {self.name}")
+    @classmethod
+    def compute_transform_schema(
+        cls,
+        input_mts: List[MetaTable],
+        output_mts: List[MetaTable],
+        transform_keys: Optional[List[str]],
+    ) -> Tuple[List[str], MetaSchema]:
+        # Hacky way to collect all the primary keys into a single set. Possible
+        # problem that is not handled here is that theres a possibility that the
+        # same key is defined differently in different input tables.
+        all_keys = {
+            col.name: col
+            for col in itertools.chain(
+                *(
+                    [dt.primary_schema for dt in input_mts]
+                    + [dt.primary_schema for dt in output_mts]
+                )
+            )
+        }
 
-        run_config = RunConfig.add_labels(run_config, {"step_name": self.name})
+        if transform_keys is not None:
+            return (transform_keys, [all_keys[k] for k in transform_keys])
 
-        with tracer.start_as_current_span("Get ids to process"):
-            idx_count, idx_gen = ds.get_full_process_ids(
+        assert len(input_mts) > 0
+
+        inp_p_keys = set.intersection(*[set(inp.primary_keys) for inp in input_mts])
+        assert len(inp_p_keys) > 0
+
+        if len(output_mts) == 0:
+            return (list(inp_p_keys), [all_keys[k] for k in inp_p_keys])
+
+        out_p_keys = set.intersection(*[set(out.primary_keys) for out in output_mts])
+        assert len(out_p_keys) > 0
+
+        inp_out_p_keys = set.intersection(inp_p_keys, out_p_keys)
+        assert len(inp_out_p_keys) > 0
+
+        return (list(inp_out_p_keys), [all_keys[k] for k in inp_out_p_keys])
+
+    def _build_changed_idx_sql(
+        self,
+        ds: DataStore,
+        run_config: Optional[RunConfig] = None,
+    ) -> Tuple[Iterable[str], select]:
+        if len(self.transform_keys) == 0:
+            raise NotImplementedError()
+
+        # TODO move to DBConn compatiblity layer
+        if ds.meta_dbconn.con.driver in ("sqlite", "pysqlite"):
+            greatest_func = func.max
+        else:
+            greatest_func = func.greatest
+
+        def _make_agg_cte(dt: DataTable, agg_fun, agg_col: str):
+            tbl = dt.meta_table.sql_table
+
+            # TODO add support for partially absent primary keys
+            key_cols = [column(k) for k in self.transform_keys]
+
+            sql = (
+                select(*key_cols + [agg_fun(tbl.c[agg_col]).label(agg_col)])
+                .select_from(tbl)
+                .group_by(*key_cols)
+            )
+
+            sql = sql_apply_runconfig_filter(sql, tbl, dt.primary_keys, run_config)
+
+            # TODO also return keys that were used for grouping
+            return sql.cte(name=f"{tbl.name}__{agg_col}")
+
+        def _make_agg_of_agg(ctes, agg_col):
+            assert len(ctes) > 0
+
+            if len(ctes) == 1:
+                return ctes[0]
+
+            if len(self.transform_keys) > 1:
+                coalesce_keys = [
+                    func.coalesce(*[subq.c[key] for subq in ctes]).label(key)
+                    for key in self.transform_keys
+                ]
+            else:
+                coalesce_keys = [subq.c[self.transform_keys[0]] for subq in ctes]
+
+            agg = greatest_func(*[subq.c[agg_col] for subq in ctes]).label(agg_col)
+
+            sql = select(*coalesce_keys + [agg]).select_from(ctes[0])
+
+            for cte in ctes[1:]:
+                sql = sql.outerjoin(
+                    cte,
+                    onclause=and_(
+                        *[ctes[0].c[key] == cte.c[key] for key in self.transform_keys]
+                    ),
+                    full=True,
+                )
+
+            return sql.cte(name=f"all__{agg_col}")
+
+        inp_ctes = [_make_agg_cte(tbl, func.max, "update_ts") for tbl in self.input_dts]
+
+        inp = _make_agg_of_agg(inp_ctes, "update_ts")
+
+        tr_tbl = self.meta_table.sql_table
+        out = (
+            select(*[column(k) for k in self.transform_keys] + [tr_tbl.c.process_ts])
+            .select_from(tr_tbl)
+            .where(tr_tbl.c.is_success == True)  # noqa
+            .group_by(*[column(k) for k in self.transform_keys])
+            .cte(name="transform")
+        )
+
+        sql = (
+            select(
+                *[
+                    func.coalesce(inp.c[key], out.c[key]).label(key)
+                    for key in self.transform_keys
+                ]
+            )
+            .select_from(inp)
+            .outerjoin(
+                out,
+                onclause=and_(
+                    *[inp.c[key] == out.c[key] for key in self.transform_keys]
+                ),
+            )
+            .where(
+                or_(
+                    inp.c.update_ts > out.c.process_ts,
+                    inp.c.update_ts == None,  # noqa
+                    out.c.process_ts == None,  # noqa
+                )
+            )
+        )
+
+        return (self.transform_keys, sql)
+
+    def get_changed_idx_count(
+        self,
+        ds: DataStore,
+        run_config: Optional[RunConfig] = None,
+    ) -> int:
+        _, sql = self._build_changed_idx_sql(ds, run_config=run_config)
+
+        idx_count = ds.meta_dbconn.con.execute(
+            select([func.count()]).select_from(
+                alias(sql.subquery(), name="union_select")
+            )
+        ).scalar()
+
+        return idx_count
+
+    def get_full_process_ids(
+        self,
+        ds: DataStore,
+        run_config: Optional[RunConfig] = None,
+    ) -> Tuple[int, Iterable[IndexDF]]:
+        """
+        Метод для получения перечня индексов для обработки.
+
+        Returns: (idx_size, iterator<idx_df>)
+
+        - idx_size - количество индексов требующих обработки
+        - idx_df - датафрейм без колонок с данными, только индексная колонка
+        """
+
+        with tracer.start_as_current_span("compute ids to process"):
+            if len(self.input_dts) == 0:
+                return (0, iter([]))
+
+            idx_count = self.get_changed_idx_count(
+                ds=ds,
+                run_config=run_config,
+            )
+
+            join_keys, u1 = self._build_changed_idx_sql(
+                ds=ds,
+                run_config=run_config,
+            )
+
+            # Список ключей из фильтров, которые нужно добавить в результат
+            extra_filters: LabelDict
+            if run_config is not None:
+                extra_filters = {
+                    k: v for k, v in run_config.filters.items() if k not in join_keys
+                }
+            else:
+                extra_filters = {}
+
+            def alter_res_df():
+                for df in pd.read_sql_query(
+                    u1, con=ds.meta_dbconn.con, chunksize=self.chunk_size
+                ):
+                    for k, v in extra_filters.items():
+                        df[k] = v
+
+                    yield df
+
+            return math.ceil(idx_count / self.chunk_size), alter_res_df()
+
+    def get_change_list_process_ids(
+        self,
+        ds: DataStore,
+        change_list: ChangeList,
+        run_config: Optional[RunConfig] = None,
+    ) -> Tuple[int, Iterable[IndexDF]]:
+        with tracer.start_as_current_span("compute ids to process"):
+            return ds.get_change_list_process_ids(
                 inputs=self.input_dts,
                 outputs=self.output_dts,
+                change_list=change_list,
                 chunk_size=self.chunk_size,
                 run_config=run_config,
             )
 
-        gen = do_batch_transform(
-            self.func,
-            ds=ds,
-            idx_count=idx_count,
-            idx_gen=idx_gen,
-            input_dts=self.input_dts,
-            output_dts=self.output_dts,
-            run_config=run_config,
-            kwargs=self.kwargs,
-        )
+    def store_batch_result(
+        self,
+        ds: DataStore,
+        idx: IndexDF,
+        output_dfs: Optional[TransformResult],
+        process_ts: float,
+        run_config: Optional[RunConfig] = None,
+    ) -> ChangeList:
+        res = super().store_batch_result(ds, idx, output_dfs, process_ts, run_config)
 
-        for changes in gen:
-            pass
+        self.meta_table.mark_rows_processed_success(
+            idx, process_ts=process_ts, run_config=run_config
+        )
 
-        ds.event_logger.log_step_full_complete(self.name)
+        return res
 
-    def run_changelist(
+    def store_batch_err(
         self,
         ds: DataStore,
-        change_list: ChangeList,
+        idx: IndexDF,
+        e: Exception,
+        process_ts: float,
         run_config: Optional[RunConfig] = None,
-    ) -> ChangeList:
-        run_config = RunConfig.add_labels(run_config, {"step_name": self.name})
+    ) -> None:
+        super().store_batch_err(ds, idx, e, process_ts, run_config)
 
-        idx_count, idx_gen = ds.get_change_list_process_ids(
-            inputs=self.input_dts,
-            outputs=self.output_dts,
-            change_list=change_list,
-            chunk_size=self.chunk_size,
+        self.meta_table.mark_rows_processed_error(
+            idx,
+            process_ts=process_ts,
+            error=str(e),
             run_config=run_config,
         )
 
-        gen = do_batch_transform(
-            self.func,
+
+@dataclass
+class DatatableBatchTransform(PipelineStep):
+    func: DatatableBatchTransformFunc
+    inputs: List[str]
+    outputs: List[str]
+    chunk_size: int = 1000
+    kwargs: Optional[Dict] = None
+    labels: Optional[Labels] = None
+
+    def build_compute(self, ds: DataStore, catalog: Catalog) -> List[ComputeStep]:
+        input_dts = [catalog.get_datatable(ds, name) for name in self.inputs]
+        output_dts = [catalog.get_datatable(ds, name) for name in self.outputs]
+
+        return [
+            DatatableBatchTransformStep(
+                ds=ds,
+                name=f"{self.func.__name__}",
+                func=self.func,
+                input_dts=input_dts,
+                output_dts=output_dts,
+                kwargs=self.kwargs,
+                chunk_size=self.chunk_size,
+                labels=self.labels,
+            )
+        ]
+
+
+class DatatableBatchTransformStep(BaseBatchTransformStep):
+    def __init__(
+        self,
+        ds: DataStore,
+        name: str,
+        func: DatatableBatchTransformFunc,
+        input_dts: List[DataTable],
+        output_dts: List[DataTable],
+        kwargs: Optional[Dict] = None,
+        transform_keys: Optional[List[str]] = None,
+        chunk_size: int = 1000,
+        labels: Optional[Labels] = None,
+    ) -> None:
+        super().__init__(
             ds=ds,
+            name=name,
+            input_dts=input_dts,
+            output_dts=output_dts,
+            transform_keys=transform_keys,
+            chunk_size=chunk_size,
+            labels=labels,
+        )
+
+        self.func = func
+        self.kwargs = kwargs
+
+    def process_batch_dts(
+        self,
+        ds: DataStore,
+        idx: IndexDF,
+        run_config: Optional[RunConfig] = None,
+    ) -> Optional[TransformResult]:
+        return self.func(
+            ds=ds,
+            idx=idx,
             input_dts=self.input_dts,
-            output_dts=self.output_dts,
-            idx_count=idx_count,
-            idx_gen=idx_gen,
             run_config=run_config,
             kwargs=self.kwargs,
         )
 
-        res_changelist = ChangeList()
 
-        for changes in gen:
-            res_changelist.extend(changes)
+@dataclass
+class BatchTransform(PipelineStep):
+    func: BatchTransformFunc
+    inputs: List[str]
+    outputs: List[str]
+    chunk_size: int = 1000
+    kwargs: Optional[Dict[str, Any]] = None
+    transform_keys: Optional[List[str]] = None
+    labels: Optional[Labels] = None
+
+    def build_compute(self, ds: DataStore, catalog: Catalog) -> List[ComputeStep]:
+        input_dts = [catalog.get_datatable(ds, name) for name in self.inputs]
+        output_dts = [catalog.get_datatable(ds, name) for name in self.outputs]
+
+        return [
+            BatchTransformStep(
+                ds=ds,
+                name=f"{self.func.__name__}",  # type: ignore # mypy bug: https://github.com/python/mypy/issues/10976
+                input_dts=input_dts,
+                output_dts=output_dts,
+                func=self.func,
+                kwargs=self.kwargs,
+                transform_keys=self.transform_keys,
+                chunk_size=self.chunk_size,
+                labels=self.labels,
+            )
+        ]
+
 
-        return res_changelist
+class BatchTransformStep(BaseBatchTransformStep):
+    def __init__(
+        self,
+        ds: DataStore,
+        name: str,
+        func: BatchTransformFunc,
+        input_dts: List[DataTable],
+        output_dts: List[DataTable],
+        kwargs: Optional[Dict[str, Any]] = None,
+        transform_keys: Optional[List[str]] = None,
+        chunk_size: int = 1000,
+        labels: Optional[Labels] = None,
+    ) -> None:
+        super().__init__(
+            ds=ds,
+            name=name,
+            input_dts=input_dts,
+            output_dts=output_dts,
+            transform_keys=transform_keys,
+            chunk_size=chunk_size,
+            labels=labels,
+        )
 
+        self.func = func
+        self.kwargs = kwargs
 
-BatchGenerateFunc = Callable[..., Iterator[Union[DataDF, Tuple[DataDF, ...]]]]
+    def process_batch_dfs(
+        self,
+        ds: DataStore,
+        idx: IndexDF,
+        input_dfs: List[DataDF],
+        run_config: Optional[RunConfig] = None,
+    ) -> TransformResult:
+        return self.func(*input_dfs, **self.kwargs or {})
 
 
 def do_batch_generate(
     func: BatchGenerateFunc,
     ds: DataStore,
     output_dts: List[DataTable],
-    kwargs: Optional[Dict[str, Any]] = None,
     run_config: Optional[RunConfig] = None,
+    kwargs: Optional[Dict] = None,
 ) -> None:
     import inspect
 
@@ -360,39 +643,27 @@ def do_batch_generate(
             dt_k.delete_stale_by_process_ts(now, run_config=run_config)
 
 
+@dataclass
 class BatchGenerate(PipelineStep):
-    def __init__(
-        self,
-        func: BatchGenerateFunc,
-        outputs: List[str],
-        kwargs: Optional[Dict] = None,
-        labels: Optional[Dict[str, str]] = None,
-    ):
-        self.func = func
-        self.outputs = outputs
-        self.kwargs = kwargs
-        self.labels = labels
+    func: BatchGenerateFunc
+    outputs: List[str]
+    kwargs: Optional[Dict] = None
+    labels: Optional[Labels] = None
 
     def build_compute(self, ds: DataStore, catalog: Catalog) -> List[ComputeStep]:
-        def transform_func(
-            ds: DataStore,
-            input_dts: List[DataTable],
-            output_dts: List[DataTable],
-            run_config: Optional[RunConfig],
-            **kwargs,
-        ):
-            return do_batch_generate(
-                func=self.func,
-                kwargs=kwargs,
-                ds=ds,
-                output_dts=output_dts,
-                run_config=run_config,
-            )
-
         return [
             DatatableTransformStep(
                 name=self.func.__name__,
-                func=cast(DatatableTransformFunc, transform_func),
+                func=cast(
+                    DatatableTransformFunc,
+                    lambda ds, input_dts, output_dts, run_config, kwargs: do_batch_generate(
+                        func=self.func,
+                        ds=ds,
+                        output_dts=output_dts,
+                        run_config=run_config,
+                        kwargs=kwargs,
+                    ),
+                ),
                 input_dts=[],
                 output_dts=[catalog.get_datatable(ds, name) for name in self.outputs],
                 check_for_changes=False,
@@ -410,7 +681,6 @@ def update_external_table(
     for ps_df in tqdm.tqdm(
         table.table_store.read_rows_meta_pseudo_df(run_config=run_config)
     ):
-
         (
             new_df,
             changed_df,
diff --git a/datapipe/datatable.py b/datapipe/datatable.py
index ba97b4e3..84ac30c1 100644
--- a/datapipe/datatable.py
+++ b/datapipe/datatable.py
@@ -4,12 +4,11 @@
 
 import pandas as pd
 from opentelemetry import trace
-from sqlalchemy import alias, and_, func, or_, select, column
 
 from datapipe.event_logger import EventLogger
 from datapipe.metastore import MetaTable
-from datapipe.run_config import LabelDict, RunConfig
-from datapipe.store.database import DBConn, sql_apply_runconfig_filter
+from datapipe.run_config import RunConfig
+from datapipe.store.database import DBConn
 from datapipe.store.table_store import TableStore
 from datapipe.types import (
     ChangeList,
@@ -174,6 +173,7 @@ def __init__(
             self.meta_dbconn, create_table=create_meta_table
         )
         self.tables: Dict[str, DataTable] = {}
+
         self.create_meta_table = create_meta_table
 
     def create_table(self, name: str, table_store: TableStore) -> DataTable:
@@ -206,6 +206,7 @@ def get_or_create_table(self, name: str, table_store: TableStore) -> DataTable:
         else:
             return self.create_table(name, table_store)
 
+    # TODO remove, because it's implemented in BatchTransformStep
     def get_join_keys(
         self, inputs: List[DataTable], outputs: List[DataTable]
     ) -> List[str]:
@@ -217,157 +218,7 @@ def get_join_keys(
     def get_table(self, name: str) -> DataTable:
         return self.tables[name]
 
-    def _build_changed_idx_sql(
-        self,
-        inputs: List[DataTable],
-        outputs: List[DataTable],
-        run_config: Optional[RunConfig] = None,
-    ) -> Tuple[Iterable[str], select]:
-        inp_keys = [set(inp.primary_keys) for inp in inputs]
-        out_keys = [set(out.primary_keys) for out in outputs]
-        join_keys = list(set.intersection(*inp_keys, *out_keys))
-
-        if self.meta_dbconn.con.driver in ("sqlite", "pysqlite"):
-            greatest_func = func.max
-        else:
-            greatest_func = func.greatest
-
-        def _make_agg_cte(dt: DataTable, agg_fun, agg_col: str):
-            tbl = dt.meta_table.sql_table
-
-            key_cols = [column(k) for k in join_keys]
-
-            sql = (
-                select(*key_cols + [agg_fun(tbl.c[agg_col]).label(agg_col)])
-                .select_from(tbl)
-                .group_by(*key_cols)
-            )
-
-            sql = sql_apply_runconfig_filter(sql, tbl, dt.primary_keys, run_config)
-
-            return sql.cte(name=f"{tbl.name}__{agg_col}")
-
-        def _make_agg_of_agg(ctes, agg_col):
-            assert len(ctes) > 0
-
-            if len(ctes) == 1:
-                return ctes[0]
-
-            if len(join_keys) > 1:
-                coalesce_keys = [
-                    func.coalesce(*[subq.c[key] for subq in ctes]).label(key)
-                    for key in join_keys
-                ]
-            else:
-                coalesce_keys = [subq.c[join_keys[0]] for subq in ctes]
-
-            agg = greatest_func(*[subq.c[agg_col] for subq in ctes]).label(agg_col)
-
-            sql = select(*coalesce_keys + [agg]).select_from(ctes[0])
-
-            for cte in ctes[1:]:
-                sql = sql.outerjoin(
-                    cte,
-                    onclause=and_(*[ctes[0].c[key] == cte.c[key] for key in join_keys]),
-                    full=True,
-                )
-
-            return sql.cte(name=f"all__{agg_col}")
-
-        inp_ctes = [_make_agg_cte(tbl, func.max, "update_ts") for tbl in inputs]
-        out_ctes = [_make_agg_cte(tbl, func.min, "process_ts") for tbl in outputs]
-
-        inp = _make_agg_of_agg(inp_ctes, "update_ts")
-        out = _make_agg_of_agg(out_ctes, "process_ts")
-
-        sql = (
-            select(
-                *[func.coalesce(inp.c[key], out.c[key]).label(key) for key in join_keys]
-            )
-            .select_from(inp)
-            .outerjoin(
-                out, onclause=and_(*[inp.c[key] == out.c[key] for key in join_keys])
-            )
-            .where(
-                or_(
-                    inp.c.update_ts > out.c.process_ts,
-                    inp.c.update_ts == None,  # noqa
-                    out.c.process_ts == None,  # noqa
-                )
-            )
-        )
-
-        return (join_keys, sql)
-
-    def get_changed_idx_count(
-        self,
-        inputs: List[DataTable],
-        outputs: List[DataTable],
-        run_config: Optional[RunConfig] = None,
-    ) -> int:
-        _, sql = self._build_changed_idx_sql(
-            inputs=inputs, outputs=outputs, run_config=run_config
-        )
-
-        idx_count = self.meta_dbconn.con.execute(
-            select([func.count()]).select_from(
-                alias(sql.subquery(), name="union_select")
-            )
-        ).scalar()
-
-        return idx_count
-
-    def get_full_process_ids(
-        self,
-        inputs: List[DataTable],
-        outputs: List[DataTable],
-        chunk_size: int = 1000,
-        run_config: Optional[RunConfig] = None,
-    ) -> Tuple[int, Iterable[IndexDF]]:
-        """
-        Метод для получения перечня индексов для обработки.
-
-        Returns: (idx_size, iterator<idx_df>)
-
-        - idx_size - количество индексов требующих обработки
-        - idx_df - датафрейм без колонок с данными, только индексная колонка
-        """
-
-        if len(inputs) == 0:
-            return (0, iter([]))
-
-        idx_count = self.get_changed_idx_count(
-            inputs=inputs,
-            outputs=outputs,
-            run_config=run_config,
-        )
-
-        join_keys, u1 = self._build_changed_idx_sql(
-            inputs=inputs,
-            outputs=outputs,
-            run_config=run_config,
-        )
-
-        # Список ключей из фильтров, которые нужно добавить в результат
-        extra_filters: LabelDict
-        if run_config is not None:
-            extra_filters = {
-                k: v for k, v in run_config.filters.items() if k not in join_keys
-            }
-        else:
-            extra_filters = {}
-
-        def alter_res_df():
-            for df in pd.read_sql_query(
-                u1, con=self.meta_dbconn.con, chunksize=chunk_size
-            ):
-                for k, v in extra_filters.items():
-                    df[k] = v
-
-                yield df
-
-        return math.ceil(idx_count / chunk_size), alter_res_df()
-
+    # TODO remove, because it's implemented in BatchTransformStep
     def get_change_list_process_ids(
         self,
         inputs: List[DataTable],
diff --git a/datapipe/lints.py b/datapipe/lints.py
new file mode 100644
index 00000000..f812b33b
--- /dev/null
+++ b/datapipe/lints.py
@@ -0,0 +1,150 @@
+from enum import Enum
+from typing import Optional, Tuple
+
+from datapipe.datatable import DataTable
+from datapipe.store.database import TableStoreDB
+from sqlalchemy import and_, func, or_, select, update, literal, not_, insert
+from termcolor import colored
+
+
+class LintStatus(Enum):
+    OK = "ok"
+    SKIP = "skip"
+    FAIL = "fail"
+
+
+class Lint:
+    desc: str
+
+    def check(self, dt: DataTable) -> Tuple[LintStatus, Optional[str]]:
+        query = self.check_query(dt)
+
+        if query is None:
+            return (LintStatus.SKIP, None)
+
+        (cnt,) = dt.meta_table.dbconn.con.execute(query).fetchone()
+
+        if cnt == 0:
+            return (LintStatus.OK, None)
+        else:
+            return (LintStatus.FAIL, f"{cnt} rows")
+
+    def check_query(self, dt: DataTable):
+        raise NotImplementedError
+
+    def fix(self, dt: DataTable) -> Tuple[LintStatus, Optional[str]]:
+        raise NotImplementedError
+
+
+class LintDeleteTSIsNewerThanUpdateOrProcess(Lint):
+    lint_id = "DTP001"
+    desc = "delete_ts is newer than update_ts or process_ts"
+
+    def check_query(self, dt: DataTable):
+        meta_tbl = dt.meta_table.sql_table
+        sql = (
+            select(func.count())
+            .select_from(meta_tbl)
+            .where(
+                and_(
+                    or_(
+                        meta_tbl.c.update_ts < meta_tbl.c.delete_ts,
+                        meta_tbl.c.process_ts < meta_tbl.c.delete_ts,
+                    ),
+                    meta_tbl.c.delete_ts != None,
+                )
+            )
+        )
+
+        return sql
+
+    def fix(self, dt: DataTable):
+        meta_tbl = dt.meta_table.sql_table
+
+        sql = (
+            update(meta_tbl)
+            .where(
+                and_(
+                    or_(
+                        meta_tbl.c.update_ts < meta_tbl.c.delete_ts,
+                        meta_tbl.c.process_ts < meta_tbl.c.delete_ts,
+                    ),
+                    meta_tbl.c.delete_ts != None,
+                )
+            )
+            .values(
+                update_ts=meta_tbl.c.delete_ts,
+                process_ts=meta_tbl.c.delete_ts,
+            )
+        )
+
+        dt.meta_table.dbconn.con.execute(sql)
+
+        return (LintStatus.OK, None)
+
+
+class LintDataWOMeta(Lint):
+    desc = "data has rows without meta"
+
+    def check_query(self, dt: DataTable):
+        if not isinstance(dt.table_store, TableStoreDB):
+            return None
+
+        meta_tbl = dt.meta_table.sql_table
+        data_tbl = dt.table_store.data_table
+
+        exists_sql = (
+            select(literal(1))
+            .select_from(meta_tbl)
+            .where(
+                and_(
+                    meta_tbl.c[col.name] == data_tbl.c[col.name]
+                    for col in meta_tbl.columns
+                    if col.primary_key
+                )
+            )
+        )
+
+        sql = (
+            select(func.count()).select_from(data_tbl).where(not_(exists_sql.exists()))
+        )
+
+        return sql
+
+    def fix(self, dt: DataTable):
+        assert isinstance(dt.table_store, TableStoreDB)
+
+        meta_tbl = dt.meta_table.sql_table
+        data_tbl = dt.table_store.data_table
+
+        exists_sql = (
+            select(literal(1))
+            .select_from(meta_tbl)
+            .where(
+                and_(
+                    meta_tbl.c[col.name] == data_tbl.c[col.name]
+                    for col in meta_tbl.columns
+                    if col.primary_key
+                )
+            )
+        )
+
+        sql = insert(meta_tbl).from_select(
+            [col.name for col in meta_tbl.columns if col.primary_key]
+            + ["hash", "create_ts", "update_ts", "delete_ts"],
+            select(
+                *[data_tbl.c[col.name] for col in meta_tbl.columns if col.primary_key]
+                + [
+                    literal(0).label("hash"),
+                    literal(0).label("create_ts"),
+                    literal(0).label("update_ts"),
+                    literal(None).label("delete_ts"),
+                ]
+            )
+            .select_from(data_tbl)
+            .where(not_(exists_sql.exists())),
+        )
+
+        dt.meta_table.dbconn.con.execute(sql)
+
+        return (LintStatus.OK, None)
diff --git a/datapipe/metastore.py b/datapipe/metastore.py
index 4474d9d2..b12a1f8c 100644
--- a/datapipe/metastore.py
+++ b/datapipe/metastore.py
@@ -3,17 +3,18 @@
 import math
 import time
 from dataclasses import dataclass
-from typing import Iterator, List, Tuple, cast, Optional
+from typing import Iterator, List, Optional, Tuple, cast
 
 import pandas as pd
 from cityhash import CityHash32
-from sqlalchemy import Column, Float, Integer, Table, func
+from sqlalchemy import Boolean, Column, Float, Integer, String, Table, func
 from sqlalchemy.sql.expression import and_, delete, or_, select, text, tuple_
 
 from datapipe.run_config import RunConfig
 from datapipe.store.database import (
     DBConn,
     MetaKey,
+    sql_apply_idx_filter,
     sql_apply_runconfig_filter,
     sql_schema_to_sqltype,
 )
@@ -29,7 +30,7 @@
 
 logger = logging.getLogger("datapipe.metastore")
 
-METADATA_SQL_SCHEMA = [
+TABLE_META_SCHEMA = [
     Column("hash", Integer),
     Column("create_ts", Float),  # Время создания строки
     Column("update_ts", Float),  # Время последнего изменения
@@ -75,9 +76,9 @@ def __init__(
             )
             self.meta_keys[target_name] = column.name
 
-        sql_schema = primary_schema + meta_schema + METADATA_SQL_SCHEMA
-
-        self.sql_schema = [copy.copy(i) for i in sql_schema]
+        self.sql_schema = [
+            copy.copy(i) for i in primary_schema + meta_schema + TABLE_META_SCHEMA
+        ]
 
         self.sql_table = Table(
             f"{self.name}_meta",
@@ -116,23 +117,8 @@ def _build_metadata_query(
                 # Когда ключей нет - не делаем ничего
                 pass
 
-            elif len(self.primary_keys) == 1:
-                # Когда ключ один - сравниваем напрямую
-                key = self.primary_keys[0]
-                sql = sql.where(self.sql_table.c[key].in_(idx[key].to_list()))
-
             else:
-                # Когда ключей много - сравниваем через tuple
-                keys = tuple_(*[self.sql_table.c[key] for key in self.primary_keys])
-
-                sql = sql.where(
-                    keys.in_(
-                        [
-                            tuple([r[key] for key in self.primary_keys])  # type: ignore
-                            for r in idx.to_dict(orient="records")
-                        ]
-                    )
-                )
+                sql = sql_apply_idx_filter(sql, self.sql_table, self.primary_keys, idx)
 
         if not include_deleted:
             sql = sql.where(self.sql_table.c.delete_ts.is_(None))
@@ -202,7 +188,7 @@ def _get_meta_data_columns(self):
         return (
             self.primary_keys
             + list(self.meta_keys.values())
-            + [column.name for column in METADATA_SQL_SCHEMA]
+            + [column.name for column in TABLE_META_SCHEMA]
         )
 
     def _get_hash_for_df(self, df) -> pd.DataFrame:
@@ -221,6 +207,12 @@ def get_existing_idx(self, idx: Optional[IndexDF] = None) -> IndexDF:
         sql = select(self.sql_schema)
 
         if idx is not None:
+            if len(idx.index) == 0:
+                # Empty index -> empty result
+                return cast(
+                    IndexDF,
+                    pd.DataFrame(columns=[column.name for column in self.sql_schema]),
+                )
             idx_cols = list(set(idx.columns.tolist()) & set(self.primary_keys))
 
             if not idx_cols:
@@ -231,7 +223,7 @@ def get_existing_idx(self, idx: Optional[IndexDF] = None) -> IndexDF:
             # FIXME поправить на сравнение кортежей
             for _, row in idx.iterrows():
                 and_params = [
-                    self.sql_table.c[key] == self._get_sql_param(row[key])
+                    self.sql_table.c[key] == self._get_sql_param(row[key])  # type: ignore
                     for key in idx_cols
                     if key in self.primary_keys
                 ]
@@ -295,10 +287,10 @@ def get_changes_for_store_chunk(
         new_idx = merged_df["hash"].isna() | merged_df["delete_ts"].notnull()
 
         # Ищем новые записи
-        new_df = data_df.loc[new_idx.values, data_cols]
+        new_df = data_df.loc[new_idx.values, data_cols]  # type: ignore
 
         # Создаем мета данные для новых записей
-        new_meta_data_df = merged_df.loc[merged_df["hash"].isna().values, data_cols]
+        new_meta_data_df = merged_df.loc[merged_df["hash"].isna().values, data_cols]  # type: ignore
         new_meta_df = self._make_new_metadata_df(now, new_meta_data_df)
 
         # Ищем изменившиеся записи
@@ -307,7 +299,7 @@ def get_changes_for_store_chunk(
             & (merged_df["delete_ts"].isnull())
             & (merged_df["hash"] != merged_df["data_hash"])
         )
-        changed_df = merged_df.loc[changed_idx.values, data_cols]
+        changed_df = merged_df.loc[changed_idx.values, data_cols]  # type: ignore
 
         # Меняем мета данные для существующих записей
         changed_meta_idx = (merged_df["hash"].notna()) & (
@@ -453,7 +445,9 @@ def mark_rows_deleted(
             self.update_meta_for_store_chunk(meta_df)
 
     def get_stale_idx(
-        self, process_ts: float, run_config: Optional[RunConfig] = None
+        self,
+        process_ts: float,
+        run_config: Optional[RunConfig] = None,
     ) -> Iterator[IndexDF]:
         idx_cols = [self.sql_table.c[key] for key in self.primary_keys]
         sql = select(idx_cols).where(
@@ -473,16 +467,98 @@ def get_stale_idx(
         )
 
 
-class MetaTableData:
-    def __init__(self, tbl: MetaTable, sql_prefix: str = "") -> None:
-        self.primary_keys = set(tbl.primary_keys)
-        self.meta_keys = set(tbl.meta_keys.keys())
-        self.meta_column_names = tbl.meta_keys
-        self.sql_table = tbl.sql_table.alias(f"{sql_prefix}_{tbl.name}")
+TRANSFORM_META_SCHEMA = [
+    Column("process_ts", Float),  # Время последней успешной обработки
+    Column("is_success", Boolean),  # Успешно ли обработана строка
+    Column("error", String),  # Текст ошибки
+]
+
 
-    def get_keys(self):
-        return self.primary_keys | self.meta_keys
+class TransformMetaTable:
+    def __init__(
+        self,
+        dbconn: DBConn,
+        name: str,
+        primary_schema: DataSchema,
+        create_table: bool = False,
+    ) -> None:
+        self.dbconn = dbconn
+        self.name = name
+        self.primary_schema = primary_schema
+        self.primary_keys = [i.name for i in primary_schema]
+
+        self.sql_schema = [i.copy() for i in primary_schema + TRANSFORM_META_SCHEMA]
+
+        self.sql_table = Table(
+            name,
+            dbconn.sqla_metadata,
+            *self.sql_schema,
+        )
+
+        if create_table:
+            self.sql_table.create(self.dbconn.con, checkfirst=True)
+
+    def mark_rows_processed_success(
+        self,
+        idx: IndexDF,
+        process_ts: float,
+        run_config: Optional[RunConfig] = None,
+    ) -> None:
+        idx = cast(IndexDF, idx[self.primary_keys])
+
+        insert_sql = self.dbconn.insert(self.sql_table).values(
+            [
+                {
+                    "process_ts": process_ts,
+                    "is_success": True,
+                    "error": None,
+                    **idx_dict,  # type: ignore
+                }
+                for idx_dict in idx.to_dict(orient="records")
+            ]
+        )
+
+        sql = insert_sql.on_conflict_do_update(
+            index_elements=self.primary_keys,
+            set_={
+                "process_ts": process_ts,
+                "is_success": True,
+                "error": None,
+            },
+        )
+
+        # execute
+        self.dbconn.con.execute(sql)
+
+    def mark_rows_processed_error(
+        self,
+        idx: IndexDF,
+        process_ts: float,
+        error: str,
+        run_config: Optional[RunConfig] = None,
+    ) -> None:
+        idx = cast(IndexDF, idx[self.primary_keys])
+
+        insert_sql = self.dbconn.insert(self.sql_table).values(
+            [
+                {
+                    "process_ts": process_ts,
+                    "is_success": False,
+                    "error": error,
+                    **idx_dict,  # type: ignore
+                }
+                for idx_dict in idx.to_dict(orient="records")
+            ]
+        )
+
+        sql = insert_sql.on_conflict_do_update(
+            index_elements=self.primary_keys,
+            set_={
+                "process_ts": process_ts,
+                "is_success": False,
+                "error": error,
+            },
+        )
 
-    def get_column(self, key: str) -> Column:
-        column_key = key if key in self.primary_keys else self.meta_column_names[key]
-        return self.sql_table.c[column_key]
+        # execute
+        self.dbconn.con.execute(sql)
diff --git a/datapipe/store/database.py b/datapipe/store/database.py
index e178d2f5..34751065 100644
--- a/datapipe/store/database.py
+++ b/datapipe/store/database.py
@@ -9,7 +9,7 @@
 from sqlalchemy import Column, Integer, MetaData, String, Table, create_engine
 from sqlalchemy.pool import SingletonThreadPool
 from sqlalchemy.schema import SchemaItem
-from sqlalchemy.sql.base import SchemaEventTarget
+from sqlalchemy.sql.base import Executable, SchemaEventTarget
 from sqlalchemy.sql.expression import delete, select, tuple_
 
 from datapipe.run_config import RunConfig
@@ -51,10 +51,18 @@ def _init(self, connstr: str, schema: Optional[str]) -> None:
 
         if connstr.startswith("sqlite"):
             self.supports_update_from = False
+
+            from sqlalchemy.dialects.sqlite import insert
+
+            self.insert = insert
         else:
             # Assume relatively new Postgres
             self.supports_update_from = True
 
+            from sqlalchemy.dialects.postgresql import insert
+
+            self.insert = insert
+
         self.con = create_engine(
             connstr,
             poolclass=SingletonThreadPool,
@@ -75,11 +83,11 @@ def __setstate__(self, state):
 
 
 def sql_apply_runconfig_filter(
-    sql: select,
+    sql: Executable,
     table: Table,
     primary_keys: List[str],
     run_config: Optional[RunConfig] = None,
-) -> select:
+) -> Executable:
     if run_config is not None:
         for k, v in run_config.filters.items():
             if k in primary_keys:
@@ -88,6 +96,33 @@ def sql_apply_runconfig_filter(
     return sql
 
 
+def sql_apply_idx_filter(
+    sql: Executable,
+    table: Table,
+    primary_keys: List[str],
+    idx: IndexDF,
+) -> Executable:
+    if len(primary_keys) == 1:
+        # Когда ключ один - сравниваем напрямую
+        key = primary_keys[0]
+        sql = sql.where(table.c[key].in_(idx[key].to_list()))
+
+    else:
+        # Когда ключей много - сравниваем по кортежу
+        keys = tuple_(*[table.c[key] for key in primary_keys])
+
+        sql = sql.where(
+            keys.in_(
+                [
+                    tuple([r[key] for key in primary_keys])  # type: ignore
+                    for r in idx.to_dict(orient="records")
+                ]
+            )
+        )
+
+    return sql
+
+
 class MetaKey(SchemaItem):
     def __init__(self, target_name: Optional[str] = None) -> None:
         self.target_name = target_name
@@ -162,27 +197,6 @@ def _chunk_idx_df(self, idx: TAnyDF) -> Iterator[TAnyDF]:
 
             yield cast(TAnyDF, chunk_idx)
 
-    def _apply_where_expression(self, sql, idx: IndexDF):
-        if len(self.primary_keys) == 1:
-            # Когда ключ один - сравниваем напрямую
-            key = self.primary_keys[0]
-            sql = sql.where(self.data_table.c[key].in_(idx[key].to_list()))
-
-        else:
-            # Когда ключей много - сравниваем через tuple
-            keys = tuple_(*[self.data_table.c[key] for key in self.primary_keys])
-
-            sql = sql.where(
-                keys.in_(
-                    [
-                        tuple([r[key] for key in self.primary_keys])  # type: ignore
-                        for r in idx.to_dict(orient="records")
-                    ]
-                )
-            )
-
-        return sql
-
     def delete_rows(self, idx: IndexDF) -> None:
         if idx is None or len(idx.index) == 0:
             return
@@ -190,7 +204,9 @@ def delete_rows(self, idx: IndexDF) -> None:
         logger.debug(f"Deleting {len(idx.index)} rows from {self.name} data")
 
         for chunk_idx in self._chunk_idx_df(idx):
-            sql = self._apply_where_expression(delete(self.data_table), chunk_idx)
+            sql = sql_apply_idx_filter(
+                delete(self.data_table), self.data_table, self.primary_keys, chunk_idx
+            )
             self.dbconn.con.execute(sql)
 
     def insert_rows(self, df: DataDF) -> None:
@@ -232,7 +248,9 @@ def read_rows(self, idx: Optional[IndexDF] = None) -> pd.DataFrame:
             res = []
 
             for chunk_idx in self._chunk_idx_df(idx):
-                chunk_sql = self._apply_where_expression(sql, chunk_idx)
+                chunk_sql = sql_apply_idx_filter(
+                    sql, self.data_table, self.primary_keys, chunk_idx
+                )
                 chunk_df = pd.read_sql_query(chunk_sql, con=self.dbconn.con)
 
                 res.append(chunk_df)
diff --git a/datapipe/store/filedir.py b/datapipe/store/filedir.py
index e5d39c2d..1b3fcf64 100644
--- a/datapipe/store/filedir.py
+++ b/datapipe/store/filedir.py
@@ -71,9 +71,7 @@ def _pattern_to_attrnames(pat: str) -> List[str]:
     assert len(attrnames) > 0, "The scheme is not valid."
     if len(attrnames) >= 2:
         duplicates_attrnames = list(duplicates(attrnames))
-        assert (
-            len(duplicates_attrnames) == 0
-        ), f"Some keys are repeated: {duplicates_attrnames}. Rename them."
+        assert len(duplicates_attrnames) == 0, f"Some keys are repeated: {duplicates_attrnames}. Rename them."
 
     return attrnames
 
@@ -81,26 +79,16 @@ def _pattern_to_attrnames(pat: str) -> List[str]:
 def _pattern_to_patterns_or(pat) -> List[str]:
     pattern_or = re.compile(r"(?P<or>\(([a-zA-Z0-9]+\|)+[a-zA-Z0-9]+\))")
     # Ищем вхождения вида (aaa|bbb|ccc), в виду list of list [[aaa, bbb, ccc], [ddd, eee], ...]
-    values = [
-        list(dict.fromkeys(match.group("or")[1:-1].split("|")))
-        for match in pattern_or.finditer(pat)
-    ]
+    values = [list(dict.fromkeys(match.group("or")[1:-1].split("|"))) for match in pattern_or.finditer(pat)]
     # Всевозможные комбинации для замены [[aaa, ddd], [aaa, eee], [bbb, ddd], ...]
-    possible_combinatios_values = [
-        list(combination) for combination in itertools.product(*values)
-    ]
+    possible_combinatios_values = [list(combination) for combination in itertools.product(*values)]
     # Получаем всевозможные списки шаблонов из комбинаций
-    filename_patterns = [
-        re.sub(pattern_or, Replacer(combination), pat)
-        for combination in possible_combinatios_values
-    ]
+    filename_patterns = [re.sub(pattern_or, Replacer(combination), pat) for combination in possible_combinatios_values]
     return filename_patterns
 
 
 def _pattern_to_glob(pat: str) -> str:
-    return re.sub(
-        r"\{([^/]+?)\}", "*", pat
-    )  # Меняем все вхождения {id1}_{id2} в звездочки *_*
+    return re.sub(r"\{([^/]+?)\}", "*", pat)  # Меняем все вхождения {id1}_{id2} в звездочки *_*
 
 
 def _pattern_to_match(pat: str) -> str:
@@ -108,12 +96,8 @@ def _pattern_to_match(pat: str) -> str:
     # * -> r'[^/]+'
     # ** -> r'([^/]+/)*?[^/]+'
 
-    pat = re.sub(
-        r"\*\*?", r"([^/]+/)*[^/]+", pat
-    )  # Меняем все вхождения * и ** в произвольные символы
-    pat = re.sub(
-        r"\{([^/]+?)\}", r"(?P<\1>[^/]+?)", pat
-    )  # Меняем все вхождения вида {id} на непустые послед. символов
+    pat = re.sub(r"\*\*?", r"([^/]+/)*[^/]+", pat)  # Меняем все вхождения * и ** в произвольные символы
+    pat = re.sub(r"\{([^/]+?)\}", r"(?P<\1>[^/]+?)", pat)  # Меняем все вхождения вида {id} на непустые послед. символов
     pat = f"{pat}\\Z"  # Учитываем конец строки
     return pat
 
@@ -194,17 +178,13 @@ def __init__(
         # Multiply extensions check
         if len(self.filename_glob) >= 2:
             if readonly is not None and not readonly:
-                raise ValueError(
-                    "When `readonly=False`, in filename_pattern shouldn't be several extensions."
-                )
+                raise ValueError("When `readonly=False`, in filename_pattern shouldn't be several extensions.")
             elif readonly:
                 readonly = True
         # Any * and ** pattern check
         if "*" in path:
             if readonly is not None and not readonly:
-                raise ValueError(
-                    "When `readonly=False`, in filename_pattern shouldn't be any `*` characters."
-                )
+                raise ValueError("When `readonly=False`, in filename_pattern shouldn't be any `*` characters.")
             elif readonly is None:
                 readonly = True
         elif readonly is None:
@@ -221,22 +201,11 @@ def __init__(
 
         if primary_schema is not None:
             assert sorted(self.attrnames) == sorted(i.name for i in primary_schema)
-            assert all(
-                [
-                    isinstance(column.type, (String, Integer))
-                    for column in primary_schema
-                ]
-            )
+            assert all([isinstance(column.type, (String, Integer)) for column in primary_schema])
             self.primary_schema = primary_schema
         else:
-            self.primary_schema = [
-                Column(attrname, String(100), primary_key=True)
-                for attrname in self.attrnames
-            ]
-        self.attrname_to_cls = {
-            column.name: type_to_cls[type(column.type)]
-            for column in self.primary_schema
-        }
+            self.primary_schema = [Column(attrname, String(100), primary_key=True) for attrname in self.attrnames]
+        self.attrname_to_cls = {column.name: type_to_cls[type(column.type)] for column in self.primary_schema}
 
     def get_primary_schema(self) -> DataSchema:
         return self.primary_schema
@@ -254,23 +223,18 @@ def delete_rows(self, idx: IndexDF) -> None:
             attrnames_series = idx.loc[row_idx, self.attrnames]
             assert isinstance(attrnames_series, pd.Series)
 
-            _, path = fsspec.core.split_protocol(
-                self._filenames_from_idxs_values(attrnames_series.tolist())[0]
-            )
+            attrnames = cast(List[str], attrnames_series.tolist())
+
+            _, path = fsspec.core.split_protocol(self._filenames_from_idxs_values(attrnames)[0])
             self.filesystem.rm(path)
 
     def _filenames_from_idxs_values(self, idxs_values: List[str]) -> List[str]:
-        return [
-            re.sub(r"\{([^/]+?)\}", Replacer(idxs_values), pat)
-            for pat in self.filename_patterns
-        ]
+        return [re.sub(r"\{([^/]+?)\}", Replacer(idxs_values), pat) for pat in self.filename_patterns]
 
     def _idxs_values_from_filepath(self, filepath: str) -> Dict[str, Any]:
         _, filepath = fsspec.core.split_protocol(filepath)
         m = re.match(self.filename_match, filepath)
-        assert (
-            m is not None
-        ), f"Filepath {filepath} does not match the pattern {self.filename_match}"
+        assert m is not None, f"Filepath {filepath} does not match the pattern {self.filename_match}"
 
         data = {}
         for attrname in self.attrnames:
@@ -281,9 +245,7 @@ def _idxs_values_from_filepath(self, filepath: str) -> Dict[str, Any]:
     def _assert_key_values(self, filepath: str, idxs_values: List[str]):
         idx_data = self._idxs_values_from_filepath(filepath)
         idxs_values_np = np.array(idxs_values)
-        idxs_values_parsed_from_filepath = np.array(
-            [idx_data[attrname] for attrname in self.attrnames]
-        )
+        idxs_values_parsed_from_filepath = np.array([idx_data[attrname] for attrname in self.attrnames])
 
         assert len(idxs_values_np) == len(idxs_values_parsed_from_filepath) and np.all(
             idxs_values_np == idxs_values_parsed_from_filepath
@@ -294,9 +256,7 @@ def _assert_key_values(self, filepath: str, idxs_values: List[str]):
             f"{idxs_values_np=} not equals {idxs_values_parsed_from_filepath=}",
         )
 
-    def insert_rows(
-        self, df: pd.DataFrame, adapter: Optional[ItemStoreFileAdapter] = None
-    ) -> None:
+    def insert_rows(self, df: pd.DataFrame, adapter: Optional[ItemStoreFileAdapter] = None) -> None:
         if df.empty:
             return
         assert not self.readonly
@@ -306,9 +266,7 @@ def insert_rows(
         # WARNING: Здесь я поставил .drop(columns=self.attrnames), тк ключи будут хранится снаружи, в имени
         for row_idx, data in zip(
             df.index,
-            cast(
-                List[Dict[str, Any]], df.drop(columns=self.attrnames).to_dict("records")
-            ),
+            cast(List[Dict[str, Any]], df.drop(columns=self.attrnames).to_dict("records")),
         ):
             attrnames_series = df.loc[row_idx, self.attrnames]
             assert isinstance(attrnames_series, pd.Series)
@@ -333,9 +291,9 @@ def _read_rows_fast(
             attrnames_series = idx.loc[row_idx, self.attrnames]
             assert isinstance(attrnames_series, pd.Series)
 
-            _, path = fsspec.core.split_protocol(
-                self._filenames_from_idxs_values(attrnames_series.tolist())[0]
-            )
+            attrnames = cast(List[str], attrnames_series.tolist())
+
+            _, path = fsspec.core.split_protocol(self._filenames_from_idxs_values(attrnames)[0])
 
             res.loc[row_idx, "filepath"] = f"{self.protocol_str}{path}"
 
@@ -353,37 +311,25 @@ def read_rows(
         if adapter is None:
             adapter = self.adapter
 
-        if (
-            (not read_data)
-            and (len(self.filename_patterns) == 1)
-            and (idx is not None)
-            and self.add_filepath_column
-        ):
+        if (not read_data) and (len(self.filename_patterns) == 1) and (idx is not None) and self.add_filepath_column:
             return self._read_rows_fast(idx)
 
         def _iterate_files():
             if idx is None:
-                for file_open in fsspec.open_files(
-                    self.filename_glob, f"r{adapter.mode}"
-                ):
+                for file_open in fsspec.open_files(self.filename_glob, f"r{adapter.mode}"):
                     yield file_open
             else:
                 filepaths_extenstions = [
-                    self._filenames_from_idxs_values(idx.loc[row_idx, self.attrnames])
-                    for row_idx in idx.index
+                    self._filenames_from_idxs_values(idx.loc[row_idx, self.attrnames]) for row_idx in idx.index
                 ]
                 for filepaths in filepaths_extenstions:
                     found_files = [
                         file_open
-                        for file_open in fsspec.open_files(
-                            filepaths, f"r{adapter.mode}"
-                        )
+                        for file_open in fsspec.open_files(filepaths, f"r{adapter.mode}")
                         if self.filesystem.exists(file_open.path)
                     ]
                     if len(found_files) == 0:
-                        raise FileNotFoundError(
-                            f"No such file: {' or '.join(filepaths)}"
-                        )
+                        raise FileNotFoundError(f"No such file: {' or '.join(filepaths)}")
                     elif len(found_files) > 1:
                         raise ValueError(
                             f"Some files are duplitcated as indexes in filepaths: {found_files}. "
@@ -400,9 +346,7 @@ def _iterate_files():
                 if read_data:
                     data = adapter.load(f)
 
-                    attrnames_in_data = [
-                        attrname for attrname in self.attrnames if attrname in data
-                    ]
+                    attrnames_in_data = [attrname for attrname in self.attrnames if attrname in data]
                     assert len(attrnames_in_data) == 0, (
                         f"Found repeated keys inside data that are already used (from scheme): "
                         f"{attrnames_in_data}. "
@@ -449,9 +393,7 @@ def read_rows_meta_pseudo_df(
             ukeys.append(files.fs.ukey(f.path))
             filepaths.append(f"{self.protocol_str}{f.path}")
 
-        keys_values = [
-            (ids[attrname][i] for attrname in self.attrnames) for i in range(len(ukeys))
-        ]
+        keys_values = [(ids[attrname][i] for attrname in self.attrnames) for i in range(len(ukeys))]
         duplicates_keys_values = list(duplicates(keys_values))
         assert len(duplicates_keys_values) == 0, (
             f"Some files are duplitcated as indexes in filepaths: {duplicates_keys_values}. "
@@ -466,12 +408,8 @@ def read_rows_meta_pseudo_df(
                     **({"filepath": filepaths} if self.add_filepath_column else {}),
                 }
             )
-            yield pseudo_data_df.astype(object)
+
+            yield pseudo_data_df.astype(object)  # type: ignore # TODO fix typing issue
         else:
             filepath_kw: Dict = {"filepath": []} if self.add_filepath_column else {}
-            yield pd.DataFrame(
-                {
-                    "ukey": [],
-                    **filepath_kw,
-                }
-            ).astype(object)
+            yield pd.DataFrame({"ukey": [], **filepath_kw}).astype(object)  # type: ignore # TODO fix typing issue
diff --git a/datapipe/types.py b/datapipe/types.py
index cf623ae9..7c865d98 100644
--- a/datapipe/types.py
+++ b/datapipe/types.py
@@ -1,7 +1,15 @@
 from __future__ import annotations  # NOQA
 
 from dataclasses import dataclass, field
-from typing import Dict, List, NewType, TypeVar, cast
+from typing import (
+    Dict,
+    List,
+    NewType,
+    Tuple,
+    TypeVar,
+    Union,
+    cast,
+)
 
 import pandas as pd
 from sqlalchemy import Column
@@ -21,12 +29,14 @@
 
 TAnyDF = TypeVar("TAnyDF", pd.DataFrame, IndexDF, MetadataDF)
 
+Labels = List[Tuple[str, str]]
+
+TransformResult = Union[DataDF, List[DataDF], Tuple[DataDF, ...]]
+
 
 @dataclass
 class ChangeList:
-    changes: Dict[str, IndexDF] = field(
-        default_factory=lambda: cast(Dict[str, IndexDF], {})
-    )
+    changes: Dict[str, IndexDF] = field(default_factory=lambda: cast(Dict[str, IndexDF], {}))
 
     def append(self, table_name: str, idx: IndexDF) -> None:
         if table_name in self.changes:
@@ -36,9 +46,7 @@ def append(self, table_name: str, idx: IndexDF) -> None:
             if self_cols != other_cols:
                 raise ValueError(f"Different IndexDF for table {table_name}")
 
-            self.changes[table_name] = cast(
-                IndexDF, pd.concat([self.changes[table_name], idx], axis="index")
-            )
+            self.changes[table_name] = cast(IndexDF, pd.concat([self.changes[table_name], idx], axis="index"))
         else:
             self.changes[table_name] = idx
 
@@ -76,9 +84,7 @@ def index_difference(idx1_df: IndexDF, idx2_df: IndexDF) -> IndexDF:
 
 
 def index_intersection(idx1_df: IndexDF, idx2_df: IndexDF) -> IndexDF:
-    assert sorted(list(idx1_df.columns.tolist())) == sorted(
-        list(idx2_df.columns.tolist())
-    )
+    assert sorted(list(idx1_df.columns.tolist())) == sorted(list(idx2_df.columns.tolist()))
     cols = idx1_df.columns.to_list()
 
     idx1_idx = idx1_df.set_index(cols).index
diff --git a/docs/.gitignore b/docs/.gitignore
new file mode 100644
index 00000000..7d0f4ec4
--- /dev/null
+++ b/docs/.gitignore
@@ -0,0 +1,2 @@
+build
+source/generated/
\ No newline at end of file
diff --git a/docs/Makefile b/docs/Makefile
new file mode 100644
index 00000000..269cadcf
--- /dev/null
+++ b/docs/Makefile
@@ -0,0 +1,20 @@
+# Minimal makefile for Sphinx documentation
+#
+
+# You can set these variables from the command line, and also
+# from the environment for the first two.
+SPHINXOPTS    ?=
+SPHINXBUILD   ?= sphinx-build
+SOURCEDIR     = source
+BUILDDIR      = build
+
+# Put it first so that "make" without argument is like "make help".
+help:
+	@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
+
+.PHONY: help Makefile
+
+# Catch-all target: route all unknown targets to Sphinx using the new
+# "make mode" option.  $(O) is meant as a shortcut for $(SPHINXOPTS).
+%: Makefile
+	@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
\ No newline at end of file
diff --git a/docs/source/api.rst b/docs/source/api.rst
new file mode 100644
index 00000000..cfb8cfb7
--- /dev/null
+++ b/docs/source/api.rst
@@ -0,0 +1,8 @@
+API
+---
+
+.. autosummary::
+    :toctree: generated
+    :recursive:
+
+    datapipe
diff --git a/architecture.drawio b/docs/source/architecture.drawio
similarity index 88%
rename from architecture.drawio
rename to docs/source/architecture.drawio
index e23d8d0d..a1e0fcc0 100644
--- a/architecture.drawio
+++ b/docs/source/architecture.drawio
@@ -1,6 +1,6 @@
 <mxfile host="65bd71144e">
     <diagram id="680uwpuYkj8ZMehc0VYk" name="Page-2">
-        <mxGraphModel dx="1227" dy="1904" grid="1" gridSize="10" guides="1" tooltips="1" connect="1" arrows="1" fold="1" page="1" pageScale="1" pageWidth="850" pageHeight="1100" math="0" shadow="0">
+        <mxGraphModel dx="1492" dy="1906" grid="1" gridSize="10" guides="1" tooltips="1" connect="1" arrows="1" fold="1" page="1" pageScale="1" pageWidth="850" pageHeight="1100" math="0" shadow="0">
             <root>
                 <mxCell id="ur8WBRzY9eALHr9f39F1-0"/>
                 <mxCell id="ur8WBRzY9eALHr9f39F1-1" parent="ur8WBRzY9eALHr9f39F1-0"/>
@@ -36,7 +36,7 @@
                     <mxGeometry x="325" y="460" width="120" height="60" as="geometry"/>
                 </mxCell>
                 <mxCell id="ur8WBRzY9eALHr9f39F1-7" value="#ComputeStep" style="rounded=1;whiteSpace=wrap;html=1;labelBackgroundColor=none;fillColor=#fff2cc;strokeColor=#d6b656;" parent="ur8WBRzY9eALHr9f39F1-1" vertex="1">
-                    <mxGeometry x="565" y="260" width="120" height="60" as="geometry"/>
+                    <mxGeometry x="740" y="250" width="120" height="60" as="geometry"/>
                 </mxCell>
                 <mxCell id="ur8WBRzY9eALHr9f39F1-9" style="edgeStyle=none;html=1;fontColor=default;" parent="ur8WBRzY9eALHr9f39F1-1" source="ur8WBRzY9eALHr9f39F1-8" target="ur8WBRzY9eALHr9f39F1-7" edge="1">
                     <mxGeometry relative="1" as="geometry"/>
@@ -56,13 +56,16 @@
                     </mxGeometry>
                 </mxCell>
                 <mxCell id="ur8WBRzY9eALHr9f39F1-10" value="#BatchTransform" style="rounded=1;whiteSpace=wrap;html=1;labelBackgroundColor=none;fontColor=default;" parent="ur8WBRzY9eALHr9f39F1-1" vertex="1">
-                    <mxGeometry x="45" y="720" width="120" height="60" as="geometry"/>
+                    <mxGeometry x="80" y="770" width="120" height="60" as="geometry"/>
                 </mxCell>
-                <mxCell id="ur8WBRzY9eALHr9f39F1-13" style="edgeStyle=none;html=1;fontColor=default;" parent="ur8WBRzY9eALHr9f39F1-1" source="ur8WBRzY9eALHr9f39F1-12" target="ur8WBRzY9eALHr9f39F1-7" edge="1">
+                <mxCell id="ur8WBRzY9eALHr9f39F1-13" style="edgeStyle=none;html=1;fontColor=default;" parent="ur8WBRzY9eALHr9f39F1-1" source="2" target="ur8WBRzY9eALHr9f39F1-7" edge="1">
+                    <mxGeometry relative="1" as="geometry"/>
+                </mxCell>
+                <mxCell id="1" style="edgeStyle=none;html=1;" edge="1" parent="ur8WBRzY9eALHr9f39F1-1" source="ur8WBRzY9eALHr9f39F1-12" target="2">
                     <mxGeometry relative="1" as="geometry"/>
                 </mxCell>
                 <mxCell id="ur8WBRzY9eALHr9f39F1-12" value="#BatchTransformStep" style="rounded=1;whiteSpace=wrap;html=1;labelBackgroundColor=none;fontColor=default;" parent="ur8WBRzY9eALHr9f39F1-1" vertex="1">
-                    <mxGeometry x="685" y="720" width="120" height="60" as="geometry"/>
+                    <mxGeometry x="950" y="700" width="120" height="60" as="geometry"/>
                 </mxCell>
                 <mxCell id="ur8WBRzY9eALHr9f39F1-17" style="edgeStyle=none;html=1;fontColor=default;" parent="ur8WBRzY9eALHr9f39F1-1" source="ur8WBRzY9eALHr9f39F1-16" target="ur8WBRzY9eALHr9f39F1-2" edge="1">
                     <mxGeometry relative="1" as="geometry"/>
@@ -116,6 +119,29 @@
                 <mxCell id="ur8WBRzY9eALHr9f39F1-36" value="#DataStore" style="rounded=1;whiteSpace=wrap;html=1;labelBackgroundColor=none;fontFamily=Helvetica;fontSize=12;fontColor=default;" parent="ur8WBRzY9eALHr9f39F1-1" vertex="1">
                     <mxGeometry x="90" y="120" width="120" height="60" as="geometry"/>
                 </mxCell>
+                <mxCell id="3" style="edgeStyle=none;html=1;" edge="1" parent="ur8WBRzY9eALHr9f39F1-1" source="0" target="2">
+                    <mxGeometry relative="1" as="geometry"/>
+                </mxCell>
+                <mxCell id="0" value="#DatatableBatchTransformStep" style="rounded=1;whiteSpace=wrap;html=1;labelBackgroundColor=none;fontColor=default;" vertex="1" parent="ur8WBRzY9eALHr9f39F1-1">
+                    <mxGeometry x="720" y="640" width="195" height="60" as="geometry"/>
+                </mxCell>
+                <mxCell id="2" value="BaseBatchTransformStep" style="rounded=1;whiteSpace=wrap;html=1;labelBackgroundColor=none;fontColor=default;" vertex="1" parent="ur8WBRzY9eALHr9f39F1-1">
+                    <mxGeometry x="840" y="370" width="195" height="60" as="geometry"/>
+                </mxCell>
+                <mxCell id="5" style="edgeStyle=none;html=1;dashed=1;" edge="1" parent="ur8WBRzY9eALHr9f39F1-1" source="4" target="0">
+                    <mxGeometry relative="1" as="geometry"/>
+                </mxCell>
+                <mxCell id="6" value="build_compute" style="edgeLabel;html=1;align=center;verticalAlign=middle;resizable=0;points=[];" vertex="1" connectable="0" parent="5">
+                    <mxGeometry x="0.1517" y="-2" relative="1" as="geometry">
+                        <mxPoint as="offset"/>
+                    </mxGeometry>
+                </mxCell>
+                <mxCell id="7" style="edgeStyle=none;html=1;" edge="1" parent="ur8WBRzY9eALHr9f39F1-1" source="4" target="ur8WBRzY9eALHr9f39F1-2">
+                    <mxGeometry relative="1" as="geometry"/>
+                </mxCell>
+                <mxCell id="4" value="#DatatableBatchTransform" style="rounded=1;whiteSpace=wrap;html=1;labelBackgroundColor=none;fontColor=default;" vertex="1" parent="ur8WBRzY9eALHr9f39F1-1">
+                    <mxGeometry x="20" y="660" width="160" height="60" as="geometry"/>
+                </mxCell>
             </root>
         </mxGraphModel>
     </diagram>
diff --git a/docs/source/compute.md b/docs/source/compute.md
new file mode 100644
index 00000000..6989496e
--- /dev/null
+++ b/docs/source/compute.md
@@ -0,0 +1,9 @@
+# Compute
+
+BatchTransform:
+
+* Tracks
+
+Гипотеза:
+
+Можно сделать один класс ComputeStep который будет корректно работать с любым типом PipelineStep
\ No newline at end of file
diff --git a/CONCEPTS.md b/docs/source/concepts.md
similarity index 100%
rename from CONCEPTS.md
rename to docs/source/concepts.md
diff --git a/docs/source/conf.py b/docs/source/conf.py
new file mode 100644
index 00000000..effd559e
--- /dev/null
+++ b/docs/source/conf.py
@@ -0,0 +1,30 @@
+import os
+import sys
+
+sys.path.insert(0, os.path.abspath("../.."))
+
+project = "Datapipe"
+copyright = "2023, Epoch8 Team and Contributors"
+author = "Epoch8 Team and Contributors"
+
+extensions = [
+    "sphinx.ext.duration",
+    "sphinx.ext.doctest",
+    "sphinx.ext.autodoc",
+    "sphinx.ext.autosummary",
+    "sphinx.ext.intersphinx",
+    "sphinx.ext.autodoc",
+    "sphinx.ext.napoleon",
+    "myst_parser",
+]
+
+exclude_patterns = [
+    "build",
+]
+
+source_suffix = [
+    ".rst",
+    ".md",
+]
+
+html_theme = "sphinx_rtd_theme"
diff --git a/docs/source/extending-cli.md b/docs/source/extending-cli.md
new file mode 100644
index 00000000..bba0eae0
--- /dev/null
+++ b/docs/source/extending-cli.md
@@ -0,0 +1,17 @@
+# Extending `datapipe` cli
+
+Datapipe offers a way to add additional cli commands. It is achieved by
+utilizing Python entrypoints mechanism.
+
+Datapipe looks for entrypoints with group name `datapipe.cli` and expects a
+function with signature:
+
+```python
+import click
+
+def register_commands(cli: click.Group) -> None:
+    ...
+```
+
+To see example of extending `datapipe` cli see
+[`datapipe-app`](https://github.com/epoch8/datapipe-app)
diff --git a/docs/source/index.rst b/docs/source/index.rst
new file mode 100644
index 00000000..381ec815
--- /dev/null
+++ b/docs/source/index.rst
@@ -0,0 +1,24 @@
+.. 
+
+.. toctree::
+    :caption: Developer Guide
+    :maxdepth: 2
+    :hidden:
+
+    concepts
+    transformation-lifecycle
+    sqlite
+    extending-cli
+
+.. toctree::
+    :caption: Reference
+    :maxdepth: 2
+    :hidden:
+
+    api
+
+Index and Search
+----------------
+
+* :ref:`genindex`
+* :ref:`search`
diff --git a/docs/sqlite.md b/docs/source/sqlite.md
similarity index 100%
rename from docs/sqlite.md
rename to docs/source/sqlite.md
diff --git a/docs/source/transformation-lifecycle.drawio b/docs/source/transformation-lifecycle.drawio
new file mode 100644
index 00000000..0d15cef8
--- /dev/null
+++ b/docs/source/transformation-lifecycle.drawio
@@ -0,0 +1,88 @@
+<mxfile host="65bd71144e">
+    <diagram id="_LABzlGnOxJPrFOFE8tK" name="run_full">
+        <mxGraphModel dx="1492" dy="806" grid="1" gridSize="10" guides="1" tooltips="1" connect="1" arrows="1" fold="1" page="1" pageScale="1" pageWidth="850" pageHeight="1100" background="#ffffff" math="0" shadow="0">
+            <root>
+                <mxCell id="0"/>
+                <mxCell id="1" parent="0"/>
+                <mxCell id="2" value="run_full" style="rounded=0;whiteSpace=wrap;html=1;fillColor=#f0a30a;strokeColor=#BD7000;strokeWidth=1;perimeterSpacing=0;glass=0;sketch=1;curveFitting=1;jiggle=2;shadow=0;fontColor=#000000;" parent="1" vertex="1">
+                    <mxGeometry x="40" y="40" width="520" height="40" as="geometry"/>
+                </mxCell>
+                <mxCell id="3" value="get_full_process_ids" style="rounded=0;whiteSpace=wrap;html=1;fillColor=#f0a30a;strokeColor=#BD7000;strokeWidth=1;perimeterSpacing=0;glass=0;sketch=1;curveFitting=1;jiggle=2;shadow=0;fontColor=#000000;" parent="1" vertex="1">
+                    <mxGeometry x="40" y="80" width="160" height="40" as="geometry"/>
+                </mxCell>
+                <mxCell id="4" value="process_batch" style="rounded=0;whiteSpace=wrap;html=1;fillColor=#b1ddf0;strokeColor=#10739e;strokeWidth=1;perimeterSpacing=0;glass=0;sketch=1;curveFitting=1;jiggle=2;shadow=0;" parent="1" vertex="1">
+                    <mxGeometry x="200" y="120" width="360" height="40" as="geometry"/>
+                </mxCell>
+                <mxCell id="5" value="foreach batch" style="rounded=0;whiteSpace=wrap;html=1;strokeWidth=1;perimeterSpacing=0;glass=0;sketch=1;curveFitting=1;jiggle=2;shadow=0;fillColor=#e3c800;strokeColor=#B09500;fontColor=#000000;" parent="1" vertex="1">
+                    <mxGeometry x="200" y="80" width="360" height="40" as="geometry"/>
+                </mxCell>
+                <mxCell id="6" value="process_batch_dts" style="rounded=0;whiteSpace=wrap;html=1;fillColor=#b1ddf0;strokeColor=#10739e;strokeWidth=1;perimeterSpacing=0;glass=0;sketch=1;curveFitting=1;jiggle=2;shadow=0;" parent="1" vertex="1">
+                    <mxGeometry x="200" y="160" width="240" height="60" as="geometry"/>
+                </mxCell>
+                <mxCell id="7" value="store_batch_result" style="rounded=0;whiteSpace=wrap;html=1;fillColor=#b1ddf0;strokeColor=#10739e;strokeWidth=1;perimeterSpacing=0;glass=0;sketch=1;curveFitting=1;jiggle=2;shadow=0;" parent="1" vertex="1">
+                    <mxGeometry x="440" y="160" width="120" height="30" as="geometry"/>
+                </mxCell>
+                <mxCell id="8" value="get_batch_input_dfs" style="rounded=0;whiteSpace=wrap;html=1;fillColor=#b1ddf0;strokeColor=#10739e;strokeWidth=1;perimeterSpacing=0;glass=0;sketch=1;curveFitting=1;jiggle=2;shadow=0;" parent="1" vertex="1">
+                    <mxGeometry x="200" y="220" width="120" height="40" as="geometry"/>
+                </mxCell>
+                <mxCell id="9" value="process_batch_dfs" style="rounded=0;whiteSpace=wrap;html=1;fillColor=#b1ddf0;strokeColor=#10739e;strokeWidth=1;perimeterSpacing=0;glass=0;sketch=1;curveFitting=1;jiggle=2;shadow=0;" parent="1" vertex="1">
+                    <mxGeometry x="320" y="220" width="120" height="40" as="geometry"/>
+                </mxCell>
+                <mxCell id="GeVadQ1i7Gmi9rNTnFdB-9" value="run_changelist" style="rounded=0;whiteSpace=wrap;html=1;fillColor=#f0a30a;strokeColor=#BD7000;strokeWidth=1;perimeterSpacing=0;glass=0;sketch=1;curveFitting=1;jiggle=2;shadow=0;fontColor=#000000;" parent="1" vertex="1">
+                    <mxGeometry x="40" y="320" width="520" height="40" as="geometry"/>
+                </mxCell>
+                <mxCell id="GeVadQ1i7Gmi9rNTnFdB-10" value="get_change_list_process_ids" style="rounded=0;whiteSpace=wrap;html=1;fillColor=#f0a30a;strokeColor=#BD7000;strokeWidth=1;perimeterSpacing=0;glass=0;sketch=1;curveFitting=1;jiggle=2;shadow=0;fontColor=#000000;" parent="1" vertex="1">
+                    <mxGeometry x="40" y="360" width="160" height="40" as="geometry"/>
+                </mxCell>
+                <mxCell id="GeVadQ1i7Gmi9rNTnFdB-11" value="process_batch" style="rounded=0;whiteSpace=wrap;html=1;fillColor=#b1ddf0;strokeColor=#10739e;strokeWidth=1;perimeterSpacing=0;glass=0;sketch=1;curveFitting=1;jiggle=2;shadow=0;" parent="1" vertex="1">
+                    <mxGeometry x="200" y="400" width="360" height="40" as="geometry"/>
+                </mxCell>
+                <mxCell id="GeVadQ1i7Gmi9rNTnFdB-12" value="foreach batch" style="rounded=0;whiteSpace=wrap;html=1;fillColor=#e3c800;strokeColor=#B09500;fontColor=#000000;strokeWidth=1;perimeterSpacing=0;glass=0;sketch=1;curveFitting=1;jiggle=2;shadow=0;" parent="1" vertex="1">
+                    <mxGeometry x="200" y="360" width="360" height="40" as="geometry"/>
+                </mxCell>
+                <mxCell id="GeVadQ1i7Gmi9rNTnFdB-15" value="get_batch_input_dfs" style="rounded=0;whiteSpace=wrap;html=1;fillColor=#b1ddf0;strokeColor=#10739e;strokeWidth=1;perimeterSpacing=0;glass=0;sketch=1;curveFitting=1;jiggle=2;shadow=0;" parent="1" vertex="1">
+                    <mxGeometry x="200" y="500" width="120" height="40" as="geometry"/>
+                </mxCell>
+                <mxCell id="GeVadQ1i7Gmi9rNTnFdB-16" value="process_batch_dfs" style="rounded=0;whiteSpace=wrap;html=1;fillColor=#b1ddf0;strokeColor=#10739e;strokeWidth=1;perimeterSpacing=0;glass=0;sketch=1;curveFitting=1;jiggle=2;shadow=0;" parent="1" vertex="1">
+                    <mxGeometry x="320" y="500" width="120" height="40" as="geometry"/>
+                </mxCell>
+                <mxCell id="11" value="run_generate" style="rounded=0;whiteSpace=wrap;html=1;fillColor=#f0a30a;strokeColor=#BD7000;strokeWidth=1;perimeterSpacing=0;glass=0;sketch=1;curveFitting=1;jiggle=2;shadow=0;fontColor=#000000;" parent="1" vertex="1">
+                    <mxGeometry x="40" y="600" width="520" height="40" as="geometry"/>
+                </mxCell>
+                <mxCell id="13" value="generate_batches" style="rounded=0;whiteSpace=wrap;html=1;fillColor=#b1ddf0;strokeColor=#10739e;strokeWidth=1;perimeterSpacing=0;glass=0;sketch=1;curveFitting=1;jiggle=2;shadow=0;" parent="1" vertex="1">
+                    <mxGeometry x="40" y="640" width="360" height="40" as="geometry"/>
+                </mxCell>
+                <mxCell id="14" value="foreach batch" style="rounded=0;whiteSpace=wrap;html=1;fillColor=#e3c800;strokeColor=#B09500;fontColor=#000000;strokeWidth=1;perimeterSpacing=0;glass=0;sketch=1;curveFitting=1;jiggle=2;shadow=0;" parent="1" vertex="1">
+                    <mxGeometry x="40" y="680" width="360" height="40" as="geometry"/>
+                </mxCell>
+                <mxCell id="18" value="store_batch_result" style="rounded=0;whiteSpace=wrap;html=1;fillColor=#b1ddf0;strokeColor=#10739e;strokeWidth=1;perimeterSpacing=0;glass=0;sketch=1;curveFitting=1;jiggle=2;shadow=0;" parent="1" vertex="1">
+                    <mxGeometry x="40" y="720" width="360" height="40" as="geometry"/>
+                </mxCell>
+                <mxCell id="19" value="finalize_generate" style="rounded=0;whiteSpace=wrap;html=1;fillColor=#b1ddf0;strokeColor=#10739e;strokeWidth=1;perimeterSpacing=0;glass=0;sketch=1;curveFitting=1;jiggle=2;shadow=0;" parent="1" vertex="1">
+                    <mxGeometry x="400" y="640" width="160" height="40" as="geometry"/>
+                </mxCell>
+                <mxCell id="20" value="Full" style="text;html=1;strokeColor=none;fillColor=none;align=left;verticalAlign=middle;whiteSpace=wrap;rounded=0;fontSize=24;" parent="1" vertex="1">
+                    <mxGeometry x="40" width="280" height="40" as="geometry"/>
+                </mxCell>
+                <mxCell id="21" value="Changelist" style="text;html=1;strokeColor=none;fillColor=none;align=left;verticalAlign=middle;whiteSpace=wrap;rounded=0;fontSize=24;" parent="1" vertex="1">
+                    <mxGeometry x="40" y="280" width="280" height="40" as="geometry"/>
+                </mxCell>
+                <mxCell id="22" value="Generate" style="text;html=1;strokeColor=none;fillColor=none;align=left;verticalAlign=middle;whiteSpace=wrap;rounded=0;fontSize=24;" parent="1" vertex="1">
+                    <mxGeometry x="40" y="560" width="280" height="40" as="geometry"/>
+                </mxCell>
+                <mxCell id="24" value="store_batch_err" style="rounded=0;whiteSpace=wrap;html=1;fillColor=#b1ddf0;strokeColor=#10739e;strokeWidth=1;perimeterSpacing=0;glass=0;sketch=1;curveFitting=1;jiggle=2;shadow=0;" vertex="1" parent="1">
+                    <mxGeometry x="440" y="190" width="120" height="30" as="geometry"/>
+                </mxCell>
+                <mxCell id="26" value="process_batch_dts" style="rounded=0;whiteSpace=wrap;html=1;fillColor=#b1ddf0;strokeColor=#10739e;strokeWidth=1;perimeterSpacing=0;glass=0;sketch=1;curveFitting=1;jiggle=2;shadow=0;" vertex="1" parent="1">
+                    <mxGeometry x="200" y="440" width="240" height="60" as="geometry"/>
+                </mxCell>
+                <mxCell id="27" value="store_batch_result" style="rounded=0;whiteSpace=wrap;html=1;fillColor=#b1ddf0;strokeColor=#10739e;strokeWidth=1;perimeterSpacing=0;glass=0;sketch=1;curveFitting=1;jiggle=2;shadow=0;" vertex="1" parent="1">
+                    <mxGeometry x="440" y="440" width="120" height="30" as="geometry"/>
+                </mxCell>
+                <mxCell id="28" value="store_batch_err" style="rounded=0;whiteSpace=wrap;html=1;fillColor=#b1ddf0;strokeColor=#10739e;strokeWidth=1;perimeterSpacing=0;glass=0;sketch=1;curveFitting=1;jiggle=2;shadow=0;" vertex="1" parent="1">
+                    <mxGeometry x="440" y="470" width="120" height="30" as="geometry"/>
+                </mxCell>
+            </root>
+        </mxGraphModel>
+    </diagram>
+</mxfile>
\ No newline at end of file
diff --git a/docs/source/transformation-lifecycle.md b/docs/source/transformation-lifecycle.md
new file mode 100644
index 00000000..666621a4
--- /dev/null
+++ b/docs/source/transformation-lifecycle.md
@@ -0,0 +1,30 @@
+# Lifecycle of a ComputeStep execution
+
+As a computational graph node, transformation consists of:
+
+* `input_dts` - Input data tables
+* `output_dts` - Output data tables
+* Transformation logic
+
+In order to run transformation, runtime performs actions with the following
+structure:
+
+* `run_full` / `run_changelist`
+    * `get_full_process_ds` / `get_change_list_process_ids` - Compute idx-es
+      that require computation
+    * For each `idx` in batch:
+        * `process_batch` - Process batch in terms of DataTable
+            * `process_batch_dts` - Process batch with DataTables as input and
+              `pd.DataFrame` as output
+                * `get_batch_input_dfs` - Retreive batch data in `pd.DataFrame`
+                  form
+                * `process_batch_df` - Process batch in terms of `pd.DataFrame`
+            * store results
+
+* `store_batch_result` is called when batch was processed successfuly
+* `store_batch_err` is called when there was an exception during batch
+  processing
+
+![lifecycle](transformation_lifecycle.png)
+
+!! Note, lifecycle of generator is different
diff --git a/docs/source/transformation-lifecycle.png b/docs/source/transformation-lifecycle.png
new file mode 100644
index 00000000..dd4cc542
Binary files /dev/null and b/docs/source/transformation-lifecycle.png differ
diff --git a/examples/.gitignore b/examples/.gitignore
index adbb97d2..990a377e 100644
--- a/examples/.gitignore
+++ b/examples/.gitignore
@@ -1,1 +1,2 @@
-data/
\ No newline at end of file
+data/
+*.sqlite
\ No newline at end of file
diff --git a/examples/datatable_batch_transform/app.py b/examples/datatable_batch_transform/app.py
new file mode 100644
index 00000000..4f958fb7
--- /dev/null
+++ b/examples/datatable_batch_transform/app.py
@@ -0,0 +1,111 @@
+from typing import Dict, List, Optional
+
+import numpy as np
+import pandas as pd
+from datapipe_app import DatapipeApp
+from sqlalchemy import Integer
+from sqlalchemy.sql import functions, select
+from sqlalchemy.sql.schema import Column
+
+from datapipe.compute import Catalog, Pipeline, Table
+from datapipe.core_steps import BatchGenerate, DatatableBatchTransform
+from datapipe.datatable import DataStore, DataTable
+from datapipe.run_config import RunConfig
+from datapipe.store.database import DBConn, TableStoreDB
+from datapipe.types import IndexDF
+
+dbconn = DBConn("sqlite+pysqlite3:///db.sqlite")
+
+
+catalog = Catalog(
+    {
+        "input": Table(
+            store=TableStoreDB(
+                dbconn=dbconn,
+                name="input",
+                data_sql_schema=[
+                    Column("group_id", Integer, primary_key=True),
+                    Column("item_id", Integer, primary_key=True),
+                ],
+            )
+        ),
+        "result": Table(
+            store=TableStoreDB(
+                dbconn=dbconn,
+                name="output",
+                data_sql_schema=[
+                    Column("group_id", Integer, primary_key=True),
+                    Column("count", Integer),
+                ],
+            )
+        ),
+    }
+)
+
+
+def generate_data():
+    # N = 10_000
+    N = 100
+    for i in range(10):
+        print(i)
+        yield pd.DataFrame(
+            {
+                "group_id": np.random.randint(0, 100, size=N),
+                "item_id": np.random.randint(0, 100_000, size=N),
+            }
+        ).drop_duplicates()
+
+
+def count(
+    input_df: pd.DataFrame,
+) -> pd.DataFrame:
+    return input_df.groupby("group_id").agg(count=("item_id", np.size)).reset_index()
+
+
+def count_tbl(
+    ds: DataStore,
+    idx: IndexDF,
+    input_dts: List[DataTable],
+    run_config: Optional[RunConfig] = None,
+    kwargs: Optional[Dict] = None,
+) -> pd.DataFrame:
+    (input_dt,) = input_dts
+
+    ts = input_dt.table_store
+    assert isinstance(ts, TableStoreDB)
+
+    tbl = ts.data_table
+    sql = (
+        select(
+            tbl.c["group_id"],
+            functions.count().label("count"),
+        )
+        .select_from(tbl)
+        .where(tbl.c["group_id"].in_(idx["group_id"]))
+        .group_by(tbl.c["group_id"])
+    )
+
+    return pd.read_sql_query(
+        sql,
+        con=ds.meta_dbconn.con,
+    )
+
+
+pipeline = Pipeline(
+    [
+        BatchGenerate(
+            generate_data,
+            outputs=["input"],
+        ),
+        DatatableBatchTransform(
+            count_tbl,
+            inputs=["input"],
+            outputs=["result"],
+        ),
+    ]
+)
+
+
+ds = DataStore(dbconn)
+
+app = DatapipeApp(ds, catalog, pipeline)
diff --git a/examples/example_image_pipeline.ipynb b/examples/example_image_pipeline.ipynb
new file mode 100644
index 00000000..f124f96a
--- /dev/null
+++ b/examples/example_image_pipeline.ipynb
@@ -0,0 +1,344 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Image pipline"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# flake8: noqa\n",
+    "\n",
+    "import numpy as np\n",
+    "import pandas as pd\n",
+    "from PIL import Image\n",
+    "\n",
+    "from datapipe.compute import (Catalog, Pipeline, Table, build_compute,\n",
+    "                              run_pipeline, run_steps)\n",
+    "from datapipe.core_steps import (BatchGenerate, BatchTransform,\n",
+    "                                 UpdateExternalTable, do_batch_generate,\n",
+    "                                 do_full_batch_transform)\n",
+    "from datapipe.datatable import DataStore\n",
+    "from datapipe.store.filedir import PILFile, TableStoreFiledir\n",
+    "from datapipe.store.database import DBConn\n",
+    "from pathlib import Path\n",
+    "\n",
+    "\n",
+    "def make_df():\n",
+    "    idx = [f'im_{i}' for i in range(10)]\n",
+    "    return pd.DataFrame(\n",
+    "        {\n",
+    "            'id': idx,\n",
+    "            'image': [Image.fromarray(np.random.randint(0, 256, (100, 100, 3)), 'RGB') for i in idx]\n",
+    "        }\n",
+    "    )\n",
+    "\n",
+    "\n",
+    "def gen_images():\n",
+    "    yield make_df()\n",
+    "\n",
+    "\n",
+    "def resize_images(df):\n",
+    "    df['image'] = df['image'].apply(lambda im: im.resize((50, 50)))\n",
+    "    return df\n",
+    "\n",
+    "\n",
+    "def test_image_datatables(dbconn, tmp_dir):\n",
+    "    ds = DataStore(dbconn, create_meta_table=True)\n",
+    "\n",
+    "    tbl1 = ds.create_table(\n",
+    "        'tbl1',\n",
+    "        table_store=TableStoreFiledir(\n",
+    "            tmp_dir / 'tbl1' / '{id}.png',\n",
+    "            adapter=PILFile('png')\n",
+    "        )\n",
+    "    )\n",
+    "\n",
+    "    tbl2 = ds.create_table(\n",
+    "        'tbl2',\n",
+    "        table_store=TableStoreFiledir(\n",
+    "            tmp_dir / 'tbl2' / '{id}.png',\n",
+    "            adapter=PILFile('png')\n",
+    "        )\n",
+    "    )\n",
+    "\n",
+    "    assert len(list(tmp_dir.glob('tbl1/*.png'))) == 0\n",
+    "    assert len(list(tmp_dir.glob('tbl2/*.png'))) == 0\n",
+    "\n",
+    "    do_batch_generate(\n",
+    "        func=gen_images,\n",
+    "        ds=ds,\n",
+    "        output_dts=[tbl1],\n",
+    "    )\n",
+    "\n",
+    "    do_full_batch_transform(\n",
+    "        func=resize_images,\n",
+    "        ds=ds,\n",
+    "        input_dts=[tbl1],\n",
+    "        output_dts=[tbl2],\n",
+    "    )\n",
+    "\n",
+    "    assert len(list(tmp_dir.glob('tbl1/*.png'))) == 10\n",
+    "    assert len(list(tmp_dir.glob('tbl2/*.png'))) == 10\n",
+    "\n",
+    "\n",
+    "def test_image_pipeline(dbconn, tmp_dir):\n",
+    "    catalog = Catalog({\n",
+    "        'tbl1': Table(\n",
+    "            store=TableStoreFiledir(\n",
+    "                tmp_dir / 'tbl1' / '{id}.png',\n",
+    "                adapter=PILFile('png')\n",
+    "            )\n",
+    "        ),\n",
+    "        'tbl2': Table(\n",
+    "            store=TableStoreFiledir(\n",
+    "                tmp_dir / 'tbl2' / '{id}.png',\n",
+    "                adapter=PILFile('png')\n",
+    "            )\n",
+    "        ),\n",
+    "    })\n",
+    "\n",
+    "    pipeline = Pipeline([\n",
+    "        BatchGenerate(\n",
+    "            gen_images,\n",
+    "            outputs=['tbl1'],\n",
+    "        ),\n",
+    "        BatchTransform(\n",
+    "            resize_images,\n",
+    "            inputs=['tbl1'],\n",
+    "            outputs=['tbl2'],\n",
+    "        )\n",
+    "    ])\n",
+    "\n",
+    "    assert len(list(tmp_dir.glob('tbl1/*.png'))) == 0\n",
+    "    assert len(list(tmp_dir.glob('tbl2/*.png'))) == 0\n",
+    "\n",
+    "    ds = DataStore(dbconn, create_meta_table=True)\n",
+    "    run_pipeline(ds, catalog, pipeline)\n",
+    "\n",
+    "    assert len(list(tmp_dir.glob('tbl1/*.png'))) == 10\n",
+    "    assert len(list(tmp_dir.glob('tbl2/*.png'))) == 10\n",
+    "\n",
+    "\n",
+    "def test_image_batch_generate_with_later_deleting(dbconn, tmp_dir):\n",
+    "\n",
+    "    # Add images to tmp_dir\n",
+    "    df_images = make_df()\n",
+    "    (tmp_dir / 'tbl1').mkdir()\n",
+    "    for _, row in df_images[['id', 'image']].iterrows():\n",
+    "        row['image'].save(tmp_dir / 'tbl1' / f'{row[\"id\"]}.png')\n",
+    "\n",
+    "    catalog = Catalog({\n",
+    "        'tbl1': Table(\n",
+    "            store=TableStoreFiledir(\n",
+    "                tmp_dir / 'tbl1' / '{id}.png',\n",
+    "                adapter=PILFile('png'),\n",
+    "                enable_rm=True\n",
+    "            )\n",
+    "        ),\n",
+    "        'tbl2': Table(\n",
+    "            store=TableStoreFiledir(\n",
+    "                tmp_dir / 'tbl2' / '{id}.png',\n",
+    "                adapter=PILFile('png'),\n",
+    "                enable_rm=True\n",
+    "            )\n",
+    "        ),\n",
+    "    })\n",
+    "\n",
+    "    pipeline = Pipeline([\n",
+    "        UpdateExternalTable(\"tbl1\"),\n",
+    "        BatchTransform(\n",
+    "            lambda df: df,\n",
+    "            inputs=[\"tbl1\"],\n",
+    "            outputs=[\"tbl2\"]\n",
+    "        )\n",
+    "    ])\n",
+    "\n",
+    "    assert len(list(tmp_dir.glob('tbl1/*.png'))) == 10\n",
+    "    assert len(list(tmp_dir.glob('tbl2/*.png'))) == 0\n",
+    "\n",
+    "    ds = DataStore(dbconn, create_meta_table=True)\n",
+    "    steps = build_compute(ds, catalog, pipeline)\n",
+    "    run_steps(ds, steps)\n",
+    "\n",
+    "    assert len(list(tmp_dir.glob('tbl1/*.png'))) == 10\n",
+    "    assert len(list(tmp_dir.glob('tbl2/*.png'))) == 10\n",
+    "    assert len(catalog.get_datatable(ds, 'tbl1').get_data()) == 10\n",
+    "    assert len(catalog.get_datatable(ds, 'tbl2').get_data()) == 10\n",
+    "\n",
+    "    # Delete some files from the folder\n",
+    "    for id in [0, 5, 7, 8, 9]:\n",
+    "        (tmp_dir / 'tbl1' / f'im_{id}.png').unlink()\n",
+    "\n",
+    "    run_steps(ds, steps)\n",
+    "\n",
+    "    assert len(list(tmp_dir.glob('tbl1/*.png'))) == 5\n",
+    "    assert len(catalog.get_datatable(ds, 'tbl1').get_data()) == 5\n",
+    "    assert len(catalog.get_datatable(ds, 'tbl1').get_metadata()) == 5\n",
+    "\n",
+    "    assert len(list(tmp_dir.glob('tbl2/*.png'))) == 5\n",
+    "    assert len(catalog.get_datatable(ds, 'tbl2').get_data()) == 5\n",
+    "    assert len(catalog.get_datatable(ds, 'tbl2').get_metadata()) == 5\n"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Images"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# img_df_1 = make_df()\n",
+    "# for index, row in img_df_1.iterrows():\n",
+    "#     row[1].save(f\"./data/tbl1/{row[0]}.png\")\n",
+    "    \n",
+    "# img_df_2 = make_df()\n",
+    "# for index, row in img_df_2.iterrows():\n",
+    "#     row[1].save(f\"./data/tbl2/{row[0]}.png\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAGQAAABkCAIAAAD/gAIDAAAZtElEQVR4nGXcd7ik5xgG8BFB9FGil1k2ImKZCKIbCRE1EzVqBqtEWYMoUY8SkghG9BKZVYNgRA2yhmhBOKKHMKJFi08XbPjD/Xuva5+df+7rnJ3v+973KfdzP8/7ne3t2fv/59zgo4PL4MnBZwbvH3xS8BHBXYKnB18dPCl4QfCGwc8F/xD8Y/C04M2CrwxuDr4h+JPgVYJXCH6o7GNT8IHB/wQfFrxz+f1ng08u178m2Ht38LvB1wZ36+34+UJ58L7BDwRfFNw9eNPgE4N3DJ5b0L+/rPze/Z8eZMgflu9fN/jC4P2ClwoeGXxo8BNBhrLec4IHBd8f3CvY+1Tw5kGee0qQpy4dPLQs+PnBlwe74DeD+wX3CT4k+OLgOHib4PbgI4OPCb4leNmgSBPhXwmuB23478HLBX9bvn+T4GkFZYwI/1ivLPjTwSsFjw8KwdsHef6fwfsEeeDWwbcHeZQj/hr8UlBkuF7EXC34o+Dfyv1+FfxGkAE/GPxdEJW8NHhWEEX8PnhK8LwgQ7pf8/hfgj8ISjU3Oj/43OC8LGhSHvTeIA8L/X8F3xQUQbsGRfJaUGpdLyhl/lSuY2Dc+OYgg38x+OGgzPlIue80eJkg7rKf3jOCIuiQ4DuCDMRgvw6KwBcETy7/7mcGs0GRxdDuc+XgMijlceH1gxcJviR4pyBHXSv41OBPgz/v7fgRYagCx4r4jcHOBZcPXj0oZ4W4iLOxLUFV9PNBnlDdFAORKsXdH4dI1UsGx8GPBpE/A904+Pogg7rv7YK47NiglHtc8K3BBwdVPfaw71sGDw62quPCZwWvGHxF8Brl+68L4qxF2cjPgutBVekJwacFbWgWtOGzg2SB+4n4E4OKy1FB8kR1Ezkcc7HgqeX59wzeO/it4N69ckMhfEK5sEYEXXPfIDnwi+D7gjgEd/wyOA3y3DHBxwdVZWQ8DL4neGGQx/cI4jp6TYodHaTrpDryFqlbg+TGl4N0V9NZQk/qkRP0Cs6xsFsFuyCOUjUI1zcGkaqN3jb4mSDDiQyUgDPWgtuCUhv5ihSGcV/67JpBEX+Psk7FA+mrulXutAVKrYsGLxFUrZAkLhKqJ5bv3SAoAt8WvEuQp0Tk94NS/+PBPwdVz1cFvxdU5jmawdxHZBweZDARhWs3BOdBnYgA+E7wbr3yCxY+IMgwyJvBbERoijQpp5qSCxxCpyFbkXxYUER9Mqj6KiYXD+LKSfBrQZGt1bFBEbUMEroiX/XkOPfh0MZZyPO/QXpC1ZDDSJOS5hFc9/AgblOGCUatA07UUnm+yL528A5BEaEKS2FU8e8gQavHsw4CWer+Jvi8oMjnaJ0EuaQqS93WrErBfwS1HjcqC1c1DwwynAi0oHsFGV7RoMClAk78epAQVixUxwcFpSJ5IHU4lpxgOB2CFk3EKkpam68GGYjB2seCpeCWgkhdlXlAkE5CirjDxkQQQ6ueUnsVJDinQanIUHpMDrVxMsD0QESIZMVBVRWJyJqDpCQdpYoqRuSLKUpvEJSjyvQqKDVUDx6iv54dVK2QqA1pWZRtkUfH0Um4htLnYZEo0kWSCKLz3hnEMSJLRpBJ1ym/XwXpyFqc2KP3riBD9IMsSgB6sEiiqOW+nowSl5oiZBjkURGn2jHMKChyfZAuvaZIiDitj0hUFU0NFBEOYmD74XCZZjqxCMqARl5aFVVEszsNEnA4ZRT8cRCXIVUyQjWUgmSECJXq28rPDGPOptdbBnEYQxG05IJIPCM4CxrPMIgMELEiXdHxvCYDeBLJqR42KtLOKkipI1sGMlbBMQyt6abcyYdBkH7DLVqRSs6KB4No2skbglJVJB9MKThsHNSka/JnQRyKC5tQ5CEbtACkTT9pTVQThqCgVUm5rnjgOAvFYeZi9BTdQ/+hCg7CVQwi1RkERRCyHMrAHKDq61kZjD7EoSKwTQCNOShxkp9+emzQXAhJaokY2hzMRBNZL4IihvLHJSKCQxiCflJkNNPGOpWETWCtm+EofjLm9ILsQPkzdNWXzaI8LKXcGIdpXvVcejOj5V3K95GiB5lkrgWVewuyIeQtkhQB0wrPE5mLoEgQkYrSLYJnBmWEwaDDGMJcZqCSLthaBsN7Uwbk5tBCdbMgC2DYfhDHfTvIYIuyQI6gX2wciSNrSl8LJHVEhI0rRvScKl0jFieJRBmze/k+nUaOsEubqSun60GpNQ6KKAtU/skLkaKpxU3GI+SBFKbYGZahVFO9G5K1LnMo6yGITTjXgsYuHE0WqOYMLOLoM/teBdthhQuOCKoWjsc8SA9nAVJFc0zICWFCUvVSBckLnmUQ8kBEUdYiDhfhuOOC60FUQokbLKIWuosOFGlkzywoM2SayCY3GicgZeTOkDiAEpeKHsjyyNeYxcJ4XqgziIWtglJEE+55PG0agvQ9j9ygr+pI2hlDrYocpLo6zSKnFkGGbhtCZhZiYPeoIFnAE8iakOVRqTEIGs1aoCplWiAyHSasgqYOFDqdg1M4mAFmQSTNkOQP4bkMkhsEsLGPWbxUZUDFrXECssUxBB15oCXogs8J8riQZXAPUg1tWGu1HsSZWg8dhdbF+ERvZz3LoEGlppouUkQ09XSYYoFjySbyxGEGClKFWwj76LYZjsctxDmbSJSSQnoVVD2MjI1B6B5C9e5BYxfcpzXSqhxafm/gJ3JQAYeQHbhOhvSD0yBDaO5Flmqvl8XlLUdVLyfHyicDWpCN8xCPKwqqCo+7bhLUETCYyLFBnKnprQYmHOkyBuRgXKX64V56iuC0Lqmr2Niv9SsWTVF7kIWSAzzqQeZag+AoKJUJU1yoV9TDOYWRKqYVqqMN0jdSCHeJPBHPgCIbmWvKUYU5nFQnMwhXVbwfZED2kfpNNnj5g36ywP2D9I2N4jbTAVXDgxkGOXLIOKhoqMKqq4mt4mF9W8vPZv+mD4oPrnU9A8sc1dL6BIhTIa2YfRoctqpnIXo7CzAPGgbJAFyjqcZp7mfEqyjQSw48Ne+4T7PsvjimCmW9q8hF2qqw1kmKoQYOkIJek1or66HLGE5EUvrNsshQWaWXRAqlqzk2HUB+5IPQJyw1vQzomElvaeGqn5QSASKzCkjyRYporVCHlsf9kDcHuS+hjAvJEZ2FfffuGtTjaUH0WLiB4ONpqerURvkXGcouzztmMv7AVVJXqhCk9f0vVXFT+VmLZB5Hx0kd1ZpBPcfzyRu6znVkkExCEW32riURiqugaiXllFunOW6Ee5RzGzCnwhHmWhS8SDatOK+g1okeGgQ5sB+k5zTbMqK+G6FaiiSpi4rIHXqPwXca3o/LDWdBG3ecxSDrQQND1emw8n0tjpbHG3kWtE/5HsMr+6rb5oLzIG7EueSP6kg4kwWoh850H5mDEjy3zbNUJVXPVKHKB2SH20SiZtkgzkKFslRSBOgjKawlumpQUZkEccsgSIZYt3GNDZE5MoHAtF4RxuCoQVVXXLqg1G3cZAMMJrXoJKTnLeFx8MCCyN+oWbduSqEpngXpHLJFh6BIrAW1IA4+yQBnBRyKK0WOlkaVljF6R2cL5naKiCKkZSJ8G6mPggxDgNbU0pQic+Su51Nm5TqStAEG3a/8HkepmlJYajgOWwZF+JHl+6q0amrjDCYTFB8cajavSVcN24fn3RCZ8hidwiBSS4tUWw+h7yRY2SYjVkHc4djJ8xlWyis2HNcFpRBOslFFg+NEkIhz0s5wphgMy4EiW3Ez32ohRlcxmN5QTuOGeRCpa2qNO7QSeiscMAqqVjiEzpIiUgdFqE48LXVxTJ3H0Vt6y0lQC2OfekH7ohcZzMRWarajdjrLeZ7IMNfBGWbo3rZhMPMpuoenpY4IkPKqjQXq7skRqWHw6LlaLr0kbhUB5lLjoJ6V4+g3GaUFMxTQ22p1dBajYJs/6bVY/qRywTLYlQUqAlKVYXEc4YvcOWYeFMmEofJP/2lmp0G6yDpd510F+gvpkyscScFrnUxkpSTqwIHt/SwW1ZziHjdWjpGuMYpySigau4ggqeOB9BI9p5zjLKGvmJATUkQEmbgqQnpOEUkAM6R1S1WGJGPcn9zBcbhZ5A2DLYXcyLRhGkSGuM18B2njDr1eF8Qd46ABGz2Fk2rz7DpVTqqPgiJaJugoCFzPQcoEsPuJHDKEfMBhhDo54RWDnQ5MB0HkiGOMO3heqtS3mQ3qRBYy98F1FDdu7IImpqrfLChCTA1kwjwoMkQKvag4KT5m/lJZitbDDELYYNNzdjqBNYvWK/G8qkCYao55luek8ChoXILUtTae6xjLwI8n6TXjFRubletwGmVv3QyFk6S2juKg8nvvyFL4Mo2B21SBcrZQ+okhpAz9YSM2rErZALK14FnQSbdu3/RgEDTO0XLhQmRM6Jo3WbdRNwPQe6YQxjAMNwzSde5vrKMVo/NkQPsHDzC/EYqaSZbWA8p9ZVbqWRDDKf+qoAgibBkM56iqNiiyNPOKEgdR8jjI2MfbzSKNXFBlUQzHoBoGw6Ge13JVV+5vdLQEZuJIUhUTOch/GaTLcNUsqDmur06uBXUMWiGGIju8vTMKKiqmH7iKY6vhcZqJKA6zHxFq3QKDozXtTU8RpAZ755cvIm8GkZJ0DrLHWSKOXNBcL4LzoKqzFlRsVEXFxDqk6jBowyKFgFUEpK5RtM5Ey+T+ONdgcBnUm7bQFlGqkA3r0Shxoa8se+AsqBiQH4qBHnMVNJEU0aoX5S9SVFmcwqGKjZTmSBFPkNKDKGJR1s0BWhtvPuIqXCgwWpURSVKLoJQ6qgSdRZcQcK4bBm2coPRyCbKk8HEgbhoFcZrjMMVEVeVoHQSHmDroDKzXlAFHSnVUI1A04fUPFNoXp0G5vqGgyEKCmmWG7AelotSUUkLdOAYH6ghUTbLCuEYE2aD7kwuaeE22t2sIU8/BcaoxEq8T1O3lZ9UUd7actkBVzIYNwIS0hWg6LRQ3yXGGoV+QLNlAOW8pP/O4t2qMTwz8pOwwyMCUvfU5/ND6qMYmwJ4j8rRkHMjwrTekn3jeuwLKq2qgt9Ir0mM8J/elLjmit5LzXnRTxvVy9aURKaYV83wjYw6Q+iKEjJCqrheh9kMoi1AGXAtyqM6h/Y00XWUB9BOPbCyIDHlCbg+DQphnRC5DijCCE+eZaxkbTYKaW6Std+No8qf+KQyOklo4twvqbQlZ1VsRqdOPtiFfoEsIOXMg1UEVUb6livGLcY7zPR2AyPKugfsQhFJdRKrCelQ6S7UiQ9yPvlK9yCHcah2aeI4nJ5zAk0XL3o4f3N4EGhIj3BwTCWWywYNFJBkxDCJdTaicN7AbBOkZHFHLvyo0Dqra1kEO0Fc4j8HJCimkl2VA+9W5MBzOtu9psEXWJCjkHMWrfutBk00LFxGqJSG4d/l3qWUhyJdB65+gMJhRs5QTyQxNbkhxsoSBFAOzeetC9oqR4uX5yN8oWsS2VsTJsAXWSanmWSThBJFovEF5MzSSVLWMhaRajRiHF0hWUTAiloKqsOu1Pqp4PQ5DDf2glDO7ZwfkXw9D2obkvpDugkgRiWoJDin/LmQJT6GOy6SAnsv9RRiBK1KkmFHvsvwe55nsanF0DiahBoQ4iuPJDAZU3PSI1oGzpXTjBApYNUS+Rr4sTuk6nRFxUteBqZDeVLBykqZahNBT5kg2TA9xjFaKTlOFUYnr50EpbB8cimNRjPUpeu3NP2WcXqF3bERvxoPK6yBoLCK1uqC3VpzSSF0pVQ9eGdxzTEzrn+NxoJdItDZaIVXzqPJ91XIYNHaZlOtwE4MLnHmweUb3bqGVawg6qaaKqHJaG0qeTtMycQjhiJOMZ8yPcJLUUTysj5A0mycotV6Lsm6RosW5sPxMGIsgnEmmMHSLHFXEhpVfnIOjKGwzbAurgzbcZuLqOkLW+R1D68FEnM8kKEI1x7gMFxK+zghwjZ5S70kAi0yveuoYGN5ztIBtUkrxUsh6MR8b0NrYKJ2iHCvXqqnI1LxqVRyEUuw4SYSpkuZcuIaixjGeiwJEIIdIYamIpLsgaiBnpJ6iZ0wlglsIigRjERukP1zo6FzKMBhlrLm2YfJCS0E4Il/krkoxEEOIfMUHKXOEc0dCmT5SHckZvaiecRQUefSh98M40HUM3wShUK/zHTpja/k+D9M1Is/sHmfQP65DvppxhpPK28v3CFsRpAp2QbqOTuMgPStqkbr01KiscxycBO0DxzbPuaHqYhbOM8hwFcQJWgTXixQcR9iRFa4TqbisHrzWVorBGIJjnE6RE6hABMuU+mrBIqgKq4IiTVOuM1B0Wvet9UC66+WBcnka1MKIjGVQ6hok4iqkSp6ILE0sLkS6/SC5wQGO7Ot7YSa9DnBFhtTWgxLGHI2CKH1qAJfLsCYccQqP2wju4GHVEYnTKzzgxFhqqKoWpCotgxQ8g66CUkjKdEG9nJ7N4QpBSSjPg47FTD1EsFRmGEUAR1kfqhFQ7UTWxvVM/aDQdBjhZRAbsMBhUMiKSKmNJJGvcq0aEZ4GdRwm0ulBRUkKGjx6rtQSUSLPc62TrBHJikfVW6ildeVCTUo4THAj0wlvq+gJRQru4wmRgktwEZIWeYqGlFH1lHGvgys6UhuH4UoOFsFIexLEvV6vosxVPcKVnMG17mcdrdzW1FGeVTvkitukpEGgUEfmDC3ChuVnC5Qa7qt1EbEiB0ci8W0Fkb51ogbVTLOOY00bZI53PQQQpT8ItnJd/yhIqAt9G6OrTCeMZ8yXRkFKme4SKcq3FNIKaZ4ZfBgkRAlZekgkUdyEs85C5FP2WjCpyKH2p7n2fJw5CaKqRsosLPRYWgthLGKMgRxXQcrfEbhUtqFq0D0LSt1BEEcSkqYZuEQvqLlH2jhQ5NN3DGwCrPcTAByiuWdI+rNxjlMb3OI0xQFo/ZMOVRSpHl4QqVLWWg2exyHmSqvg2eV7Uk9EciTHMMiZBUUq0iaMNecizqmQSO8HBRDKcELeqg0PI1MXal3mQeXcglQP4x45b0rhfS/3R+beJBTi5IgJaH0TD1XgWDKnDhZxKO6ix/S+9BVdNw1quXCd1o5Bm2fdSKo4VaFflH0e6oKaTL2kB4ksGycovVNBWJpIImmjbLpGFSYnlHsUcGr5vqKC7KWQCGVQBt6/fF8xEUDs0/7YCelRuDY0D0oJ5dqwX4tAmZMPzvkGQRyHk2zMdMBEUiuCM5A2ObFb+XdTBgY2GuZALRrHa2FUQevAgajI/nD4Tn9pYZBHsROobmRDLG2j4+B60MCPIckRHucY91GFeFZRwYlSUTUmA+gl+kdG4EgOR+a4DKmTBfQYajDYFBgc006kCUMGYTiptAxSxrjG6JnHRIBZulRTVYx19HBSSEuCi7Q0uE7VE+HkiHXhFocqUsjMn6NwnHchyAfyh2EY3MicA1tvJsSFJg7R9KqayF5vhlNEgt6KcscVIskCVcdp0IZGQRt36qQacyTD6ulEiFMdkWGjRtooAgeqygwj9TnWuEjkt1TRihCKqpOje5FnFs7Ajp+E+Mbys4XhBNzCUXSchYpMkUpnuU7LY/AnlWWIYmXC6n7mYxzqeQ5JcB0Dm15oqZqOUl1Uh1X5vQNY3METyJ3cYBjkysDuY9wi9Pct/46DCEQRxxBSjJDU5B5XfpYZ7k8f4jY6isEZlM5TjLxS0AzG4jjBR89GeJoC4CwPZgADQzqGh3CKwZ/XkBwK0Fm4RIraMLmiZbER6/Kat6KzDI6CHOdDt+FM65NyXdAJePvDBxFimkCxOvnFCQQcgaaH4qF+UIvjOhwoNU03cMcsyFFShJD0HClCmFqfyFMM9I6qm8giUBUtKU5HGhcR0rjSc9s7pc7vpkGhqZrxBNJbBoW+pplhPIBh6DcGqa8T+UhFjvGXECKDXJD6yNx0RCdBf+kwtErHl9+LTMVEhjCcSCe0m2LFLfSHENScmkaoMlJHVTKCpaOMV4xDeIaSFxnjoFOVZdA5ouc6JiMQFRUprapJJYNEnOP8rx+0P/dTzckcgYJbUUvztPLOkoTYNHhB+XdVSCpKaT2WsYwUsRARJoIZWkSRG+ZaSB93qKKqnp6T7hOhphGu36t836BzHpSCxk8Gg43LNb84RtWhh+Q+EuQRrQAS5jnHZtOg4mG8ghsZmH6xcZxDuXtnYh7ENXQfwSrSCVEGdx/yQEqK2Pq+l9SjE3FkezmEbvEAntZsEqZ7lH8Xwl6qMC9y1K1lUsX0XO57RvlZq2Qe5blaEE2wKoq0OUQVpcTJGYbiMNRhv+6vhzVaVqxEYJsL1Ymh055xUPdOkNItFo5jeOiE8rMq5D70zMHle7jCPE1VNHjUKklhKeWUh0OlnuKkNzTl0Duq7vVtID1xfYGucZZyjqQJP8p7GTQgrBxjgVohUwQpY6N0muJhWkEnMZT7iIhzys+oobZEUqeujyJXPR2u0Fsil+MUFYZt5dSsW0rRLyaIyE8PptmliwZBk0yp53r3N7kkQwwSkfcwyMNaIa3TIkioInHrVr3IjmlwFjylfN+LeuZV1qGXdMqD9NuFTnUYhMXpJuRo4VKqCyJHPeMoSLcoz7p6C5yVBWqWbZyOE5lN95T7oxTciVJENg6lBxlGEdCBaH1UVQ5rukPv016CKDecBHctiCNE0BHl9xbGEf72ZhaUIoOgVGAo3IgacFQXVLWkDLmit3Q+yLEOfFVR8kDq41AOpxZwYmtO5TjByEMWrtcjL5C45prgbP/DRj6by8/mURvKv5uXcRihihIIaKln1F3/OFzVnQRNUnGcYQGZgTt1BgToWtB04n+fJ2lMwr97XQAAAABJRU5ErkJggg==",
+      "text/plain": [
+       "<PIL.Image.Image image mode=RGB size=100x100>"
+      ]
+     },
+     "execution_count": 3,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "make_df().iloc[0].image"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "image/png": "iVBORw0KGgoAAAANSUhEUgAAADIAAAAyCAIAAACRXR/mAAAVTUlEQVR4nCXY13IcWbYY0H32sZlZ3sODINjNNnP7zoMiFHL//6AHKaanmwYk4U0VCmXTHLv1oPUZC/7LsC/Ffx/9L5P1zi9+4UVR/KKxhdkfvxW8+C3/Wav8ZMY0wAx+liYfnhWF4r/NWkWH//5fJ+2W/OcfY2CS4dBoeTqb5oX66fepaqnpf/5TZe3RxT8Nb5/mfxTQ/oXPMlTvso+FyH/rgyxg/Ov/VN3e+H8YKbDf+W8guzDs5iB+wf/tgqBNb+VCeN72WCPE7Shz8t1LagT86Oca+DTNpNGT0+ckXNP1GJMup9ar+1fDAse3WvN00D0TwvR6LWq4/y6VZZP7GmQUo7lUbjz91mB1I/ssk/LsLmq7OywY57xYQvSw/cB0rg/vCtb8KqoIcY+DwjO+xqfc2A+ntyY2p9WMpPKje0P2zDUupbn1Jaa/TWEsP3lwICn1M0N45h4bsvd0kLhuslcb/WPtWYvEf3b2SfxZrXUIB+uaiQTDX5gs9JiQg2qfcm86r4fklF14Hqi9yVhEXm3JxfhyySDX6CvkMSvOXDKLescgKLq1VfPwb1MJ9mUAIcGqcdym/CkFoq1vW8Wfp8sGwjX76Fi+YCE6Kp8ZEbiQ8YTdxmuZjkZHFHR8O0pRh933jOpz39RV/P71S9TV5hgkY2NEAOC0YxRF2DUsfucvDBzHXkBPYvlNUT2mU5+bxQfLGbVlySF2mDUmnZ9qQ/zUJc6gUCHVVF3vIAWoEico0nPG7UW7K6UYjr97aBa0kSoOZrsqi1eXMVmqHspKxC/tQiY+sW3v+PrlxrL6ie+Tinb8DWRNA5khu8y2FsIddgsSPPWtZfQEHe/EcnHCUGM+MEmelg+B201rHFHuDQSgLVNZxMvteWKZ008quNFGOIEvXcdZ6vDTGPW+XqXG25tS+Nhe3XLdFMfvTczOqlcBvgtnoA07OGFkMNaCUgsmAIoJYRV7mLYFiC66LjEeZZ9ZwZ/+L8JegVGEBy6vEv/UpERgfeMwzfOlluFwOGqMvDnfcOHzjubI2tlH5HnevvKsXogXdE4/C06sFStwEZ4wBfDrKmIqdQUQiV2L2BT7bYLYUMCYzHbJnW89zRhXokgMSKF7lTYM7jVyVPk7VMZMnVe0mkEm4f0UiSBALpEPdB491i9D5eV04QRRJ3QdF8/5Hlk08if0efYaMJGyY8vUA5uC1GyMVWBfVj0H4gWDJbrzPR5EaztVjZze96pGfr45BKfS6xPVLly1EYTA7nFkai8Sl9SezgXzI/4JWBnFawP+JhVQcfa5ji5Wz+dWZ49nt+AbfGu5xF/ciwiut5zH2u2//WCsZuIEM1NccBCMBox5wMU7xbKpKhlEHoeiUINfCmAcxJKpIKdvGffvCyTGEubAEVobD36J1z9CsJut5w21vzuA5GNGFv2DzWK62J8QaAvP2riDEw7AErWTEHXPAk+i2EYZ9tMLgixQAYwjb2Hkulyq5KeNJ53gZJnIV44Jgl6wjtIihdrS9dMrSq9HPBi2efcGwsdOSgJ83wMQIUwCk1vYE0TL9jXE70yiQNMfx6Cq3YJL3+5TI+lmNJfRDTalyuLk/YRIOf/Rp2Jt7wXabraJEPbgAqQtmIbwNjacJc2XMfnKbSOLpayRoooPRtrTQa+pxM3nc+vN/bKDWmQH3Szx91WHgZTo+yAEH6DN4OHiXFjTe1oDBfIWKEU7FEIOez3RyN7NXlDswSx4tZ27PNFPjdGABwSewYL9gqHItt891MuUEpJVa9eExY2QjPWFl5D6NBVMdaCrvJgsh5kR746/YqqzWleWfXuwLtDLHg3ABfoQkfbqTDlzsEgK4IBGAeQuvTrjlu8aoORj7j1fL1uN5Tf3n0pf/fVacxZzdYWpMo6QUSb36KJ6OQVmSO2Ax1h0mBbiaIgksxg90so8hMZub8uSx3/nmRd81Z8pqUbjNWchk1uN4VAdJdA1Th02pK9TiLDa96wWt8cNhZT2HpBAae5Za1lzHYuDxAAEdWTig3jfoL0VHYoi7uuUUu0iKGKTnfbxaJGgIX4HgWDrilTw5uOeWHTyPTMZO//VUHHmP0OqwD+H4LbrKkBa8QmiMvpTgrLBjHEJ+eSHEHW39w1dnT2jLKD/K+oEB/aWtKXDNfeheJsAl6wYJZCWpI9sWVcB0podS66HRQky4nDpgn/efJS8GOZbxcJENMhiJkFGGJaJGCSxioUv/6PDPS/mk+hltWIMQQvjE77ZAwv6DtvPJFzqHjCmFQsAxNhlQ/md/zU1reb+2Cf92oyIpI9zDf6IcWPY6ekoJembN5Red/9mVEo/CUyt+VFAvcly4hgLYxgeq9wx/sRb1vLHh03mwrvnV5nCoJpZpx5fNhmF935okjzz30HVNHkQ0AxxvjYhHL5JF9jz9oJlRr5bGfIn/o1JL/orAM9YITUfHHkGiaMDSdh/yoQ77//W+OLm7dgF87xlIcJbDZABnVchpTff9yQ36YFqm66mBvU7NXReLJaVs3HxsBQQehQ9wTw8R3AbwBShrpgCNsPeuRdmNaxJxNB5SaVv/uolkI6AEfCQC85bhaaAcWt8wtdqFcHvcBqD2r89oLbmhEvNBgdLkcKwfhDO9l4XGtwxXUdXl49rEUNnb32RXn+PsQX7f4ylkf2TjczD8OcOCpG1z1Qyh+6IO1M8ndWQXeHVXjVxcjsCq+BpHVnYCRtYelM+Yiq1oz2lP4lFEhUnw+Ipt4E9bsug4nrWUSRm7gc0tbgumQ+w/MNA6xR2TEY+ijGj7QUXhvUushBwvW5YnfBHDVViX33CVJkDadTwwpNKfhhJEk2eNNhDHE09qje5BhPw3OuYTipICXx1AtLw4SigXJuy0vHztHCez5cZS4jNlDml5tvGhetFHoAv+XuW5fI8YiSzO8aksxqUZ5PVmKT0w+9FVX/8MlfSzfrbIMNuYALxfazqFL+VrUBi5UsK0a6dhXSHJboI82oZye/81HJ1X1wr1oy15U0yP+7I2Hiik8Cm9wlZaSRXjM2gYSwhP0Myys1iocqPP4KvN2+rJgvX51tHYR4yRzgPFizh3SFDrSWrBbvpqRiw3DPlYPjywL0tls8AFtFhlsxFKoD+QJgDeRamMqpBmUPgaZMQSevr6JpqOSQS0c2Z9/o108SPPMUI+/Ilcec6TAt22r+F0PDdKTEThSeklEvuUb8VUfKyz52HxxWvM/b98iJvzPubPmgRZjVQ5O5IS308fhPMd+QxRAO7iwj5FrtQGLg8WyRy1oEp4Py3sUtyXmfI0bQKD3zJagoxbIZMSBwwBexAnCUwTZRJsLrdQRLG3yBvVPHsg12tEpfUGXiV0sg1gqd+/iTItr1hAcVGysjG1ciReqa/y1D+e8Uqx77OG2vT831BwCO+9FwSq4ZxA+3TvRfxtf0A1sLDIXItM2+ATumQc523iUVSm2fP7Kv4Yprq5D55R8tXnnm4WCSfaN2cIWRaDD3KhT7kVrceLqTJJucFJ17EMqi0nt262MzLIYIwiXPPum/DnORP7p7rpjh6rcFdYf0Wpa9GvVRK+3kiGtm7aWXEL9MzCpd1K+ajWBrJcMTrmqfvbcYZtLAjctG7eMqju9xtbBYfPpwaZ04Xgjz412eV3Cy8ELgEX1xdvVwxV8HLFVhNTwdd7eXh8kYI2+9qKXDQmTRC3fV/9qxYiPsCmt+QQ7CwunuN3G3NVUj1NoDT8HTgvEuvr9+jrsvTexas2JypYIZbERMr6zdwHuYjyiS96yYnmuc5U45NHgVzPZ5sgueSUYLoxoar04yMgvPZQlt/8DgFrqA4a6K53ldMJ36yZsmzaimZH4jLCEWF7akIqrvqCCW6B8cMNTMT5dRsPrWgnmOGHuXymHHNO195qlsViwhrk1UcP3dfI/mSiCT5UWGJP4UuT7xoBiBE6q7BBDqpgorrWZtJzgeHlvRzumK6wsMFCRcG2BBce0eRQtUh4IG9BfBveNGVTPTkAyabNYXgvJ93aia+8RfD3XGnhUqY0S14y+a8kXA3CiyC2k+5Va27GfeqWI6Uk9NHJgJ0V4icqVYHk9C+LxrZv8s9w4X0qSL3qdGQDiCCIN7pYhJ5M5IgR2yRmLdMc4etRUUQI27+paicJMk5K1pYR/at3CsTp6clQARgluCB9lbF+8N3gmedFiCRTgVKboa3KBs5uMKszt4/RXK7JGJiVW3JpTRPQdP6QAMgACMGSX2zsXooh+AlrB+Ys+y5CIlvK8w0XJ4EJkh1KoLYIA5JCT/t+ijeqvtWsP94u3G2eXnOm8jvS5UYepYrxIlZKOZmcg8UyV4FXq9bS5I+DosUeLP9WfP8sN2WmRhdGlLoRkeC6Y58JefT0zCi3KrjpE09o7qBrz9IExyFhQyut1XJsvr+BlwTnlsAnKFe2OgX+9PUGHszjEqUhz+ZkJ9vglI0Hq+p8v7zgJPs2F3j4u1TQEHZIAfOeevCQnafhg7knF3Xsb7er6zzD0//Yq4UL9Q0cHuDEBiU1jNayo3gYdjByFmZ57bAh985ZIxfnJAwNPgQVV6NGw3pBGufFNWTjdShfyCSR/f6p8X9nbaO0ku8UJhNsoIAPVMMGWaIkeldP2vE2U0HmADdEjXvXokc8BJuFTQjHIGSOL6Rpumf2iRS1bkSsuoOdirG8W6j0B/kQkd2sD2wUT/Ut8Qahw5jypt+A/IGd1a7dPgSrKD77h4wop6FTG3PP2jMT7RPmsIBes4W+Qy9zhYD5CIrxpQk1Tw2rPkBTAM7LJLgrqsaiTeDAjnXXZcgOZBSsEkXNMFxfB9stnn6hYWCrdsBxIa98NLlf7aD4KvuEyeXx0pDPMLWzEu1HApD7DwmDMS3g2wnL/5Pw2My5WNidm9uKTahCiCJJkcVM595cgpepj/avP4jv3WsmfOCA89IasYOhUWXshdDgAkEAXN0Al7T25uIvleVqGI2ywTDARJn0JLr1AR3N+ZcFnlBwAm3Z4HprXpwZJ9DZIbMKUrGhlJSYLTuSCvGD5EFEKvSYnzU19LXo1UTWFqplg98tYscqcidozRPA55k13VC4OuVljX2rkvRpMHdrsni9Tm3nN1rxMSU/RxEtekeNkLddScMlTRL4J7nuQI+xaKKKdVNlXz0r6NayC+9fanip1nhEn+2b4L5DkTPaZWPEKXRw0hyRwkC0SoERZvZr4KKrucUWdz3Si/+nH/1WC9MYTP+9A4ZA46nZE167IDmeJI3ij9MS+Xi6LWWKfVqJ4H6TIbINmURgK/RfNKymg3aUYpyUpKL9NKNgu+6OwlhCO99yJa7S4XZuNiBizTPRIb9c4HIsqzNiPOwTSx4iYmBE3eCNQNuhMbBwZtgoadik+jW1iJL/XeZiny0fWRgOcwiqkrdCGz6xgZKG7emENxOeGALDB8oZH79LmPmEqsWpt/lmseQ71IwsD591sodjgXVLHzfGRXOxo82uLv1nYhNd3fjmublxmpGM96WkQ/KhjCFbKObMLtBziCDc86zVuu7i/Vzdet5s+xUoontHzoRunTJMc/zSeLKtYYNymvlFKRDvDcJ0PFrYo2ThXd8+fqYOfv+WUWH5ZyqCF8qITmb5BABStbWnh+v//8aaClw2H4sK/v3lXMFvX7sRyfLl9NKmK+DOrJYya9oKnE2zEl+8E10af906SB7wR4GoZYxCtq3lUisY+c5cx/5HMEr/KCQo84PkLSODw00t/Hcmez1fMd1aB22BON9vLapuavRevbwthDCd9oUPezXdYqpLoPO6fiDQUIZ2kLy7mDBrSuePpDL48sFKzPxV8c3fHE35EG2d0sjw+ng2Wr3ePKVh6q9bSSlge8QipBTBNrjp+sQm12JzIOct1UmDi5LH+Lrak0hhPWrEL4zKCXGoX6T0g8GJ6R1ONqoFMbNJZOZnmgZcLh+0mSPYQnowZSsiOrnXcD4lm0BA1O1U+l1wiOySqUG6IHxZFn5KBvB7rptLnirl3yi17KNIDLMRpxL0xIUmN800qbh3aRg6hfeE1F0K95wuC0Kr/nbmWY16h/35C29Kq9xebhXIU5Wrkrpk9/WPH4x65RC4947lz3Nlzz41mrNWSj4k45uVr5i7uUHJ0LqrFKh6D/GB4LpNvoa6AcNkpRu1G9AfMdWgQ2aHwkkwPAkoa4zCpE29dhy9ZBPRVCd7ViA7CaBiulxN0axq7PEsJHOIy21V5EON8hqUNcdCCJtrAKaJuQAHXjnVPZyKGKEqrwoyPwaS0GpH03yWL7NolPlyxfWVPyOVIKJ9wboHOkqmGZ7PnfOLeZ3XNii75qMbt9/y0L1bvuJp6rbWIxJ79apCfV9F1EodZCStpWKmm0PNTPI3yUFMAtGMOwKr3wavwUASoz5BtbXMxS6aH+NTV3eUyKqElaCXXU9RYp1Mkin+p5bW9x/CVDuUGwEhW4qJcaB6RDjgTctSr+5HWYxfyddYE9v0Tt4m88AFNA3Huui3Kc6NN8bWaXh9wkx5YShDMOlZzzJIjEBslsLk3onVQHxF2gHLlataZXU16YduFjnfYxC7TaUh/j+Y6mKv2bjGuSPlAfgSyymlqkX1nKaPx6Di+ypVKxCcfXRNvnz/UgbdXS2gOSgGTuSc39EQkH/Xuf25BIQwAQRAys3obH044EjY0ZE4MRagA7M/A6TVe4GYp2qkUZ5pJEbaJ8tI4QS32Tw/fU1pkaXywS+xqSADvGhNBSPU89w8c6Q8XSx3HoMK71SKYzqaQ3yBy9kiw/+sZcYe2bSgPpORkU+qjqJC9tpIXKtO0Ly/viw4vpLoWJk5Z5Xnn1eHgejN2c2xeTKGyYbHIwoyLBZ8eTzeg8skqhARBgd50J/6Mga8BO214GHdRFCoq17segesgKBZ9guOf+r+CRtNbi3zqXFqxQGeyd75eP0DcEBe+nFJKo6pETB7iTFUVigc9lTXcr09zBPiE4/e+eWzx2pxGC6dNK/DBa89sX3fpBiNxkFLrcdRTWL3+6CtdtHxgAM+qmu1eEVsQS8GgdSK/vIRdPqJslh2JIITNEpceNNbALdbzURxkbtNfvX0RemKpxsgwhv3W4KolqRMXB0IdAztRwokON0gTLLhzW6pF6II3SMSRrtQcUxtcRYRNXb/5xlxYeLU4aGmzYHYbDdY1HAvo2e67WUiY3dlvNoshw1V4dCcjbNB4bL0+5jHuzPmydWOP17mcf4U904nuatDoFIcRNUWB8shPPdp7Yg3kmkDMzel8BiFBMPcsUy5hBf2hr4IU0lqUGyjtLC/whUr4MQOev/lGvgF/8Pg+CReHlQsIYAAAAASUVORK5CYII=",
+      "text/plain": [
+       "<PIL.Image.Image image mode=RGB size=50x50>"
+      ]
+     },
+     "execution_count": 4,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "resize_images(make_df()).iloc[0].image"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Demo"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "dbconn = DBConn('postgresql://postgres:postgres@localhost/examplesDB', schema='public')\n",
+    "tmp_dir = Path('./data/')"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "test_image_datatables(dbconn, tmp_dir) # restart to rerun "
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 7,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "100%|██████████| 1/1 [00:00<00:00, 16.05it/s]\n"
+     ]
+    }
+   ],
+   "source": [
+    "test_image_pipeline(dbconn, tmp_dir) # restart to rerun "
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Remove files"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 10,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "!rm -rf data/*"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3.10.8 64-bit",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.10.8 (main, Nov  1 2022, 14:18:21) [GCC 12.2.0]"
+  },
+  "orig_nbformat": 4,
+  "vscode": {
+   "interpreter": {
+    "hash": "e7370f93d1d0cde622a1f8e1c04877d8463912d04d973331ad4851f04de6915a"
+   }
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}
diff --git a/examples/many_to_zero/app.py b/examples/many_to_zero/app.py
index 082dc9b5..9a30c72f 100644
--- a/examples/many_to_zero/app.py
+++ b/examples/many_to_zero/app.py
@@ -1,20 +1,17 @@
-from typing import List
+from typing import Dict, List, Optional
 
 import pandas as pd
-
+from datapipe_app import DatapipeApp
 from sqlalchemy import Integer
 from sqlalchemy.sql.schema import Column
 
-from datapipe_app import DatapipeApp
-
+from datapipe.compute import Catalog, Pipeline, Table
+from datapipe.core_steps import BatchGenerate, DatatableTransform
 from datapipe.datatable import DataStore, DataTable
-from datapipe.compute import Pipeline, Catalog, Table, DatatableTransform
-from datapipe.core_steps import BatchGenerate
 from datapipe.run_config import RunConfig
 from datapipe.store.database import DBConn
 from datapipe.store.pandas import TableStoreJsonLine
 
-
 catalog = Catalog(
     {
         "input": Table(
@@ -48,7 +45,8 @@ def count(
     ds: DataStore,
     input_dts: List[DataTable],
     output_dts: List[DataTable],
-    run_config: RunConfig = None,
+    kwargs: Dict,
+    run_config: Optional[RunConfig] = None,
 ) -> None:
     assert len(input_dts) == 1
     assert len(output_dts) == 1
@@ -70,14 +68,15 @@ def count(
             outputs=["input"],
         ),
         DatatableTransform(
-            count,
+            count,  # type: ignore
             inputs=["input"],
             outputs=["result"],
+            check_for_changes=False,
         ),
     ]
 )
 
 
-ds = DataStore(DBConn("sqlite+pysqlite3:///metadata.sqlite"))
+ds = DataStore(DBConn("sqlite+pysqlite3:///db.sqlite"))
 
 app = DatapipeApp(ds, catalog, pipeline)
diff --git a/examples/one_to_many_pipeline/app.py b/examples/one_to_many_pipeline/app.py
index 7a6da5e2..c199edc1 100644
--- a/examples/one_to_many_pipeline/app.py
+++ b/examples/one_to_many_pipeline/app.py
@@ -1,16 +1,12 @@
 import pandas as pd
-
-from sqlalchemy.sql.schema import Column
-from sqlalchemy.sql.sqltypes import Integer, String, Boolean, JSON
-
 from datapipe_app import DatapipeApp
+from sqlalchemy.sql.schema import Column
+from sqlalchemy.sql.sqltypes import JSON, Boolean, Integer, String
 
-from datapipe.store.database import TableStoreDB
-from datapipe.datatable import DataStore
 from datapipe.compute import Catalog, Pipeline, Table
 from datapipe.core_steps import BatchGenerate, BatchTransform
-from datapipe.store.database import DBConn
-
+from datapipe.datatable import DataStore
+from datapipe.store.database import DBConn, TableStoreDB
 
 dbconn = DBConn("sqlite+pysqlite3:///db.sqlite")
 # dbconn = DBConn('postgresql://postgres:password@localhost/postgres', schema='test')
@@ -18,7 +14,7 @@
 
 
 def generate_products():
-    for pipeline_id in range(1, 3):
+    for pipeline_id in range(1, 4):
         df_data = []
 
         for offer_id in range(1, 6):
@@ -109,7 +105,6 @@ def gen_product_all(df1: pd.DataFrame, df2: pd.DataFrame) -> pd.DataFrame:
 
 
 def gen_product_store(df1: pd.DataFrame, df2: pd.DataFrame) -> pd.DataFrame:
-
     keys = ["pipeline_id", "offer_id"]
     merged_df = pd.merge(df1, df2, how="outer", left_on=keys, right_on=keys)
 
diff --git a/examples/one_to_many_pipeline/research_database.ipynb b/examples/one_to_many_pipeline/research_database.ipynb
new file mode 100644
index 00000000..2ab53905
--- /dev/null
+++ b/examples/one_to_many_pipeline/research_database.ipynb
@@ -0,0 +1,180 @@
+{
+ "cells": [
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from sqlalchemy import create_engine\n",
+    "import pandas as pd"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "engine = create_engine(\"sqlite+pysqlite3:///db.sqlite\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 8,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "df = pd.read_sql_query(\"SELECT * from test_products_data\", con=engine)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 9,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/html": [
+       "<div>\n",
+       "<style scoped>\n",
+       "    .dataframe tbody tr th:only-of-type {\n",
+       "        vertical-align: middle;\n",
+       "    }\n",
+       "\n",
+       "    .dataframe tbody tr th {\n",
+       "        vertical-align: top;\n",
+       "    }\n",
+       "\n",
+       "    .dataframe thead th {\n",
+       "        text-align: right;\n",
+       "    }\n",
+       "</style>\n",
+       "<table border=\"1\" class=\"dataframe\">\n",
+       "  <thead>\n",
+       "    <tr style=\"text-align: right;\">\n",
+       "      <th></th>\n",
+       "      <th>pipeline_id</th>\n",
+       "      <th>offer_id</th>\n",
+       "      <th>attributes</th>\n",
+       "    </tr>\n",
+       "  </thead>\n",
+       "  <tbody>\n",
+       "    <tr>\n",
+       "      <th>0</th>\n",
+       "      <td>1</td>\n",
+       "      <td>1</td>\n",
+       "      <td>{\"attr_1\": 1}</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>1</th>\n",
+       "      <td>1</td>\n",
+       "      <td>2</td>\n",
+       "      <td>{\"attr_1\": 1, \"attr_2\": 2}</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>2</th>\n",
+       "      <td>1</td>\n",
+       "      <td>3</td>\n",
+       "      <td>{\"attr_1\": 1, \"attr_2\": 2, \"attr_3\": 3}</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>3</th>\n",
+       "      <td>1</td>\n",
+       "      <td>4</td>\n",
+       "      <td>{\"attr_1\": 1, \"attr_2\": 2, \"attr_3\": 3, \"attr_...</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>4</th>\n",
+       "      <td>1</td>\n",
+       "      <td>5</td>\n",
+       "      <td>{\"attr_1\": 1, \"attr_2\": 2, \"attr_3\": 3, \"attr_...</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>5</th>\n",
+       "      <td>2</td>\n",
+       "      <td>1</td>\n",
+       "      <td>{\"attr_1\": 1}</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>6</th>\n",
+       "      <td>2</td>\n",
+       "      <td>2</td>\n",
+       "      <td>{\"attr_1\": 1, \"attr_2\": 2}</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>7</th>\n",
+       "      <td>2</td>\n",
+       "      <td>3</td>\n",
+       "      <td>{\"attr_1\": 1, \"attr_2\": 2, \"attr_3\": 3}</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>8</th>\n",
+       "      <td>2</td>\n",
+       "      <td>4</td>\n",
+       "      <td>{\"attr_1\": 1, \"attr_2\": 2, \"attr_3\": 3, \"attr_...</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>9</th>\n",
+       "      <td>2</td>\n",
+       "      <td>5</td>\n",
+       "      <td>{\"attr_1\": 1, \"attr_2\": 2, \"attr_3\": 3, \"attr_...</td>\n",
+       "    </tr>\n",
+       "  </tbody>\n",
+       "</table>\n",
+       "</div>"
+      ],
+      "text/plain": [
+       "   pipeline_id  offer_id                                         attributes\n",
+       "0            1         1                                      {\"attr_1\": 1}\n",
+       "1            1         2                         {\"attr_1\": 1, \"attr_2\": 2}\n",
+       "2            1         3            {\"attr_1\": 1, \"attr_2\": 2, \"attr_3\": 3}\n",
+       "3            1         4  {\"attr_1\": 1, \"attr_2\": 2, \"attr_3\": 3, \"attr_...\n",
+       "4            1         5  {\"attr_1\": 1, \"attr_2\": 2, \"attr_3\": 3, \"attr_...\n",
+       "5            2         1                                      {\"attr_1\": 1}\n",
+       "6            2         2                         {\"attr_1\": 1, \"attr_2\": 2}\n",
+       "7            2         3            {\"attr_1\": 1, \"attr_2\": 2, \"attr_3\": 3}\n",
+       "8            2         4  {\"attr_1\": 1, \"attr_2\": 2, \"attr_3\": 3, \"attr_...\n",
+       "9            2         5  {\"attr_1\": 1, \"attr_2\": 2, \"attr_3\": 3, \"attr_..."
+      ]
+     },
+     "execution_count": 9,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "df"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "base",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.9.12"
+  },
+  "orig_nbformat": 4
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}
diff --git a/pyproject.toml b/pyproject.toml
index cc2f84c2..45f5300a 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -1,6 +1,6 @@
 [tool.poetry]
 name = "datapipe-core"
-version = "0.11.12-dev.1"
+version = "0.12.0-dev.4"
 description = ""
 authors = ["Andrey Tatarinov <a@tatarinov.co>"]
 packages = [
@@ -39,6 +39,13 @@ redis = {version="^4.3.4", optional=true}
 pysqlite3-binary = {version="^0.5.0", optional=true}
 sqlalchemy-pysqlite3-binary = {version="^0.0.4", optional=true}
 
+click = ">=7.1.2"
+rich = "^13.3.2"
+
+# Copypaste without thinking from https://github.com/meltano/sdk/blob/main/pyproject.toml
+sphinx = {version = ">=4.5,<6.0", optional = true}
+myst-parser = {version = ">=0.17.2,<1.1.0", optional = true}
+
 [tool.poetry.extras]
 
 sqlite = ["pysqlite3-binary", "sqlalchemy-pysqlite3-binary"]
@@ -48,6 +55,8 @@ gcsfs = ["gcsfs"]
 s3fs = ["s3fs"]
 redis = ["redis"]
 
+docs = ["sphinx", "myst-parser"]
+
 [tool.poetry.group.dev.dependencies]
 pytest = "*"
 flake8 = "*"
@@ -57,7 +66,12 @@ pandas-stubs = "*"
 pytest-cases = "*"
 types-setuptools = "*"
 types-redis = "*"
-datapipe-app = "^0.2.1.post1"
+types-tqdm = "*"
+types-pillow = "*"
+types-PyYAML = "*"
+
+[tool.poetry.scripts]
+datapipe = 'datapipe.cli:main'
 
 [build-system]
 requires = ["poetry-core>=1.0.0"]


Test patch:
diff --git a/.github/workflows/pytest.yaml b/.github/workflows/pytest.yaml
index 229e11d0..dcf165c5 100644
--- a/.github/workflows/pytest.yaml
+++ b/.github/workflows/pytest.yaml
@@ -13,8 +13,8 @@ jobs:
       matrix:
         python-version:
           # - "3.8"
-          - "3.9"
-          # - "3.10"
+          # - "3.9"
+          - "3.10"
 
     steps:
       - uses: actions/checkout@v3
@@ -58,7 +58,7 @@ jobs:
     strategy:
       matrix:
         python-version:
-          # - "3.8"
+          - "3.8"
           - "3.9"
           - "3.10"
         test-db-env:
diff --git a/tests/docker-compose.yml b/tests/docker-compose.yml
index 8a48ea11..bf3f310e 100644
--- a/tests/docker-compose.yml
+++ b/tests/docker-compose.yml
@@ -9,7 +9,7 @@ services:
       POSTGRES_PASSWORD: password
 
   label-studio:
-    image: heartexlabs/label-studio:1.0.1
+    image: ghcr.io/epoch8/label-studio/label-studio:1.5.0post.0rq1
     ports:
         - 8080:8080
 
diff --git a/tests/test_chunked_processing_pipeline.py b/tests/test_chunked_processing_pipeline.py
index 85736fa2..55358b8b 100644
--- a/tests/test_chunked_processing_pipeline.py
+++ b/tests/test_chunked_processing_pipeline.py
@@ -1,4 +1,6 @@
 import os
+from typing import List, Optional
+from datapipe.run_config import RunConfig
 
 import pytest
 import numpy as np
@@ -6,10 +8,9 @@
 from sqlalchemy import Column
 from sqlalchemy.sql.sqltypes import Integer
 
-from datapipe.compute import (Catalog, Pipeline, Table, build_compute,
-                              run_changelist, run_steps)
-from datapipe.core_steps import BatchTransform, UpdateExternalTable
-from datapipe.datatable import DataStore
+from datapipe.compute import Catalog, Pipeline, Table, build_compute, run_changelist, run_steps
+from datapipe.core_steps import BatchTransform, DatatableTransform, UpdateExternalTable
+from datapipe.datatable import DataStore, DataTable
 from datapipe.store.database import TableStoreDB
 from datapipe.store.pandas import TableStoreJsonLine
 from datapipe.types import ChangeList, data_to_index
@@ -20,14 +21,14 @@
 CHUNK_SIZE_SMALL = 3
 
 TEST_SCHEMA = [
-    Column('id', Integer, primary_key=True),
-    Column('a', Integer),
+    Column("id", Integer, primary_key=True),
+    Column("a", Integer),
 ]
 
 TEST_DF = pd.DataFrame(
     {
-        'id': range(10),
-        'a': range(10),
+        "id": range(10),
+        "a": range(10),
     },
 )
 
@@ -38,42 +39,46 @@ def conversion(df, multiply):
         return df
 
     x = pd.Series(np.arange(2 * CHUNK_SIZE, dtype=np.int32))
-    test_df = pd.DataFrame({
-        "id": x.apply(str),
-        "x": x,
-    })
+    test_df = pd.DataFrame(
+        {
+            "id": x.apply(str),
+            "x": x,
+        }
+    )
     test_input_fname = os.path.join(tmp_dir, "table-input-pandas.json")
     test_output_fname = os.path.join(tmp_dir, "table-output-pandas.json")
     test_df.to_json(test_input_fname, orient="records", lines=True)
 
     ds = DataStore(dbconn, create_meta_table=True)
-    catalog = Catalog({
-        "input_data": Table(
-            store=TableStoreJsonLine(test_input_fname),
-        ),
-        "output_data": Table(
-            store=TableStoreJsonLine(test_output_fname),
-        ),
-    })
-    pipeline = Pipeline([
-        UpdateExternalTable(
-            "input_data"
-        ),
-        BatchTransform(
-            conversion,
-            inputs=["input_data"],
-            outputs=["output_data"],
-            chunk_size=CHUNK_SIZE,
-            kwargs=dict(
-                multiply=2,
+    catalog = Catalog(
+        {
+            "input_data": Table(
+                store=TableStoreJsonLine(test_input_fname),
             ),
-        ),
-    ])
+            "output_data": Table(
+                store=TableStoreJsonLine(test_output_fname),
+            ),
+        }
+    )
+    pipeline = Pipeline(
+        [
+            UpdateExternalTable("input_data"),
+            BatchTransform(
+                conversion,
+                inputs=["input_data"],
+                outputs=["output_data"],
+                chunk_size=CHUNK_SIZE,
+                kwargs=dict(
+                    multiply=2,
+                ),
+            ),
+        ]
+    )
 
     steps = build_compute(ds, catalog, pipeline)
     run_steps(ds, steps)
 
-    df_transformed = catalog.get_datatable(ds, 'output_data').get_data()
+    df_transformed = catalog.get_datatable(ds, "output_data").get_data()
     assert len(df_transformed) == 2 * CHUNK_SIZE
     assert all(df_transformed["y"].values == (df_transformed["x"].values ** 2))
     assert len(set(df_transformed["x"].values).symmetric_difference(set(x))) == 0
@@ -81,61 +86,60 @@ def conversion(df, multiply):
 
 def test_transform_with_many_input_and_output_tables(tmp_dir, dbconn):
     ds = DataStore(dbconn, create_meta_table=True)
-    catalog = Catalog({
-        "inp1": Table(
-            store=TableStoreDB(
-                dbconn,
-                'inp1_data',
-                TEST_SCHEMA,
-                create_table=True,
-            )
-        ),
-        "inp2": Table(
-            store=TableStoreDB(
-                dbconn,
-                'inp2_data',
-                TEST_SCHEMA,
-                create_table=True,
-            )
-        ),
-        "out1": Table(
-            store=TableStoreDB(
-                dbconn,
-                'out1_data',
-                TEST_SCHEMA,
-                create_table=True,
-            )
-        ),
-        "out2": Table(
-            store=TableStoreDB(
-                dbconn,
-                'out2_data',
-                TEST_SCHEMA,
-                create_table=True,
-            )
-        ),
-    })
+    catalog = Catalog(
+        {
+            "inp1": Table(
+                store=TableStoreDB(
+                    dbconn,
+                    "inp1_data",
+                    TEST_SCHEMA,
+                    create_table=True,
+                )
+            ),
+            "inp2": Table(
+                store=TableStoreDB(
+                    dbconn,
+                    "inp2_data",
+                    TEST_SCHEMA,
+                    create_table=True,
+                )
+            ),
+            "out1": Table(
+                store=TableStoreDB(
+                    dbconn,
+                    "out1_data",
+                    TEST_SCHEMA,
+                    create_table=True,
+                )
+            ),
+            "out2": Table(
+                store=TableStoreDB(
+                    dbconn,
+                    "out2_data",
+                    TEST_SCHEMA,
+                    create_table=True,
+                )
+            ),
+        }
+    )
 
     def transform(df1, df2):
         return df1, df2
 
-    pipeline = Pipeline([
-        BatchTransform(
-            transform,
-            inputs=["inp1", "inp2"],
-            outputs=["out1", "out2"],
-            chunk_size=CHUNK_SIZE
-        ),
-    ])
+    pipeline = Pipeline(
+        [
+            BatchTransform(transform, inputs=["inp1", "inp2"], outputs=["out1", "out2"], chunk_size=CHUNK_SIZE),
+        ]
+    )
 
-    catalog.get_datatable(ds, 'inp1').store_chunk(TEST_DF)
-    catalog.get_datatable(ds, 'inp2').store_chunk(TEST_DF)
+    catalog.get_datatable(ds, "inp1").store_chunk(TEST_DF)
+    catalog.get_datatable(ds, "inp2").store_chunk(TEST_DF)
 
     steps = build_compute(ds, catalog, pipeline)
     run_steps(ds, steps)
 
-    out1 = catalog.get_datatable(ds, 'out1')
-    out2 = catalog.get_datatable(ds, 'out2')
+    out1 = catalog.get_datatable(ds, "out1")
+    out2 = catalog.get_datatable(ds, "out2")
 
     assert_datatable_equal(out1, TEST_DF)
     assert_datatable_equal(out2, TEST_DF)
@@ -143,66 +147,74 @@ def transform(df1, df2):
 
 def test_run_changelist_simple(dbconn):
     ds = DataStore(dbconn, create_meta_table=True)
-    catalog = Catalog({
-        "inp": Table(
-            store=TableStoreDB(
-                dbconn,
-                'inp_data',
-                TEST_SCHEMA,
-                create_table=True,
-            )
-        ),
-        "out": Table(
-            store=TableStoreDB(
-                dbconn,
-                'out_data',
-                TEST_SCHEMA,
-                create_table=True,
-            )
-        )
-    })
+    catalog = Catalog(
+        {
+            "inp": Table(
+                store=TableStoreDB(
+                    dbconn,
+                    "inp_data",
+                    TEST_SCHEMA,
+                    create_table=True,
+                )
+            ),
+            "out": Table(
+                store=TableStoreDB(
+                    dbconn,
+                    "out_data",
+                    TEST_SCHEMA,
+                    create_table=True,
+                )
+            ),
+        }
+    )
 
     def transform(df):
         return df
 
-    pipeline = Pipeline([
-        BatchTransform(
-            transform,
-            inputs=["inp"],
-            outputs=["out"],
-            chunk_size=CHUNK_SIZE,
-        ),
-    ])
+    pipeline = Pipeline(
+        [
+            BatchTransform(
+                transform,
+                inputs=["inp"],
+                outputs=["out"],
+                chunk_size=CHUNK_SIZE,
+            ),
+        ]
+    )
 
-    changeIdx = data_to_index(TEST_DF.loc[[2, 3, 4]], ['id'])
-    changelist = ChangeList.create('inp', changeIdx)
+    changeIdx = data_to_index(TEST_DF.loc[[2, 3, 4]], ["id"])
+    changelist = ChangeList.create("inp", changeIdx)
 
-    catalog.get_datatable(ds, 'inp').store_chunk(TEST_DF, now=0)
+    catalog.get_datatable(ds, "inp").store_chunk(TEST_DF, now=0)
 
     run_changelist(ds, catalog, pipeline, changelist)
 
-    assert_datatable_equal(catalog.get_datatable(ds, 'out'), TEST_DF.loc[changeIdx.index])
+    assert_datatable_equal(catalog.get_datatable(ds, "out"), TEST_DF.loc[changeIdx.index])
 
 
 def test_run_changelist_with_duplicate_input_keys(dbconn):
     ds = DataStore(dbconn, create_meta_table=True)
-    catalog = Catalog({
-        "inp": Table(
-            store=TableStoreDB(
-                dbconn,
-                'inp_data',
-                TEST_SCHEMA,
-                create_table=True,
-            )
-        ),
-    })
+    catalog = Catalog(
+        {
+            "inp": Table(
+                store=TableStoreDB(
+                    dbconn,
+                    "inp_data",
+                    TEST_SCHEMA,
+                    create_table=True,
+                )
+            ),
+        }
+    )
 
-    test_df_with_duplicates = pd.DataFrame({
-        "id": [1, 1],
-        "a": [1, 2],
-    })
+    test_df_with_duplicates = pd.DataFrame(
+        {
+            "id": [1, 1],
+            "a": [1, 2],
+        }
+    )
 
-    dt = catalog.get_datatable(ds, 'inp')
+    dt = catalog.get_datatable(ds, "inp")
 
     dt.store_chunk(TEST_DF)
 
@@ -214,24 +226,26 @@ def test_run_changelist_with_duplicate_input_keys(dbconn):
 
 def test_run_changelist_by_chunk_size_simple(dbconn):
     ds = DataStore(dbconn, create_meta_table=True)
-    catalog = Catalog({
-        "inp": Table(
-            store=TableStoreDB(
-                dbconn,
-                'inp_data',
-                TEST_SCHEMA,
-                create_table=True,
-            )
-        ),
-        "out": Table(
-            store=TableStoreDB(
-                dbconn,
-                'out_data',
-                TEST_SCHEMA,
-                create_table=True,
-            )
-        )
-    })
+    catalog = Catalog(
+        {
+            "inp": Table(
+                store=TableStoreDB(
+                    dbconn,
+                    "inp_data",
+                    TEST_SCHEMA,
+                    create_table=True,
+                )
+            ),
+            "out": Table(
+                store=TableStoreDB(
+                    dbconn,
+                    "out_data",
+                    TEST_SCHEMA,
+                    create_table=True,
+                )
+            ),
+        }
+    )
 
     def transform(df):
         if len(df) > CHUNK_SIZE_SMALL:
@@ -239,78 +253,136 @@ def transform(df):
 
         return df
 
-    pipeline = Pipeline([
-        BatchTransform(
-            transform,
-            inputs=["inp"],
-            outputs=["out"],
-            chunk_size=CHUNK_SIZE_SMALL,
-        ),
-    ])
+    pipeline = Pipeline(
+        [
+            BatchTransform(
+                transform,
+                inputs=["inp"],
+                outputs=["out"],
+                chunk_size=CHUNK_SIZE_SMALL,
+            ),
+        ]
+    )
 
-    changeIdx = data_to_index(TEST_DF.loc[[2, 3, 4, 5, 6]], ['id'])
-    changelist = ChangeList.create('inp', changeIdx)
+    changeIdx = data_to_index(TEST_DF.loc[[2, 3, 4, 5, 6]], ["id"])
+    changelist = ChangeList.create("inp", changeIdx)
 
-    catalog.get_datatable(ds, 'inp').store_chunk(TEST_DF, now=0)
+    catalog.get_datatable(ds, "inp").store_chunk(TEST_DF, now=0)
 
     run_changelist(ds, catalog, pipeline, changelist)
 
-    assert_datatable_equal(catalog.get_datatable(ds, 'out'), TEST_DF.loc[changeIdx.index])
+    assert_datatable_equal(catalog.get_datatable(ds, "out"), TEST_DF.loc[changeIdx.index])
 
 
 def test_run_changelist_cycle(dbconn):
     ds = DataStore(dbconn, create_meta_table=True)
-    catalog = Catalog({
-        "a": Table(
-            store=TableStoreDB(
-                dbconn,
-                'a_data',
-                TEST_SCHEMA,
-                create_table=True,
-            )
-        ),
-        "b": Table(
-            store=TableStoreDB(
-                dbconn,
-                'b_data',
-                TEST_SCHEMA,
-                create_table=True,
-            )
-        ),
-    })
+    catalog = Catalog(
+        {
+            "a": Table(
+                store=TableStoreDB(
+                    dbconn,
+                    "a_data",
+                    TEST_SCHEMA,
+                    create_table=True,
+                )
+            ),
+            "b": Table(
+                store=TableStoreDB(
+                    dbconn,
+                    "b_data",
+                    TEST_SCHEMA,
+                    create_table=True,
+                )
+            ),
+        }
+    )
 
     def inc(df):
-        return df.assign(a=df['a'] + 1)
+        return df.assign(a=df["a"] + 1)
 
     def cap_10(df):
-        return df.assign(a=df['a'].clip(0, 10))
-
-    pipeline = Pipeline([
-        BatchTransform(
-            inc,
-            inputs=["a"],
-            outputs=["b"],
-            chunk_size=CHUNK_SIZE,
-        ),
-        BatchTransform(
-            cap_10,
-            inputs=["b"],
-            outputs=["a"],
-            chunk_size=CHUNK_SIZE,
-        ),
-    ])
+        return df.assign(a=df["a"].clip(0, 10))
+
+    pipeline = Pipeline(
+        [
+            BatchTransform(
+                inc,
+                inputs=["a"],
+                outputs=["b"],
+                chunk_size=CHUNK_SIZE,
+            ),
+            BatchTransform(
+                cap_10,
+                inputs=["b"],
+                outputs=["a"],
+                chunk_size=CHUNK_SIZE,
+            ),
+        ]
+    )
 
-    changeIdx = data_to_index(TEST_DF.loc[[2, 3, 4]], ['id'])
-    changelist = ChangeList.create('a', changeIdx)
+    changeIdx = data_to_index(TEST_DF.loc[[2, 3, 4]], ["id"])
+    changelist = ChangeList.create("a", changeIdx)
 
-    catalog.get_datatable(ds, 'a').store_chunk(TEST_DF, now=0)
+    catalog.get_datatable(ds, "a").store_chunk(TEST_DF, now=0)
 
     run_changelist(ds, catalog, pipeline, changelist)
 
     assert_df_equal(
-        catalog.get_datatable(ds, 'a').get_data(),
-        pd.DataFrame({
-            "id": range(10),
-            "a": [0, 1, 10, 10, 10, 5, 6, 7, 8, 9],
-        }),
+        catalog.get_datatable(ds, "a").get_data(),
+        pd.DataFrame(
+            {
+                "id": range(10),
+                "a": [0, 1, 10, 10, 10, 5, 6, 7, 8, 9],
+            }
+        ),
     )
+
+
+def test_run_changelist_with_datatable_transform(dbconn):
+    ds = DataStore(dbconn, create_meta_table=True)
+    catalog = Catalog(
+        {
+            "inp": Table(
+                store=TableStoreDB(
+                    dbconn,
+                    "inp_data",
+                    TEST_SCHEMA,
+                    create_table=True,
+                )
+            ),
+            "out": Table(
+                store=TableStoreDB(
+                    dbconn,
+                    "out_data",
+                    TEST_SCHEMA,
+                    create_table=True,
+                )
+            ),
+        }
+    )
+
+    def transform(
+        ds: DataStore,
+        input_dts: List[DataTable],
+        output_dts: List[DataTable],
+        run_config: Optional[RunConfig] = None,
+    ):
+        input_dt = input_dts[0]
+        output_dt = output_dts[0]
+        df = input_dt.get_data()
+        output_dt.store_chunk(df)
+
+    pipeline = Pipeline(
+        [
+            DatatableTransform(
+                transform,
+                inputs=["inp"],
+                outputs=["out"],
+            ),
+        ]
+    )
+
+    changeIdx = data_to_index(TEST_DF.loc[[2, 3, 4, 5, 6]], ["id"])
+    changelist = ChangeList.create("inp", changeIdx)
+    catalog.get_datatable(ds, "inp").store_chunk(TEST_DF, now=0)
+    run_changelist(ds, catalog, pipeline, changelist)
diff --git a/tests/test_core_steps1.py b/tests/test_core_steps1.py
index f51c4bdb..9e5955f3 100644
--- a/tests/test_core_steps1.py
+++ b/tests/test_core_steps1.py
@@ -1,65 +1,64 @@
 # Ex-test_datatable
 
-from typing import cast
-import pytest
 from functools import partial
+from typing import cast
 
 import pandas as pd
+import pytest
 from sqlalchemy import Column
-from sqlalchemy.sql.sqltypes import Integer, JSON
+from sqlalchemy.sql.sqltypes import JSON, Integer
 
-from datapipe.store.database import TableStoreDB
+from datapipe.core_steps import BatchTransformStep, do_batch_generate
 from datapipe.datatable import DataStore
-from datapipe.core_steps import do_batch_generate, do_full_batch_transform
+from datapipe.store.database import TableStoreDB
 from datapipe.types import IndexDF, data_to_index
 
-from .util import assert_df_equal, assert_datatable_equal
-
+from .util import assert_datatable_equal, assert_df_equal
 
 TEST_SCHEMA = [
-    Column('id', Integer, primary_key=True),
-    Column('a', Integer),
+    Column("id", Integer, primary_key=True),
+    Column("a", Integer),
 ]
 
 TEST_SCHEMA_OTM = [
-    Column('id', Integer, primary_key=True),
-    Column('a', JSON),
+    Column("id", Integer, primary_key=True),
+    Column("a", JSON),
 ]
 
 TEST_SCHEMA_OTM2 = [
-    Column('id', Integer, primary_key=True),
-    Column('a', Integer, primary_key=True)
+    Column("id", Integer, primary_key=True),
+    Column("a", Integer, primary_key=True),
 ]
 
 TEST_SCHEMA_OTM3 = [
-    Column('a', Integer, primary_key=True),
-    Column('b', Integer, primary_key=True),
-    Column('ids', JSON)
+    Column("a", Integer, primary_key=True),
+    Column("b", Integer, primary_key=True),
+    Column("ids", JSON),
 ]
 
 TEST_DF = pd.DataFrame(
     {
-        'id': range(10),
-        'a': range(10),
+        "id": range(10),
+        "a": range(10),
     },
 )
 
 TEST_OTM_DF = pd.DataFrame(
     {
-        'id': range(10),
-        'a': [[j for j in range(i)] for i in range(10)],
+        "id": range(10),
+        "a": [[j for j in range(i)] for i in range(10)],
     },
 )
 
 
-TEST_DF_INC1 = TEST_DF.assign(a=lambda df: df['a'] + 1)
-TEST_DF_INC2 = TEST_DF.assign(a=lambda df: df['a'] + 2)
-TEST_DF_INC3 = TEST_DF.assign(a=lambda df: df['a'] + 3)
+TEST_DF_INC1 = TEST_DF.assign(a=lambda df: df["a"] + 1)
+TEST_DF_INC2 = TEST_DF.assign(a=lambda df: df["a"] + 2)
+TEST_DF_INC3 = TEST_DF.assign(a=lambda df: df["a"] + 3)
 
 
 def yield_df(data):
     def f(*args, **kwargs):
-        yield pd.DataFrame.from_records(data, columns=['id', 'a']).set_index('id')
+        yield pd.DataFrame.from_records(data, columns=["id", "a"]).set_index("id")
 
     return f
 
@@ -68,18 +67,13 @@ def test_gen_process(dbconn) -> None:
     ds = DataStore(dbconn, create_meta_table=True)
 
     tbl1 = ds.create_table(
-        'tbl1',
-        table_store=TableStoreDB(dbconn, 'tbl1_data', TEST_SCHEMA, True)
+        "tbl1", table_store=TableStoreDB(dbconn, "tbl1_data", TEST_SCHEMA, True)
     )
 
     def gen():
         yield TEST_DF
 
-    do_batch_generate(
-        func=gen,
-        ds=ds,
-        output_dts=[tbl1]
-    )
+    do_batch_generate(func=gen, ds=ds, output_dts=[tbl1])
 
     assert_datatable_equal(tbl1, TEST_DF)
 
@@ -87,23 +81,17 @@ def func():
         return TEST_DF
 
     with pytest.raises(Exception):
-        do_batch_generate(
-            func=func,
-            ds=ds,
-            output_dts=[tbl1]
-        )
+        do_batch_generate(func=func, ds=ds, output_dts=[tbl1])
 
 
 def test_inc_process_modify_values(dbconn) -> None:
     ds = DataStore(dbconn, create_meta_table=True)
 
     tbl1 = ds.create_table(
-        'tbl1',
-        table_store=TableStoreDB(dbconn, 'tbl1_data', TEST_SCHEMA, True)
+        "tbl1", table_store=TableStoreDB(dbconn, "tbl1_data", TEST_SCHEMA, True)
     )
     tbl2 = ds.create_table(
-        'tbl2',
-        table_store=TableStoreDB(dbconn, 'tbl2_data', TEST_SCHEMA, True)
+        "tbl2", table_store=TableStoreDB(dbconn, "tbl2_data", TEST_SCHEMA, True)
     )
 
     def id_func(df):
@@ -111,24 +99,22 @@ def id_func(df):
 
     tbl1.store_chunk(TEST_DF)
 
-    do_full_batch_transform(
-        func=id_func,
+    step = BatchTransformStep(
         ds=ds,
+        name="test",
+        func=id_func,
         input_dts=[tbl1],
         output_dts=[tbl2],
     )
 
+    step.run_full(ds)
+
     assert_datatable_equal(tbl2, TEST_DF)
 
     ##########################
     tbl1.store_chunk(TEST_DF_INC1)
 
-    do_full_batch_transform(
-        func=id_func,
-        ds=ds,
-        input_dts=[tbl1],
-        output_dts=[tbl2],
-    )
+    step.run_full(ds)
 
     assert_datatable_equal(tbl2, TEST_DF_INC1)
 
@@ -137,12 +123,10 @@ def test_inc_process_delete_values_from_input(dbconn) -> None:
     ds = DataStore(dbconn, create_meta_table=True)
 
     tbl1 = ds.create_table(
-        'tbl1',
-        table_store=TableStoreDB(dbconn, 'tbl1_data', TEST_SCHEMA, True)
+        "tbl1", table_store=TableStoreDB(dbconn, "tbl1_data", TEST_SCHEMA, True)
     )
     tbl2 = ds.create_table(
-        'tbl2',
-        table_store=TableStoreDB(dbconn, 'tbl2_data', TEST_SCHEMA, True)
+        "tbl2", table_store=TableStoreDB(dbconn, "tbl2_data", TEST_SCHEMA, True)
     )
 
     def id_func(df):
@@ -150,27 +134,26 @@ def id_func(df):
 
     tbl1.store_chunk(TEST_DF)
 
-    do_full_batch_transform(
-        func=id_func,
+    step = BatchTransformStep(
         ds=ds,
+        name="test",
+        func=id_func,
         input_dts=[tbl1],
         output_dts=[tbl2],
     )
 
+    step.run_full(ds)
+
     assert_datatable_equal(tbl2, TEST_DF)
 
     ##########################
-    tbl1.store_chunk(TEST_DF[:5], processed_idx=data_to_index(TEST_DF, tbl1.primary_keys))
+    tbl1.store_chunk(
+        TEST_DF[:5], processed_idx=data_to_index(TEST_DF, tbl1.primary_keys)
+    )
 
     assert_datatable_equal(tbl1, TEST_DF[:5])
 
-    do_full_batch_transform(
-        func=id_func,
-        ds=ds,
-        input_dts=[tbl1],
-        output_dts=[tbl2],
-        chunk_size=2,
-    )
+    step.run_full(ds)
 
     assert_datatable_equal(tbl2, TEST_DF[:5])
 
@@ -179,12 +162,10 @@ def test_inc_process_delete_values_from_proc(dbconn) -> None:
     ds = DataStore(dbconn, create_meta_table=True)
 
     tbl1 = ds.create_table(
-        'tbl1',
-        table_store=TableStoreDB(dbconn, 'tbl1_data', TEST_SCHEMA, True)
+        "tbl1", table_store=TableStoreDB(dbconn, "tbl1_data", TEST_SCHEMA, True)
     )
     tbl2 = ds.create_table(
-        'tbl2',
-        table_store=TableStoreDB(dbconn, 'tbl2_data', TEST_SCHEMA, True)
+        "tbl2", table_store=TableStoreDB(dbconn, "tbl2_data", TEST_SCHEMA, True)
     )
 
     def id_func(df):
@@ -194,13 +175,16 @@ def id_func(df):
 
     tbl1.store_chunk(TEST_DF)
 
-    do_full_batch_transform(
-        func=id_func,
+    step = BatchTransformStep(
         ds=ds,
+        name="test",
+        func=id_func,
         input_dts=[tbl1],
         output_dts=[tbl2],
     )
 
+    step.run_full(ds)
+
     assert_datatable_equal(tbl2, TEST_DF[:5])
 
 
@@ -208,12 +192,10 @@ def test_inc_process_proc_no_change(dbconn) -> None:
     ds = DataStore(dbconn, create_meta_table=True)
 
     tbl1 = ds.create_table(
-        'tbl1',
-        table_store=TableStoreDB(dbconn, 'tbl1_data', TEST_SCHEMA, True)
+        "tbl1", table_store=TableStoreDB(dbconn, "tbl1_data", TEST_SCHEMA, True)
     )
     tbl2 = ds.create_table(
-        'tbl2',
-        table_store=TableStoreDB(dbconn, 'tbl2_data', TEST_SCHEMA, True)
+        "tbl2", table_store=TableStoreDB(dbconn, "tbl2_data", TEST_SCHEMA, True)
     )
 
     def id_func(df):
@@ -222,20 +204,23 @@ def id_func(df):
     tbl2.store_chunk(TEST_DF)
     tbl1.store_chunk(TEST_DF)
 
-    count, idx_gen = ds.get_full_process_ids([tbl1], [tbl2])
+    step = BatchTransformStep(
+        ds=ds,
+        name="step",
+        func=id_func,
+        input_dts=[tbl1],
+        output_dts=[tbl2],
+    )
+
+    count, idx_gen = step.get_full_process_ids(ds)
     idx_dfs = list(idx_gen)
     idx_len = len(pd.concat(idx_dfs)) if len(idx_dfs) > 0 else 0
 
     assert idx_len == len(TEST_DF)
 
-    do_full_batch_transform(
-        func=id_func,
-        ds=ds,
-        input_dts=[tbl1],
-        output_dts=[tbl2],
-    )
+    step.run_full(ds)
 
-    count, idx_gen = ds.get_full_process_ids([tbl1], [tbl2])
+    count, idx_gen = step.get_full_process_ids(ds)
     idx_dfs = list(idx_gen)
     idx_len = len(pd.concat(idx_dfs)) if len(idx_dfs) > 0 else 0
 
@@ -243,25 +228,21 @@ def id_func(df):
 
     tbl1.store_chunk(TEST_DF_INC1)
 
-    count, idx_gen = ds.get_full_process_ids([tbl1], [tbl2])
+    count, idx_gen = step.get_full_process_ids(ds)
     idx_dfs = list(idx_gen)
     idx_len = len(pd.concat(idx_dfs)) if len(idx_dfs) > 0 else 0
 
     assert idx_len == len(TEST_DF)
 
-    do_full_batch_transform(
-        func=id_func,
-        ds=ds,
-        input_dts=[tbl1],
-        output_dts=[tbl2],
-    )
+    step.run_full(ds)
 
-    count, idx_gen = ds.get_full_process_ids([tbl1], [tbl2])
+    count, idx_gen = step.get_full_process_ids(ds)
     idx_dfs = list(idx_gen)
     idx_len = len(pd.concat(idx_dfs)) if len(idx_dfs) > 0 else 0
 
     assert idx_len == 0
 
+
 # TODO тест inc_process 2->1
 # TODO тест inc_process 2->1, удаление строки, 2->1
 
@@ -270,20 +251,16 @@ def test_gen_process_many(dbconn) -> None:
     ds = DataStore(dbconn, create_meta_table=True)
 
     tbl_gen = ds.create_table(
-        'tbl_gen',
-        table_store=TableStoreDB(dbconn, 'tbl_gen_data', TEST_SCHEMA, True)
+        "tbl_gen", table_store=TableStoreDB(dbconn, "tbl_gen_data", TEST_SCHEMA, True)
     )
     tbl1_gen = ds.create_table(
-        'tbl1_gen',
-        table_store=TableStoreDB(dbconn, 'tbl1_gen_data', TEST_SCHEMA, True)
+        "tbl1_gen", table_store=TableStoreDB(dbconn, "tbl1_gen_data", TEST_SCHEMA, True)
     )
     tbl2_gen = ds.create_table(
-        'tbl2_gen',
-        table_store=TableStoreDB(dbconn, 'tbl2_gen_data', TEST_SCHEMA, True)
+        "tbl2_gen", table_store=TableStoreDB(dbconn, "tbl2_gen_data", TEST_SCHEMA, True)
     )
     tbl3_gen = ds.create_table(
-        'tbl3_gen',
-        table_store=TableStoreDB(dbconn, 'tbl3_gen_data', TEST_SCHEMA, True)
+        "tbl3_gen", table_store=TableStoreDB(dbconn, "tbl3_gen_data", TEST_SCHEMA, True)
     )
 
     def gen():
@@ -305,40 +282,39 @@ def test_inc_process_many_modify_values(dbconn) -> None:
     ds = DataStore(dbconn, create_meta_table=True)
 
     tbl = ds.create_table(
-        'tbl',
-        table_store=TableStoreDB(dbconn, 'tbl_data', TEST_SCHEMA, True)
+        "tbl", table_store=TableStoreDB(dbconn, "tbl_data", TEST_SCHEMA, True)
     )
     tbl1 = ds.create_table(
-        'tbl1',
-        table_store=TableStoreDB(dbconn, 'tbl1_data', TEST_SCHEMA, True)
+        "tbl1", table_store=TableStoreDB(dbconn, "tbl1_data", TEST_SCHEMA, True)
     )
     tbl2 = ds.create_table(
-        'tbl2',
-        table_store=TableStoreDB(dbconn, 'tbl2_data', TEST_SCHEMA, True)
+        "tbl2", table_store=TableStoreDB(dbconn, "tbl2_data", TEST_SCHEMA, True)
     )
     tbl3 = ds.create_table(
-        'tbl3',
-        table_store=TableStoreDB(dbconn, 'tbl3_data', TEST_SCHEMA, True)
+        "tbl3", table_store=TableStoreDB(dbconn, "tbl3_data", TEST_SCHEMA, True)
     )
 
     def inc_func(df):
         df1 = df.copy()
         df2 = df.copy()
         df3 = df.copy()
-        df1['a'] += 1
-        df2['a'] += 2
-        df3['a'] += 3
+        df1["a"] += 1
+        df2["a"] += 2
+        df3["a"] += 3
         return df1, df2, df3
 
     tbl.store_chunk(TEST_DF)
 
-    do_full_batch_transform(
-        func=inc_func,
+    step_inc = BatchTransformStep(
         ds=ds,
+        name="step_inc",
+        func=inc_func,
         input_dts=[tbl],
         output_dts=[tbl1, tbl2, tbl3],
     )
 
+    step_inc.run_full(ds)
+
     assert_datatable_equal(tbl1, TEST_DF_INC1)
     assert_datatable_equal(tbl2, TEST_DF_INC2)
     assert_datatable_equal(tbl3, TEST_DF_INC3)
@@ -350,18 +326,21 @@ def inc_func_inv(df):
         df1 = df.copy()
         df2 = df.copy()
         df3 = df.copy()
-        df1['a'] += 1
-        df2['a'] += 2
-        df3['a'] += 3
+        df1["a"] += 1
+        df2["a"] += 2
+        df3["a"] += 3
         return df3, df2, df1
 
-    do_full_batch_transform(
-        func=inc_func_inv,
+    step_inc_inv = BatchTransformStep(
         ds=ds,
+        name="step_inc_inv",
+        func=inc_func_inv,
         input_dts=[tbl],
         output_dts=[tbl3, tbl2, tbl1],
     )
 
+    step_inc_inv.run_full(ds)
+
     assert_datatable_equal(tbl1, TEST_DF_INC1[:5])
     assert_datatable_equal(tbl2, TEST_DF_INC2[:5])
     assert_datatable_equal(tbl3, TEST_DF_INC3[:5])
@@ -370,12 +349,7 @@ def inc_func_inv(df):
 
     tbl.store_chunk(TEST_DF[5:])
 
-    do_full_batch_transform(
-        func=inc_func,
-        ds=ds,
-        input_dts=[tbl],
-        output_dts=[tbl1, tbl2, tbl3],
-    )
+    step_inc.run_full(ds)
 
     assert_datatable_equal(tbl1, TEST_DF_INC1)
     assert_datatable_equal(tbl2, TEST_DF_INC2)
@@ -386,138 +360,102 @@ def test_inc_process_many_several_inputs(dbconn) -> None:
     ds = DataStore(dbconn, create_meta_table=True)
 
     tbl = ds.create_table(
-        'tbl',
+        "tbl",
         table_store=TableStoreDB(
             dbconn,
-            'tbl_data',
+            "tbl_data",
             [
-                Column('id', Integer, primary_key=True),
-                Column('a_first', Integer),
-                Column('a_second', Integer)
+                Column("id", Integer, primary_key=True),
+                Column("a_first", Integer),
+                Column("a_second", Integer),
             ],
-            True
-        )
+            True,
+        ),
     )
     tbl1 = ds.create_table(
-        'tbl1',
-        table_store=TableStoreDB(dbconn, 'tbl1_data', TEST_SCHEMA, True)
+        "tbl1", table_store=TableStoreDB(dbconn, "tbl1_data", TEST_SCHEMA, True)
     )
     tbl2 = ds.create_table(
-        'tbl2',
-        table_store=TableStoreDB(dbconn, 'tbl2_data', TEST_SCHEMA, True)
+        "tbl2", table_store=TableStoreDB(dbconn, "tbl2_data", TEST_SCHEMA, True)
     )
 
     def inc_func(df1, df2):
-        df = pd.merge(
-            left=df1,
-            right=df2,
-            on=['id'],
-            suffixes=('_first', '_second')
-        )
-        df['a_first'] += 1
-        df['a_second'] += 2
+        df = pd.merge(left=df1, right=df2, on=["id"], suffixes=("_first", "_second"))
+        df["a_first"] += 1
+        df["a_second"] += 2
         return df
 
     tbl1.store_chunk(TEST_DF)
     tbl2.store_chunk(TEST_DF)
 
-    do_full_batch_transform(
+    step = BatchTransformStep(
         ds=ds,
+        name="test",
+        func=inc_func,
         input_dts=[tbl1, tbl2],
         output_dts=[tbl],
-        func=inc_func,
     )
 
+    step.run_full(ds)
+
     assert_datatable_equal(
         tbl,
         pd.DataFrame(
             {
-                'id': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9],
-                'a_first': [1, 2, 3, 4, 5, 6, 7, 8, 9, 10],
-                'a_second': [2, 3, 4, 5, 6, 7, 8, 9, 10, 11]
+                "id": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9],
+                "a_first": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10],
+                "a_second": [2, 3, 4, 5, 6, 7, 8, 9, 10, 11],
             }
-        )
+        ),
     )
 
     changed_ids = [0, 4, 6]
-    changed_ids_df = cast(IndexDF, pd.DataFrame({'id': changed_ids}))
+    changed_ids_df = cast(IndexDF, pd.DataFrame({"id": changed_ids}))
     not_changed_ids = [1, 2, 3, 5, 7, 8, 9]
-    not_changed_ids_df = cast(IndexDF, pd.DataFrame({'id': not_changed_ids}))
+    not_changed_ids_df = cast(IndexDF, pd.DataFrame({"id": not_changed_ids}))
 
-    tbl2.store_chunk(
-        pd.DataFrame(
-            {
-                'id': changed_ids,
-                'a': [10, 10, 10]
-            }
-        )
-    )
+    tbl2.store_chunk(pd.DataFrame({"id": changed_ids, "a": [10, 10, 10]}))
 
-    do_full_batch_transform(
-        ds=ds,
-        input_dts=[tbl1, tbl2],
-        output_dts=[tbl],
-        func=inc_func
-    )
+    step.run_full(ds)
 
     assert_df_equal(
         tbl.get_data(idx=changed_ids_df),
         pd.DataFrame(
-            {
-                'id': changed_ids,
-                'a_first': [1, 5, 7],
-                'a_second': [12, 12, 12]
-            }
-        )
+            {"id": changed_ids, "a_first": [1, 5, 7], "a_second": [12, 12, 12]}
+        ),
     )
 
     assert_df_equal(
         tbl.get_data(idx=not_changed_ids_df),
         pd.DataFrame(
             {
-                'id': not_changed_ids,
-                'a_first': [2, 3, 4, 6, 8, 9, 10],
-                'a_second': [3, 4, 5, 7, 9, 10, 11]
+                "id": not_changed_ids,
+                "a_first": [2, 3, 4, 6, 8, 9, 10],
+                "a_second": [3, 4, 5, 7, 9, 10, 11],
             }
-        )
+        ),
     )
 
-    tbl1.store_chunk(
-        pd.DataFrame(
-            {
-                'id': changed_ids,
-                'a': [20, 20, 20]
-            }
-        )
-    )
+    tbl1.store_chunk(pd.DataFrame({"id": changed_ids, "a": [20, 20, 20]}))
 
-    do_full_batch_transform(
-        func=inc_func,
-        ds=ds,
-        input_dts=[tbl1, tbl2],
-        output_dts=[tbl],
-    )
+    step.run_full(ds)
 
     assert_df_equal(
         tbl.get_data(idx=changed_ids_df),
         pd.DataFrame(
-            {
-                'id': changed_ids,
-                'a_first': [21, 21, 21],
-                'a_second': [12, 12, 12]
-            }
-        )
+            {"id": changed_ids, "a_first": [21, 21, 21], "a_second": [12, 12, 12]}
+        ),
     )
 
     assert_df_equal(
         tbl.get_data(idx=not_changed_ids_df),
         pd.DataFrame(
             {
-                'id': not_changed_ids,
-                'a_first': [2, 3, 4, 6, 8, 9, 10],
-                'a_second': [3, 4, 5, 7, 9, 10, 11]
+                "id": not_changed_ids,
+                "a_first": [2, 3, 4, 6, 8, 9, 10],
+                "a_second": [3, 4, 5, 7, 9, 10, 11],
             }
-        )
+        ),
     )
 
 
@@ -528,43 +466,38 @@ def test_inc_process_many_several_outputs(dbconn) -> None:
     good_ids = [2, 3, 4, 6, 7, 9]
 
     tbl = ds.create_table(
-        'tbl',
-        table_store=TableStoreDB(dbconn, 'tbl_data', TEST_SCHEMA, True)
+        "tbl", table_store=TableStoreDB(dbconn, "tbl_data", TEST_SCHEMA, True)
     )
     tbl_good = ds.create_table(
-        'tbl_good',
-        table_store=TableStoreDB(dbconn, 'tbl_good_data', TEST_SCHEMA, True)
+        "tbl_good", table_store=TableStoreDB(dbconn, "tbl_good_data", TEST_SCHEMA, True)
     )
     tbl_bad = ds.create_table(
-        'tbl_bad',
-        table_store=TableStoreDB(dbconn, 'tbl_bad_data', TEST_SCHEMA, True)
+        "tbl_bad", table_store=TableStoreDB(dbconn, "tbl_bad_data", TEST_SCHEMA, True)
     )
 
     tbl.store_chunk(TEST_DF)
 
     def inc_func(df):
-        df_good = df[df['id'].isin(good_ids)]
-        df_bad = df[df['id'].isin(bad_ids)]
+        df_good = df[df["id"].isin(good_ids)]
+        df_bad = df[df["id"].isin(bad_ids)]
         return df_good, df_bad
 
-    do_full_batch_transform(
-        func=inc_func,
+    step = BatchTransformStep(
         ds=ds,
+        name="test",
+        func=inc_func,
         input_dts=[tbl],
         output_dts=[tbl_good, tbl_bad],
     )
 
+    step.run_full(ds)
+
     assert_datatable_equal(tbl, TEST_DF)
     assert_datatable_equal(tbl_good, TEST_DF.loc[good_ids])
     assert_datatable_equal(tbl_bad, TEST_DF.loc[bad_ids])
 
     # Check this not delete the tables
-    do_full_batch_transform(
-        func=inc_func,
-        ds=ds,
-        input_dts=[tbl],
-        output_dts=[tbl_good, tbl_bad],
-    )
+    step.run_full(ds)
 
     assert_datatable_equal(tbl, TEST_DF)
     assert_datatable_equal(tbl_good, TEST_DF.loc[good_ids])
@@ -575,69 +508,62 @@ def test_inc_process_many_one_to_many(dbconn) -> None:
     ds = DataStore(dbconn, create_meta_table=True)
 
     tbl = ds.create_table(
-        'tbl',
-        table_store=TableStoreDB(dbconn, 'tbl_data', TEST_SCHEMA_OTM, True)
+        "tbl", table_store=TableStoreDB(dbconn, "tbl_data", TEST_SCHEMA_OTM, True)
     )
     tbl_rel = ds.create_table(
-        'tbl_rel',
-        table_store=TableStoreDB(dbconn, 'tbl_rel_data', TEST_SCHEMA_OTM2, True)
+        "tbl_rel",
+        table_store=TableStoreDB(dbconn, "tbl_rel_data", TEST_SCHEMA_OTM2, True),
     )
     tbl2 = ds.create_table(
-        'tbl2',
-        table_store=TableStoreDB(dbconn, 'tbl2_data', TEST_SCHEMA_OTM, True)
+        "tbl2", table_store=TableStoreDB(dbconn, "tbl2_data", TEST_SCHEMA_OTM, True)
     )
 
     tbl.store_chunk(TEST_OTM_DF)
 
     def inc_func_unpack(df):
-        res_df = df.explode('a')
+        res_df = df.explode("a")
 
-        return res_df[res_df['a'].notna()]
+        return res_df[res_df["a"].notna()]
 
     def inc_func_pack(df):
         res_df = pd.DataFrame()
-        res_df['a'] = df.groupby('id').apply(lambda x: x['a'].dropna().to_list())
+        res_df["a"] = df.groupby("id").apply(lambda x: x["a"].dropna().to_list())
 
         return res_df.reset_index()
 
-    rel_df = TEST_OTM_DF.explode('a')
-    rel_df = rel_df[rel_df['a'].notna()]
+    rel_df = TEST_OTM_DF.explode("a")
+    rel_df = rel_df[rel_df["a"].notna()]
 
-    do_full_batch_transform(
-        inc_func_unpack,
+    step_unpack = BatchTransformStep(
         ds=ds,
+        name="unpack",
+        func=inc_func_unpack,
         input_dts=[tbl],
         output_dts=[tbl_rel],
     )
-    do_full_batch_transform(
-        inc_func_pack,
+    step_pack = BatchTransformStep(
         ds=ds,
+        name="pack",
+        func=inc_func_pack,
         input_dts=[tbl_rel],
         output_dts=[tbl2],
     )
 
+    step_unpack.run_full(ds)
+    step_pack.run_full(ds)
+
     assert_datatable_equal(tbl, TEST_OTM_DF)
     assert_datatable_equal(tbl_rel, rel_df)
     assert_datatable_equal(tbl2, TEST_OTM_DF.loc[1:])
 
     # Delete rows test
-    tbl.delete_by_idx(cast(IndexDF, TEST_OTM_DF.loc[[9], ['id']]))
+    tbl.delete_by_idx(cast(IndexDF, TEST_OTM_DF.loc[[9], ["id"]]))
 
-    rel_df = TEST_OTM_DF.loc[:8].explode('a')
-    rel_df = rel_df[rel_df['a'].notna()]
+    rel_df = TEST_OTM_DF.loc[:8].explode("a")
+    rel_df = rel_df[rel_df["a"].notna()]
 
-    do_full_batch_transform(
-        inc_func_unpack,
-        ds=ds,
-        input_dts=[tbl],
-        output_dts=[tbl_rel],
-    )
-    do_full_batch_transform(
-        inc_func_pack,
-        ds=ds,
-        input_dts=[tbl_rel],
-        output_dts=[tbl2],
-    )
+    step_unpack.run_full(ds)
+    step_pack.run_full(ds)
 
     assert_datatable_equal(tbl, TEST_OTM_DF.loc[:8])
     assert_datatable_equal(tbl_rel, rel_df)
@@ -648,110 +574,93 @@ def test_inc_process_many_one_to_many_change_primary(dbconn) -> None:
     ds = DataStore(dbconn, create_meta_table=True)
 
     tbl = ds.create_table(
-        'tbl',
-        table_store=TableStoreDB(dbconn, 'tbl_data', TEST_SCHEMA_OTM, True)
+        "tbl", table_store=TableStoreDB(dbconn, "tbl_data", TEST_SCHEMA_OTM, True)
     )
     tbl_rel = ds.create_table(
-        'tbl_rel',
-        table_store=TableStoreDB(dbconn, 'tbl_rel_data', TEST_SCHEMA_OTM2, True)
+        "tbl_rel",
+        table_store=TableStoreDB(dbconn, "tbl_rel_data", TEST_SCHEMA_OTM2, True),
     )
     tbl2 = ds.create_table(
-        'tbl2',
-        table_store=TableStoreDB(dbconn, 'tbl2_data', TEST_SCHEMA_OTM3, True)
+        "tbl2", table_store=TableStoreDB(dbconn, "tbl2_data", TEST_SCHEMA_OTM3, True)
     )
 
     tbl.store_chunk(TEST_OTM_DF)
 
     def inc_func_unpack(df):
-        res_df = df.explode('a')
-        return res_df[res_df['a'].notna()]
+        res_df = df.explode("a")
+        return res_df[res_df["a"].notna()]
 
     def inc_func_pack(df):
         res_df = pd.DataFrame()
-        res_df['ids'] = df.groupby('a').apply(lambda x: x['id'].dropna().to_list())
-        res_df['b'] = 1
+        res_df["ids"] = df.groupby("a").apply(lambda x: x["id"].dropna().to_list())
+        res_df["b"] = 1
 
         return res_df.reset_index()
 
-    rel_df = TEST_OTM_DF.explode('a')
-    rel_df = rel_df[rel_df['a'].notna()]
+    rel_df = TEST_OTM_DF.explode("a")
+    rel_df = rel_df[rel_df["a"].notna()]
 
     a_df = pd.DataFrame()
-    a_df['ids'] = rel_df.groupby('a').apply(lambda x: x['id'].dropna().to_list())
-    a_df['b'] = 1
+    a_df["ids"] = rel_df.groupby("a").apply(lambda x: x["id"].dropna().to_list())
+    a_df["b"] = 1
 
     a_df.reset_index(inplace=True)
 
-    do_full_batch_transform(
-        inc_func_unpack,
+    step_unpack = BatchTransformStep(
         ds=ds,
+        name="unpack",
+        func=inc_func_unpack,
         input_dts=[tbl],
         output_dts=[tbl_rel],
     )
-    do_full_batch_transform(
-        inc_func_pack,
+    step_pack = BatchTransformStep(
         ds=ds,
+        name="pack",
+        func=inc_func_pack,
         input_dts=[tbl_rel],
         output_dts=[tbl2],
     )
 
+    step_unpack.run_full(ds)
+    step_pack.run_full(ds)
+
     assert_datatable_equal(tbl, TEST_OTM_DF)
     assert_datatable_equal(tbl_rel, rel_df)
     assert_datatable_equal(tbl2, a_df)
 
     # Delete row with empty relations
-    tbl.delete_by_idx(cast(IndexDF, TEST_OTM_DF.loc[[0], ['id']]))
+    tbl.delete_by_idx(cast(IndexDF, TEST_OTM_DF.loc[[0], ["id"]]))
 
-    rel_df = TEST_OTM_DF.loc[1:].explode('a')
-    rel_df = rel_df[rel_df['a'].notna()]
+    rel_df = TEST_OTM_DF.loc[1:].explode("a")
+    rel_df = rel_df[rel_df["a"].notna()]
 
     a_df = pd.DataFrame()
-    a_df['ids'] = rel_df.groupby('a').apply(lambda x: x['id'].dropna().to_list())
-    a_df['b'] = 1
+    a_df["ids"] = rel_df.groupby("a").apply(lambda x: x["id"].dropna().to_list())
+    a_df["b"] = 1
 
     a_df.reset_index(inplace=True)
 
-    do_full_batch_transform(
-        inc_func_unpack,
-        ds=ds,
-        input_dts=[tbl],
-        output_dts=[tbl_rel],
-    )
-    do_full_batch_transform(
-        inc_func_pack,
-        ds=ds,
-        input_dts=[tbl_rel],
-        output_dts=[tbl2],
-    )
+    step_unpack.run_full(ds)
+    step_pack.run_full(ds)
 
     assert_datatable_equal(tbl, TEST_OTM_DF.loc[1:])
     assert_datatable_equal(tbl_rel, rel_df)
     assert_datatable_equal(tbl2, a_df)
 
     # Delete rows test
-    tbl.delete_by_idx(cast(IndexDF, TEST_OTM_DF.loc[[1], ['id']]))
+    tbl.delete_by_idx(cast(IndexDF, TEST_OTM_DF.loc[[1], ["id"]]))
 
-    rel_df = TEST_OTM_DF.loc[2:].explode('a')
-    rel_df = rel_df[rel_df['a'].notna()]
+    rel_df = TEST_OTM_DF.loc[2:].explode("a")
+    rel_df = rel_df[rel_df["a"].notna()]
 
     a_df = pd.DataFrame()
-    a_df['ids'] = rel_df.groupby('a').apply(lambda x: x['id'].dropna().to_list())
-    a_df['b'] = 1
+    a_df["ids"] = rel_df.groupby("a").apply(lambda x: x["id"].dropna().to_list())
+    a_df["b"] = 1
 
     a_df.reset_index(inplace=True)
 
-    do_full_batch_transform(
-        inc_func_unpack,
-        ds=ds,
-        input_dts=[tbl],
-        output_dts=[tbl_rel],
-    )
-    do_full_batch_transform(
-        inc_func_pack,
-        ds=ds,
-        input_dts=[tbl_rel],
-        output_dts=[tbl2],
-    )
+    step_unpack.run_full(ds)
+    step_pack.run_full(ds)
 
     assert_datatable_equal(tbl, TEST_OTM_DF.loc[2:])
     assert_datatable_equal(tbl_rel, rel_df)
@@ -766,13 +675,13 @@ def test_error_handling(dbconn) -> None:
     ds = DataStore(dbconn, create_meta_table=True)
 
     tbl = ds.create_table(
-        'tbl',
-        table_store=TableStoreDB(dbconn, 'tbl1_data', TEST_SCHEMA, True)
+        "tbl",
+        table_store=TableStoreDB(dbconn, "tbl1_data", TEST_SCHEMA, True),
     )
 
     tbl_good = ds.create_table(
-        'tbl_good',
-        table_store=TableStoreDB(dbconn, 'tbl_good_data', TEST_SCHEMA, True)
+        "tbl_good",
+        table_store=TableStoreDB(dbconn, "tbl_good_data", TEST_SCHEMA, True),
     )
 
     def gen_bad1(chunk_size: int = 1000):
@@ -782,7 +691,7 @@ def gen_bad1(chunk_size: int = 1000):
             if i >= chunk_size * 3:
                 raise Exception("Test")
 
-            yield TEST_DF.loc[idx[i:i+chunk_size]]
+            yield TEST_DF.loc[idx[i : i + chunk_size]]
 
     def gen_bad2(chunk_size: int = 1000):
         idx = TEST_DF.index
@@ -791,7 +700,7 @@ def gen_bad2(chunk_size: int = 1000):
             if i >= chunk_size * 2:
                 raise Exception("Test")
 
-            yield TEST_DF.loc[idx[i:i+chunk_size]]
+            yield TEST_DF.loc[idx[i : i + chunk_size]]
 
     # with pytest.raises(Exception):
     do_batch_generate(
@@ -803,30 +712,34 @@ def gen_bad2(chunk_size: int = 1000):
     assert_datatable_equal(tbl, TEST_DF.loc[GOOD_IDXS1])
 
     def inc_func_bad(df):
-        if BAD_ID in df['id'].values:
-            raise Exception('TEST')
+        if BAD_ID in df["id"].values:
+            raise Exception("TEST")
         return df
 
     def inc_func_good(df):
         return df
 
-    do_full_batch_transform(
-        func=inc_func_bad,
+    step_bad = BatchTransformStep(
         ds=ds,
+        name="bad",
+        func=inc_func_bad,
         input_dts=[tbl],
         output_dts=[tbl_good],
         chunk_size=1,
     )
+    step_bad.run_full(ds)
 
     assert_datatable_equal(tbl_good, TEST_DF.loc[[0, 1, 2, 4, 5]])
 
-    do_full_batch_transform(
-        func=inc_func_good,
+    step_good = BatchTransformStep(
         ds=ds,
+        name="good",
+        func=inc_func_good,
         input_dts=[tbl],
         output_dts=[tbl_good],
         chunk_size=CHUNKSIZE,
     )
+    step_good.run_full(ds)
 
     assert_datatable_equal(tbl_good, TEST_DF.loc[GOOD_IDXS1])
 
@@ -840,27 +753,19 @@ def inc_func_good(df):
 
     assert_datatable_equal(tbl, TEST_DF.loc[GOOD_IDXS1])
 
-    do_full_batch_transform(
-        func=inc_func_bad,
-        ds=ds,
-        input_dts=[tbl],
-        output_dts=[tbl_good],
-    )
+    step_bad.run_full(ds)
 
     assert_datatable_equal(tbl_good, TEST_DF.loc[GOOD_IDXS1])
 
 
 def test_gen_from_empty_rows(dbconn) -> None:
     ds = DataStore(dbconn, create_meta_table=True)
-    tbl = ds.create_table('test', table_store=TableStoreDB(dbconn, 'tbl_data', TEST_SCHEMA, True))
+    tbl = ds.create_table(
+        "test", table_store=TableStoreDB(dbconn, "tbl_data", TEST_SCHEMA, True)
+    )
 
     def proc_func():
-        yield pd.DataFrame.from_records(
-            {
-                key: []
-                for key in tbl.primary_keys
-            }
-        )
+        yield pd.DataFrame.from_records({key: [] for key in tbl.primary_keys})
 
     # This should be ok
     do_batch_generate(
@@ -872,7 +777,9 @@ def proc_func():
 
 def test_gen_from_empty_df(dbconn) -> None:
     ds = DataStore(dbconn, create_meta_table=True)
-    tbl = ds.create_table('test', table_store=TableStoreDB(dbconn, 'tbl_data', TEST_SCHEMA, True))
+    tbl = ds.create_table(
+        "test", table_store=TableStoreDB(dbconn, "tbl_data", TEST_SCHEMA, True)
+    )
 
     def proc_func():
         yield pd.DataFrame()
diff --git a/tests/test_core_steps2.py b/tests/test_core_steps2.py
index 6f044cbc..fe3765eb 100644
--- a/tests/test_core_steps2.py
+++ b/tests/test_core_steps2.py
@@ -6,11 +6,10 @@
 import time
 
 import pandas as pd
-from sqlalchemy import Column
+from sqlalchemy import Column, String
 from sqlalchemy.sql.sqltypes import Integer
 
-from datapipe.core_steps import (BatchTransformStep, do_batch_generate,
-                                 do_full_batch_transform)
+from datapipe.core_steps import BatchTransformStep, do_batch_generate
 from datapipe.datatable import DataStore
 from datapipe.run_config import RunConfig
 from datapipe.store.database import MetaKey, TableStoreDB
@@ -19,70 +18,60 @@
 from .util import assert_datatable_equal, assert_df_equal
 
 TEST_SCHEMA1 = [
-    Column('item_id', Integer, primary_key=True),
-    Column('pipeline_id', Integer, primary_key=True),
-    Column('a', Integer),
+    Column("item_id", Integer, primary_key=True),
+    Column("pipeline_id", Integer, primary_key=True),
+    Column("a", Integer),
 ]
 
 TEST_SCHEMA2 = [
-    Column('item_id', Integer, primary_key=True),
-    Column('a', Integer),
-]
-
-TEST_SCHEMA2 = [
-    Column('item_id', Integer, primary_key=True),
-    Column('a', Integer),
-]
-
-TEST_SCHEMA2 = [
-    Column('item_id', Integer, primary_key=True),
-    Column('a', Integer),
+    Column("item_id", Integer, primary_key=True),
+    Column("a", Integer),
 ]
 
 PRODUCTS_SCHEMA = [
-    Column('product_id', Integer, primary_key=True),
-    Column('pipeline_id', Integer, primary_key=True),
-    Column('b', Integer),
+    Column("product_id", Integer, primary_key=True),
+    Column("pipeline_id", Integer, primary_key=True),
+    Column("b", Integer),
 ]
 
 ITEMS_SCHEMA = [
-    Column('item_id', Integer, primary_key=True),
-    Column('pipeline_id', Integer, primary_key=True),
-    Column('product_id', Integer, MetaKey()),
-    Column('a', Integer),
+    Column("item_id", Integer, primary_key=True),
+    Column("pipeline_id", Integer, primary_key=True),
+    Column("product_id", Integer, MetaKey()),
+    Column("a", Integer),
 ]
 
 TEST_DF1_1 = pd.DataFrame(
     {
-        'item_id': range(10),
-        'pipeline_id': [i // 5 for i in range(10)],
-        'a': range(10),
+        "item_id": range(10),
+        "pipeline_id": [i // 5 for i in range(10)],
+        "a": range(10),
     },
 )
 
 
 TEST_DF1_2 = pd.DataFrame(
     {
-        'item_id': list(range(5)) * 2,
-        'pipeline_id': [i // 5 for i in range(10)],
-        'a': range(10),
+        "item_id": list(range(5)) * 2,
+        "pipeline_id": [i // 5 for i in range(10)],
+        "a": range(10),
     },
 )
 
 PRODUCTS_DF = pd.DataFrame(
     {
-        'product_id': list(range(2)),
-        'pipeline_id': list(range(2)),
-        'b': range(10, 12),
+        "product_id": list(range(2)),
+        "pipeline_id": list(range(2)),
+        "b": range(10, 12),
     }
 )
 
 ITEMS_DF = pd.DataFrame(
     {
-        'item_id': list(range(5)) * 2,
-        'pipeline_id': list(range(2)) * 5,
-        'product_id': list(range(2)) * 5,
-        'a': range(10),
+        "item_id": list(range(5)) * 2,
+        "pipeline_id": list(range(2)) * 5,
+        "product_id": list(range(2)) * 5,
+        "a": range(10),
     }
 )
 
@@ -91,111 +80,112 @@ def test_batch_transform(dbconn):
     ds = DataStore(dbconn, create_meta_table=True)
 
     tbl1 = ds.create_table(
-        'tbl1',
-        table_store=TableStoreDB(dbconn, 'tbl1_data', TEST_SCHEMA1, True)
+        "tbl1", table_store=TableStoreDB(dbconn, "tbl1_data", TEST_SCHEMA1, True)
     )
 
     tbl2 = ds.create_table(
-        'tbl2',
-        table_store=TableStoreDB(dbconn, 'tbl2_data', TEST_SCHEMA1, True)
+        "tbl2", table_store=TableStoreDB(dbconn, "tbl2_data", TEST_SCHEMA1, True)
     )
 
     tbl1.store_chunk(TEST_DF1_1, now=0)
 
-    do_full_batch_transform(
-        func=lambda df: df,
+    step = BatchTransformStep(
         ds=ds,
+        name="test",
+        func=lambda df: df,
         input_dts=[tbl1],
         output_dts=[tbl2],
     )
 
+    step.run_full(ds)
+
     meta_df = tbl2.get_metadata()
 
-    update_ts = max(meta_df['update_ts'])
-    process_ts = max(meta_df['process_ts'])
+    update_ts = max(meta_df["update_ts"])
+    process_ts = max(meta_df["process_ts"])
 
     time.sleep(0.1)
 
-    do_full_batch_transform(
-        func=lambda df: df,
-        ds=ds,
-        input_dts=[tbl1],
-        output_dts=[tbl2],
-    )
+    step.run_full(ds)
 
     meta_df = tbl2.get_metadata()
 
-    assert all(meta_df['update_ts'] == update_ts)
-    assert all(meta_df['process_ts'] == process_ts)
+    assert all(meta_df["update_ts"] == update_ts)
+    assert all(meta_df["process_ts"] == process_ts)
 
 
 def test_batch_transform_with_filter(dbconn):
     ds = DataStore(dbconn, create_meta_table=True)
 
     tbl1 = ds.create_table(
-        'tbl1',
-        table_store=TableStoreDB(dbconn, 'tbl1_data', TEST_SCHEMA1, True)
+        "tbl1", table_store=TableStoreDB(dbconn, "tbl1_data", TEST_SCHEMA1, True)
     )
 
     tbl2 = ds.create_table(
-        'tbl2',
-        table_store=TableStoreDB(dbconn, 'tbl2_data', TEST_SCHEMA1, True)
+        "tbl2", table_store=TableStoreDB(dbconn, "tbl2_data", TEST_SCHEMA1, True)
     )
 
     tbl1.store_chunk(TEST_DF1_1, now=0)
 
-    do_full_batch_transform(
-        func=lambda df: df,
+    step = BatchTransformStep(
         ds=ds,
+        name="test",
+        func=lambda df: df,
         input_dts=[tbl1],
         output_dts=[tbl2],
-        run_config=RunConfig(filters={'pipeline_id': 0})
+    )
+    step.run_full(
+        ds,
+        run_config=RunConfig(
+            filters={"pipeline_id": 0},
+        ),
     )
 
-    assert_datatable_equal(tbl2, TEST_DF1_1.query('pipeline_id == 0'))
+    assert_datatable_equal(tbl2, TEST_DF1_1.query("pipeline_id == 0"))
 
 
 def test_batch_transform_with_filter_not_in_transform_index(dbconn):
     ds = DataStore(dbconn, create_meta_table=True)
 
     tbl1 = ds.create_table(
-        'tbl1',
-        table_store=TableStoreDB(dbconn, 'tbl1_data', TEST_SCHEMA1, True)
+        "tbl1", table_store=TableStoreDB(dbconn, "tbl1_data", TEST_SCHEMA1, True)
     )
 
     tbl2 = ds.create_table(
-        'tbl2',
-        table_store=TableStoreDB(dbconn, 'tbl2_data', TEST_SCHEMA2, True)
+        "tbl2", table_store=TableStoreDB(dbconn, "tbl2_data", TEST_SCHEMA2, True)
     )
 
     tbl1.store_chunk(TEST_DF1_2, now=0)
 
-    do_full_batch_transform(
-        func=lambda df: df[['item_id', 'a']],
+    step = BatchTransformStep(
         ds=ds,
+        name="test",
+        func=lambda df: df[["item_id", "a"]],
         input_dts=[tbl1],
         output_dts=[tbl2],
-        run_config=RunConfig(filters={'pipeline_id': 0}),
     )
 
-    assert_datatable_equal(tbl2, TEST_DF1_2.query('pipeline_id == 0')[['item_id', 'a']])
+    step.run_full(
+        ds,
+        run_config=RunConfig(filters={"pipeline_id": 0}),
+    )
+
+    assert_datatable_equal(tbl2, TEST_DF1_2.query("pipeline_id == 0")[["item_id", "a"]])
 
 
 def test_batch_transform_with_dt_on_input_and_output(dbconn):
     ds = DataStore(dbconn, create_meta_table=True)
 
     tbl1 = ds.create_table(
-        'tbl1',
-        table_store=TableStoreDB(dbconn, 'tbl1_data', TEST_SCHEMA1, True)
+        "tbl1", table_store=TableStoreDB(dbconn, "tbl1_data", TEST_SCHEMA1, True)
     )
 
     tbl2 = ds.create_table(
-        'tbl2',
-        table_store=TableStoreDB(dbconn, 'tbl2_data', TEST_SCHEMA1, True)
+        "tbl2", table_store=TableStoreDB(dbconn, "tbl2_data", TEST_SCHEMA1, True)
     )
 
     df2 = TEST_DF1_1.loc[range(3, 8)]
-    df2['a'] = df2['a'].apply(lambda x: x + 10)
+    df2["a"] = df2["a"].apply(lambda x: x + 10)
 
     tbl1.store_chunk(TEST_DF1_1, now=0)
     tbl2.store_chunk(df2, now=0)
@@ -208,13 +198,16 @@ def update_df(df1: pd.DataFrame, df2: pd.DataFrame):
 
         return df1.reset_index()
 
-    do_full_batch_transform(
-        func=update_df,
+    step = BatchTransformStep(
         ds=ds,
+        name="test",
+        func=update_df,
         input_dts=[tbl1, tbl2],
         output_dts=[tbl2],
     )
 
+    step.run_full(ds)
+
     df_res = TEST_DF1_1.copy()
 
     df_res.update(df2)
@@ -222,40 +215,104 @@ def update_df(df1: pd.DataFrame, df2: pd.DataFrame):
     assert_datatable_equal(tbl2, df_res)
 
 
+def test_batch_transform_with_fails(dbconn):
+    ds = DataStore(dbconn, create_meta_table=True)
+
+    tbl1 = ds.create_table(
+        "tbl1",
+        table_store=TableStoreDB(
+            dbconn,
+            "tbl1",
+            [
+                Column("id", Integer, primary_key=True),
+                Column("data", String),
+            ],
+            create_table=True,
+        ),
+    )
+
+    tbl2 = ds.create_table(
+        "tbl2",
+        table_store=TableStoreDB(
+            dbconn,
+            "tbl2",
+            [
+                Column("id", Integer, primary_key=True),
+                Column("data", String),
+            ],
+            create_table=True,
+        ),
+    )
+
+    context = {
+        "fail": True,
+    }
+
+    def transform_func(df, context=context):
+        if context["fail"]:
+            raise ValueError("fail")
+        return df
+
+    step = BatchTransformStep(
+        ds=ds,
+        name="step1",
+        func=transform_func,
+        input_dts=[tbl1],
+        output_dts=[tbl2],
+    )
+
+    tbl1.store_chunk(
+        pd.DataFrame(
+            {
+                "id": range(10),
+                "data": ["a"] * 10,
+            }
+        )
+    )
+
+    step.run_full(ds)
+    assert len(tbl2.get_data()) == 0
+
+    context["fail"] = False
+
+    step.run_full(ds)
+
+    assert len(tbl2.get_data()) == 10
+
+
 def test_gen_with_filter(dbconn):
     ds = DataStore(dbconn, create_meta_table=True)
 
     tbl = ds.create_table(
-        'tbl',
-        table_store=TableStoreDB(dbconn, 'tbl_data', TEST_SCHEMA1, True)
+        "tbl", table_store=TableStoreDB(dbconn, "tbl_data", TEST_SCHEMA1, True)
     )
 
     tbl.store_chunk(TEST_DF1_1, now=0)
 
     def gen_func():
-        yield TEST_DF1_1.query('pipeline_id == 0 and item_id == 0')
+        yield TEST_DF1_1.query("pipeline_id == 0 and item_id == 0")
 
     do_batch_generate(
         func=gen_func,
         ds=ds,
         output_dts=[tbl],
-        run_config=RunConfig(filters={'pipeline_id': 0})
+        run_config=RunConfig(filters={"pipeline_id": 0}),
     )
 
-    assert_datatable_equal(tbl, TEST_DF1_1.query('(pipeline_id == 0 and item_id == 0) or pipeline_id == 1'))
+    assert_datatable_equal(
+        tbl, TEST_DF1_1.query("(pipeline_id == 0 and item_id == 0) or pipeline_id == 1")
+    )
 
 
 def test_transform_with_changelist(dbconn):
     ds = DataStore(dbconn, create_meta_table=True)
 
     tbl1 = ds.create_table(
-        'tbl1',
-        table_store=TableStoreDB(dbconn, 'tbl1_data', TEST_SCHEMA1, True)
+        "tbl1", table_store=TableStoreDB(dbconn, "tbl1_data", TEST_SCHEMA1, True)
     )
 
     tbl2 = ds.create_table(
-        'tbl2',
-        table_store=TableStoreDB(dbconn, 'tbl2_data', TEST_SCHEMA1, True)
+        "tbl2", table_store=TableStoreDB(dbconn, "tbl2_data", TEST_SCHEMA1, True)
     )
 
     tbl1.store_chunk(TEST_DF1_1, now=0)
@@ -264,70 +321,68 @@ def func(df):
         return df
 
     step = BatchTransformStep(
-        'test',
+        ds=ds,
+        name="test",
         func=func,
         input_dts=[tbl1],
-        output_dts=[tbl2]
+        output_dts=[tbl2],
     )
 
     change_list = ChangeList()
 
-    idx_keys = ['item_id', 'pipeline_id']
+    idx_keys = ["item_id", "pipeline_id"]
     changes_df = TEST_DF1_1.loc[[0, 1, 2]]
     changes_idx = IndexDF(changes_df[idx_keys])
 
-    change_list.append('tbl1', changes_idx)
+    change_list.append("tbl1", changes_idx)
 
     next_change_list = step.run_changelist(ds, change_list)
 
     assert_datatable_equal(tbl2, changes_df)
 
-    assert list(next_change_list.changes.keys()) == ['tbl2']
+    assert list(next_change_list.changes.keys()) == ["tbl2"]
 
-    assert_df_equal(next_change_list.changes['tbl2'], changes_idx, index_cols=idx_keys)
+    assert_df_equal(next_change_list.changes["tbl2"], changes_idx, index_cols=idx_keys)
 
 
 def test_batch_transform_with_entity(dbconn):
     ds = DataStore(dbconn, create_meta_table=True)
 
     products = ds.create_table(
-        'products',
-        table_store=TableStoreDB(dbconn, 'products_data', PRODUCTS_SCHEMA, True)
+        "products",
+        table_store=TableStoreDB(dbconn, "products_data", PRODUCTS_SCHEMA, True),
     )
 
     items = ds.create_table(
-        'items',
-        table_store=TableStoreDB(dbconn, 'items_data', ITEMS_SCHEMA, True)
+        "items", table_store=TableStoreDB(dbconn, "items_data", ITEMS_SCHEMA, True)
     )
 
     items2 = ds.create_table(
-        'items2',
-        table_store=TableStoreDB(dbconn, 'items2_data', ITEMS_SCHEMA, True)
+        "items2", table_store=TableStoreDB(dbconn, "items2_data", ITEMS_SCHEMA, True)
     )
 
     products.store_chunk(PRODUCTS_DF, now=0)
     items.store_chunk(ITEMS_DF, now=0)
 
     def update_df(products: pd.DataFrame, items: pd.DataFrame):
-        merged_df = pd.merge(items, products, on=['product_id', 'pipeline_id'])
-        merged_df['a'] = merged_df.apply(lambda x: x['a'] + x['b'], axis=1)
+        merged_df = pd.merge(items, products, on=["product_id", "pipeline_id"])
+        merged_df["a"] = merged_df.apply(lambda x: x["a"] + x["b"], axis=1)
 
-        return merged_df[['item_id', 'pipeline_id', 'product_id', 'a']]
+        return merged_df[["item_id", "pipeline_id", "product_id", "a"]]
 
-    do_full_batch_transform(
-        func=update_df,
+    step = BatchTransformStep(
         ds=ds,
+        name="test",
+        func=update_df,
         input_dts=[products, items],
         output_dts=[items2],
     )
 
-    merged_df = pd.merge(ITEMS_DF, PRODUCTS_DF, on=['product_id', 'pipeline_id'])
-    merged_df['a'] = merged_df.apply(lambda x: x['a'] + x['b'], axis=1)
+    step.run_full(ds)
 
-    items2_df = merged_df[['item_id', 'pipeline_id', 'product_id', 'a']]
+    merged_df = pd.merge(ITEMS_DF, PRODUCTS_DF, on=["product_id", "pipeline_id"])
+    merged_df["a"] = merged_df.apply(lambda x: x["a"] + x["b"], axis=1)
 
-    assert_df_equal(
-        items2.get_data(),
-        items2_df,
-        index_cols=['item_id', 'pipeline_id']
-    )
+    items2_df = merged_df[["item_id", "pipeline_id", "product_id", "a"]]
+
+    assert_df_equal(items2.get_data(), items2_df, index_cols=["item_id", "pipeline_id"])
diff --git a/tests/test_datatable.py b/tests/test_datatable.py
index 4b27d80e..728b9154 100644
--- a/tests/test_datatable.py
+++ b/tests/test_datatable.py
@@ -10,49 +10,49 @@
 from .util import assert_datatable_equal, assert_df_equal
 
 TEST_SCHEMA = [
-    Column('id', Integer, primary_key=True),
-    Column('a', Integer),
+    Column("id", Integer, primary_key=True),
+    Column("a", Integer),
 ]
 
 TEST_SCHEMA_OTM = [
-    Column('id', Integer, primary_key=True),
-    Column('a', JSON),
+    Column("id", Integer, primary_key=True),
+    Column("a", JSON),
 ]
 
 TEST_SCHEMA_OTM2 = [
-    Column('id', Integer, primary_key=True),
-    Column('a', Integer, primary_key=True)
+    Column("id", Integer, primary_key=True),
+    Column("a", Integer, primary_key=True),
 ]
 
 TEST_SCHEMA_OTM3 = [
-    Column('a', Integer, primary_key=True),
-    Column('b', Integer, primary_key=True),
-    Column('ids', JSON)
+    Column("a", Integer, primary_key=True),
+    Column("b", Integer, primary_key=True),
+    Column("ids", JSON),
 ]
 
 TEST_DF = pd.DataFrame(
     {
-        'id': range(10),
-        'a': range(10),
+        "id": range(10),
+        "a": range(10),
     },
 )
 
 TEST_OTM_DF = pd.DataFrame(
     {
-        'id': range(10),
-        'a': [[j for j in range(i)] for i in range(10)],
+        "id": range(10),
+        "a": [[j for j in range(i)] for i in range(10)],
     },
 )
 
 
-TEST_DF_INC1 = TEST_DF.assign(a=lambda df: df['a'] + 1)
-TEST_DF_INC2 = TEST_DF.assign(a=lambda df: df['a'] + 2)
-TEST_DF_INC3 = TEST_DF.assign(a=lambda df: df['a'] + 3)
+TEST_DF_INC1 = TEST_DF.assign(a=lambda df: df["a"] + 1)
+TEST_DF_INC2 = TEST_DF.assign(a=lambda df: df["a"] + 2)
+TEST_DF_INC3 = TEST_DF.assign(a=lambda df: df["a"] + 3)
 
 
 def yield_df(data):
     def f(*args, **kwargs):
-        yield pd.DataFrame.from_records(data, columns=['id', 'a']).set_index('id')
+        yield pd.DataFrame.from_records(data, columns=["id", "a"]).set_index("id")
 
     return f
 
@@ -61,8 +61,7 @@ def test_cloudpickle(dbconn) -> None:
     ds = DataStore(dbconn, create_meta_table=True)
 
     tbl = ds.create_table(
-        name='test',
-        table_store=TableStoreDB(dbconn, 'test_data', TEST_SCHEMA, True)
+        name="test", table_store=TableStoreDB(dbconn, "test_data", TEST_SCHEMA, True)
     )
 
     dump = cloudpickle.dumps([ds, tbl])
@@ -72,10 +71,10 @@ def test_cloudpickle(dbconn) -> None:
     dbconn_a = tbl.meta_dbconn
     dbconn_b: DBConn = tbl_desrl.meta_dbconn
 
-    assert (
-        (dbconn_a.connstr, dbconn_a.schema, dbconn_a.supports_update_from)
-        ==
-        (dbconn_b.connstr, dbconn_b.schema, dbconn_b.supports_update_from)
+    assert (dbconn_a.connstr, dbconn_a.schema, dbconn_a.supports_update_from) == (
+        dbconn_b.connstr,
+        dbconn_b.schema,
+        dbconn_b.supports_update_from,
     )
 
 
@@ -83,8 +82,7 @@ def test_simple(dbconn) -> None:
     ds = DataStore(dbconn, create_meta_table=True)
 
     tbl = ds.create_table(
-        'test',
-        table_store=TableStoreDB(dbconn, 'test_data', TEST_SCHEMA, True)
+        "test", table_store=TableStoreDB(dbconn, "test_data", TEST_SCHEMA, True)
     )
 
     tbl.store_chunk(TEST_DF)
@@ -96,8 +94,7 @@ def test_store_less_values(dbconn) -> None:
     ds = DataStore(dbconn, create_meta_table=True)
 
     tbl = ds.create_table(
-        'test',
-        table_store=TableStoreDB(dbconn, 'test_data', TEST_SCHEMA, True)
+        "test", table_store=TableStoreDB(dbconn, "test_data", TEST_SCHEMA, True)
     )
 
     tbl.store_chunk(TEST_DF)
@@ -107,54 +104,23 @@ def test_store_less_values(dbconn) -> None:
     assert_datatable_equal(tbl, TEST_DF[:5])
 
 
-def test_get_process_ids(dbconn) -> None:
-    ds = DataStore(dbconn, create_meta_table=True)
-
-    tbl1 = ds.create_table(
-        'tbl1',
-        table_store=TableStoreDB(dbconn, 'tbl1_data', TEST_SCHEMA, True)
-    )
-    tbl2 = ds.create_table(
-        'tbl2',
-        table_store=TableStoreDB(dbconn, 'tbl2_data', TEST_SCHEMA, True)
-    )
-
-    tbl1.store_chunk(TEST_DF)
-
-    count, idx_dfs = ds.get_full_process_ids([tbl1], [tbl2])
-    idx = pd.concat(list(idx_dfs))
-
-    assert sorted(list(idx.index)) == list(TEST_DF.index)
-
-    tbl2.store_chunk(tbl1.get_data())
-
-    upd_df = TEST_DF[:5].copy()
-    upd_df['a'] += 1
-
-    tbl1.store_chunk(upd_df)
-
-    count, idx_dfs = ds.get_full_process_ids([tbl1], [tbl2])
-    idx = pd.concat(list(idx_dfs))
-
-    assert_df_equal(idx, upd_df[['id']])
-
-
 def test_store_chunk_changelist(dbconn) -> None:
     ds = DataStore(dbconn, create_meta_table=True)
 
     tbl = ds.create_table(
-        'tbl1',
-        table_store=TableStoreDB(dbconn, 'tbl1_data', TEST_SCHEMA, True)
+        "tbl1", table_store=TableStoreDB(dbconn, "tbl1_data", TEST_SCHEMA, True)
     )
 
     tbl.store_chunk(TEST_DF)
 
     upd_df = TEST_DF.copy()
 
-    upd_df.loc[1, 'a'] = 10
-    upd_df = pd.concat([upd_df, pd.DataFrame.from_records([{'id': 10, 'a': 11}])], axis='index')
+    upd_df.loc[1, "a"] = 10
+    upd_df = pd.concat(
+        [upd_df, pd.DataFrame.from_records([{"id": 10, "a": 11}])], axis="index"
+    )
 
-    proc_idx = IndexDF(upd_df[['id']])
+    proc_idx = IndexDF(upd_df[["id"]])
     idx = IndexDF(pd.DataFrame({"id": [0, 1, 10]}))
 
     change_idx = tbl.store_chunk(upd_df[1:], processed_idx=proc_idx)
@@ -166,8 +132,7 @@ def test_get_size(dbconn) -> None:
     ds = DataStore(dbconn, create_meta_table=True)
 
     tbl = ds.create_table(
-        'tbl1',
-        table_store=TableStoreDB(dbconn, 'tbl1_data', TEST_SCHEMA, True)
+        "tbl1", table_store=TableStoreDB(dbconn, "tbl1_data", TEST_SCHEMA, True)
     )
 
     tbl.store_chunk(TEST_DF)
diff --git a/tests/test_filtering.py b/tests/test_filtering.py
index 760c0353..bc12936b 100644
--- a/tests/test_filtering.py
+++ b/tests/test_filtering.py
@@ -1,141 +1,144 @@
 import pandas as pd
+import pytest
 from sqlalchemy import Column
-from sqlalchemy.sql.sqltypes import Integer, Boolean
-
-from datapipe.store.database import TableStoreDB
-from datapipe.datatable import DataStore
+from sqlalchemy.sql.sqltypes import Boolean, Integer
 
 from datapipe.compute import Catalog, Pipeline, Table, run_pipeline
 from datapipe.core_steps import BatchGenerate, BatchTransform
-from .util import assert_datatable_equal
+from datapipe.datatable import DataStore
+from datapipe.store.database import TableStoreDB
 
+from .util import assert_datatable_equal
 
 TEST_SCHEMA = [
-    Column('id1', Integer, primary_key=True),
-    Column('id2', Integer, primary_key=True),
-    Column('a', Integer),
+    Column("id1", Integer, primary_key=True),
+    Column("id2", Integer, primary_key=True),
+    Column("a", Integer),
 ]
 
 TEST_FINAL_SCHEMA = [
-    Column('id1', Integer, primary_key=True),
-    Column('a', Integer),
+    Column("id1", Integer, primary_key=True),
+    Column("a", Integer),
 ]
 
 TEST_FILTER_SCHEMA = [
-    Column('id1', Integer, primary_key=True),
-    Column('id2', Integer, primary_key=True),
-    Column('filter', Boolean),
+    Column("id1", Integer, primary_key=True),
+    Column("id2", Integer, primary_key=True),
+    Column("filter", Boolean),
 ]
 
 TEST_DF = pd.DataFrame(
     {
-        'id1': range(10),
-        'id2': [-i for i in range(10)],
-        'a': range(10),
+        "id1": range(10),
+        "id2": [-i for i in range(10)],
+        "a": range(10),
     },
 )
 
 
 def get_tbl_filter(df, value):
-    df['filter'] = df['a'] <= value
-    return df[['id1', 'id2', 'filter']]
+    df["filter"] = df["a"] <= value
+    return df[["id1", "id2", "filter"]]
 
 
 def get_tbl_final(df, df_filter):
-    df = pd.merge(df, df_filter, on=['id1', 'id2'])
-    df = df[df['filter']]
-    return df[['id1', 'id2', 'a']]
+    df = pd.merge(df, df_filter, on=["id1", "id2"])
+    df = df[df["filter"]]
+    return df[["id1", "id2", "a"]]
 
 
+@pytest.mark.xfail(reason="Пока не реализовано удаление таблицы")
 def test_delete_table_after_filter(dbconn):
-    catalog = Catalog({
-        'tbl': Table(
-            store=TableStoreDB(dbconn, 'tbl', TEST_SCHEMA, True)
-        ),
-        'tbl_filter': Table(
-            store=TableStoreDB(dbconn, 'tbl_filter', TEST_FILTER_SCHEMA, True)
-        ),
-        'tbl_final_id1_id2': Table(
-            store=TableStoreDB(dbconn, 'tbl_final_id1_id2', TEST_SCHEMA, True)
-        ),
-        'tbl_final_id1': Table(
-            store=TableStoreDB(dbconn, 'tbl_final_id1', TEST_FINAL_SCHEMA, True)
-        ),
-    })
+    catalog = Catalog(
+        {
+            "tbl": Table(store=TableStoreDB(dbconn, "tbl", TEST_SCHEMA, True)),
+            "tbl_filter": Table(
+                store=TableStoreDB(dbconn, "tbl_filter", TEST_FILTER_SCHEMA, True)
+            ),
+            "tbl_final_id1_id2": Table(
+                store=TableStoreDB(dbconn, "tbl_final_id1_id2", TEST_SCHEMA, True)
+            ),
+            "tbl_final_id1": Table(
+                store=TableStoreDB(dbconn, "tbl_final_id1", TEST_FINAL_SCHEMA, True)
+            ),
+        }
+    )
 
     def gen_tbl():
         yield TEST_DF
 
-    old_pipeline = Pipeline([
-        BatchGenerate(
-            func=gen_tbl,
-            outputs=['tbl'],
-        ),
-        BatchTransform(
-            func=get_tbl_filter,
-            inputs=['tbl'],
-            outputs=['tbl_filter'],
-            kwargs=dict(
-                value=10
-            )
-        ),
-        BatchTransform(
-            func=get_tbl_final,
-            inputs=['tbl', 'tbl_filter'],
-            outputs=['tbl_final_id1_id2'],
-        ),
-        BatchTransform(
-            func=lambda df: df[['id1', 'a']],
-            inputs=['tbl_final_id1_id2'],
-            outputs=['tbl_final_id1'],
-        ),
-    ])
+    old_pipeline = Pipeline(
+        [
+            BatchGenerate(
+                func=gen_tbl,
+                outputs=["tbl"],
+            ),
+            BatchTransform(
+                func=get_tbl_filter,
+                inputs=["tbl"],
+                outputs=["tbl_filter"],
+                kwargs=dict(value=10),
+            ),
+            BatchTransform(
+                func=get_tbl_final,
+                inputs=["tbl", "tbl_filter"],
+                outputs=["tbl_final_id1_id2"],
+            ),
+            BatchTransform(
+                func=lambda df: df[["id1", "a"]],
+                inputs=["tbl_final_id1_id2"],
+                outputs=["tbl_final_id1"],
+            ),
+        ]
+    )
 
     # Отличается от old_pipeline другой фильтрацией
-    new_pipeline = Pipeline([
-        BatchGenerate(
-            func=gen_tbl,
-            outputs=['tbl'],
-        ),
-        BatchTransform(
-            func=get_tbl_filter,
-            inputs=['tbl'],
-            outputs=['tbl_filter'],
-            kwargs=dict(
-                value=4
-            )
-        ),
-        BatchTransform(
-            func=get_tbl_final,
-            inputs=['tbl', 'tbl_filter'],
-            outputs=['tbl_final_id1_id2'],
-        ),
-        BatchTransform(
-            func=lambda df: df[['id1', 'a']],
-            inputs=['tbl_final_id1_id2'],
-            outputs=['tbl_final_id1'],
-        ),
-    ])
+    new_pipeline = Pipeline(
+        [
+            BatchGenerate(
+                func=gen_tbl,
+                outputs=["tbl"],
+            ),
+            BatchTransform(
+                func=get_tbl_filter,
+                inputs=["tbl"],
+                outputs=["tbl_filter"],
+                kwargs=dict(value=4),
+            ),
+            BatchTransform(
+                func=get_tbl_final,
+                inputs=["tbl", "tbl_filter"],
+                outputs=["tbl_final_id1_id2"],
+            ),
+            BatchTransform(
+                func=lambda df: df[["id1", "a"]],
+                inputs=["tbl_final_id1_id2"],
+                outputs=["tbl_final_id1"],
+            ),
+        ]
+    )
 
     ds = DataStore(dbconn, create_meta_table=True)
 
-    tbl_filter = catalog.get_datatable(ds, 'tbl_filter')
-    tbl_final_id1_id2 = catalog.get_datatable(ds, 'tbl_final_id1_id2')
-    tbl_final_id1 = catalog.get_datatable(ds, 'tbl_final_id1')
+    tbl_filter = catalog.get_datatable(ds, "tbl_filter")
+    tbl_final_id1_id2 = catalog.get_datatable(ds, "tbl_final_id1_id2")
+    tbl_final_id1 = catalog.get_datatable(ds, "tbl_final_id1")
 
     # Чистый Фильтр (10 значений)
     run_pipeline(ds, catalog, old_pipeline)
     assert len(tbl_final_id1_id2.get_data()) == 10
     assert len(tbl_final_id1.get_data()) == 10
     assert_datatable_equal(tbl_final_id1_id2, TEST_DF)
-    assert_datatable_equal(tbl_final_id1, TEST_DF.drop(columns=['id2']))
+    assert_datatable_equal(tbl_final_id1, TEST_DF.drop(columns=["id2"]))
 
     # Меняем пайплайн в одной трансформации -> делаем сброс метаданных на целевой таблице
     tbl_filter.reset_metadata()
 
     # Фильтр отсеивает 5 значений, поэтому они должны быть удалены из дальнейших таблиц
+    ds = DataStore(dbconn, create_meta_table=True)
+
     run_pipeline(ds, catalog, new_pipeline)
     assert len(tbl_final_id1_id2.get_data()) == 5
     assert len(tbl_final_id1.get_data()) == 5
     assert_datatable_equal(tbl_final_id1_id2, TEST_DF[:5])
-    assert_datatable_equal(tbl_final_id1, TEST_DF[:5].drop(columns=['id2']))
+    assert_datatable_equal(tbl_final_id1, TEST_DF[:5].drop(columns=["id2"]))
diff --git a/tests/test_image_pipeline.py b/tests/test_image_pipeline.py
index 3803dbd1..c7389d45 100644
--- a/tests/test_image_pipeline.py
+++ b/tests/test_image_pipeline.py
@@ -4,21 +4,34 @@
 import pandas as pd
 from PIL import Image
 
-from datapipe.compute import (Catalog, Pipeline, Table, build_compute,
-                              run_pipeline, run_steps)
-from datapipe.core_steps import (BatchGenerate, BatchTransform,
-                                 UpdateExternalTable, do_batch_generate,
-                                 do_full_batch_transform)
+from datapipe.compute import (
+    Catalog,
+    Pipeline,
+    Table,
+    build_compute,
+    run_pipeline,
+    run_steps,
+)
+from datapipe.core_steps import (
+    BatchGenerate,
+    BatchTransform,
+    BatchTransformStep,
+    UpdateExternalTable,
+    do_batch_generate,
+)
 from datapipe.datatable import DataStore
 from datapipe.store.filedir import PILFile, TableStoreFiledir
 
 
 def make_df():
-    idx = [f'im_{i}' for i in range(10)]
+    idx = [f"im_{i}" for i in range(10)]
     return pd.DataFrame(
         {
-            'id': idx,
-            'image': [Image.fromarray(np.random.randint(0, 256, (100, 100, 3)), 'RGB') for i in idx]
+            "id": idx,
+            "image": [
+                Image.fromarray(np.random.randint(0, 256, (100, 100, 3)), "RGB")
+                for i in idx
+            ],
         }
     )
 
@@ -28,7 +41,7 @@ def gen_images():
 
 
 def resize_images(df):
-    df['image'] = df['image'].apply(lambda im: im.resize((50, 50)))
+    df["image"] = df["image"].apply(lambda im: im.resize((50, 50)))
     return df
 
 
@@ -36,23 +49,21 @@ def test_image_datatables(dbconn, tmp_dir):
     ds = DataStore(dbconn, create_meta_table=True)
 
     tbl1 = ds.create_table(
-        'tbl1',
+        "tbl1",
         table_store=TableStoreFiledir(
-            tmp_dir / 'tbl1' / '{id}.png',
-            adapter=PILFile('png')
-        )
+            tmp_dir / "tbl1" / "{id}.png", adapter=PILFile("png")
+        ),
     )
 
     tbl2 = ds.create_table(
-        'tbl2',
+        "tbl2",
         table_store=TableStoreFiledir(
-            tmp_dir / 'tbl2' / '{id}.png',
-            adapter=PILFile('png')
-        )
+            tmp_dir / "tbl2" / "{id}.png", adapter=PILFile("png")
+        ),
     )
 
-    assert len(list(tmp_dir.glob('tbl1/*.png'))) == 0
-    assert len(list(tmp_dir.glob('tbl2/*.png'))) == 0
+    assert len(list(tmp_dir.glob("tbl1/*.png"))) == 0
+    assert len(list(tmp_dir.glob("tbl2/*.png"))) == 0
 
     do_batch_generate(
         func=gen_images,
@@ -60,111 +71,115 @@ def test_image_datatables(dbconn, tmp_dir):
         output_dts=[tbl1],
     )
 
-    do_full_batch_transform(
-        func=resize_images,
+    step = BatchTransformStep(
         ds=ds,
+        name="resize_images",
+        func=resize_images,
         input_dts=[tbl1],
         output_dts=[tbl2],
     )
 
-    assert len(list(tmp_dir.glob('tbl1/*.png'))) == 10
-    assert len(list(tmp_dir.glob('tbl2/*.png'))) == 10
+    step.run_full(ds)
+
+    assert len(list(tmp_dir.glob("tbl1/*.png"))) == 10
+    assert len(list(tmp_dir.glob("tbl2/*.png"))) == 10
 
 
 def test_image_pipeline(dbconn, tmp_dir):
-    catalog = Catalog({
-        'tbl1': Table(
-            store=TableStoreFiledir(
-                tmp_dir / 'tbl1' / '{id}.png',
-                adapter=PILFile('png')
-            )
-        ),
-        'tbl2': Table(
-            store=TableStoreFiledir(
-                tmp_dir / 'tbl2' / '{id}.png',
-                adapter=PILFile('png')
-            )
-        ),
-    })
+    catalog = Catalog(
+        {
+            "tbl1": Table(
+                store=TableStoreFiledir(
+                    tmp_dir / "tbl1" / "{id}.png", adapter=PILFile("png")
+                )
+            ),
+            "tbl2": Table(
+                store=TableStoreFiledir(
+                    tmp_dir / "tbl2" / "{id}.png", adapter=PILFile("png")
+                )
+            ),
+        }
+    )
 
-    pipeline = Pipeline([
-        BatchGenerate(
-            gen_images,
-            outputs=['tbl1'],
-        ),
-        BatchTransform(
-            resize_images,
-            inputs=['tbl1'],
-            outputs=['tbl2'],
-        )
-    ])
+    pipeline = Pipeline(
+        [
+            BatchGenerate(
+                gen_images,
+                outputs=["tbl1"],
+            ),
+            BatchTransform(
+                resize_images,
+                inputs=["tbl1"],
+                outputs=["tbl2"],
+            ),
+        ]
+    )
 
-    assert len(list(tmp_dir.glob('tbl1/*.png'))) == 0
-    assert len(list(tmp_dir.glob('tbl2/*.png'))) == 0
+    assert len(list(tmp_dir.glob("tbl1/*.png"))) == 0
+    assert len(list(tmp_dir.glob("tbl2/*.png"))) == 0
 
     ds = DataStore(dbconn, create_meta_table=True)
     run_pipeline(ds, catalog, pipeline)
 
-    assert len(list(tmp_dir.glob('tbl1/*.png'))) == 10
-    assert len(list(tmp_dir.glob('tbl2/*.png'))) == 10
+    assert len(list(tmp_dir.glob("tbl1/*.png"))) == 10
+    assert len(list(tmp_dir.glob("tbl2/*.png"))) == 10
 
 
 def test_image_batch_generate_with_later_deleting(dbconn, tmp_dir):
-
     # Add images to tmp_dir
     df_images = make_df()
-    (tmp_dir / 'tbl1').mkdir()
-    for _, row in df_images[['id', 'image']].iterrows():
-        row['image'].save(tmp_dir / 'tbl1' / f'{row["id"]}.png')
-
-    catalog = Catalog({
-        'tbl1': Table(
-            store=TableStoreFiledir(
-                tmp_dir / 'tbl1' / '{id}.png',
-                adapter=PILFile('png'),
-                enable_rm=True
-            )
-        ),
-        'tbl2': Table(
-            store=TableStoreFiledir(
-                tmp_dir / 'tbl2' / '{id}.png',
-                adapter=PILFile('png'),
-                enable_rm=True
-            )
-        ),
-    })
+    (tmp_dir / "tbl1").mkdir()
+    for _, row in df_images[["id", "image"]].iterrows():
+        row["image"].save(tmp_dir / "tbl1" / f'{row["id"]}.png')
 
-    pipeline = Pipeline([
-        UpdateExternalTable("tbl1"),
-        BatchTransform(
-            lambda df: df,
-            inputs=["tbl1"],
-            outputs=["tbl2"]
-        )
-    ])
+    catalog = Catalog(
+        {
+            "tbl1": Table(
+                store=TableStoreFiledir(
+                    tmp_dir / "tbl1" / "{id}.png",
+                    adapter=PILFile("png"),
+                    enable_rm=True,
+                )
+            ),
+            "tbl2": Table(
+                store=TableStoreFiledir(
+                    tmp_dir / "tbl2" / "{id}.png",
+                    adapter=PILFile("png"),
+                    enable_rm=True,
+                )
+            ),
+        }
+    )
+
+    pipeline = Pipeline(
+        [
+            UpdateExternalTable("tbl1"),
+            BatchTransform(lambda df: df, inputs=["tbl1"], outputs=["tbl2"]),
+        ]
+    )
 
-    assert len(list(tmp_dir.glob('tbl1/*.png'))) == 10
-    assert len(list(tmp_dir.glob('tbl2/*.png'))) == 0
+    assert len(list(tmp_dir.glob("tbl1/*.png"))) == 10
+    assert len(list(tmp_dir.glob("tbl2/*.png"))) == 0
 
     ds = DataStore(dbconn, create_meta_table=True)
     steps = build_compute(ds, catalog, pipeline)
     run_steps(ds, steps)
 
-    assert len(list(tmp_dir.glob('tbl1/*.png'))) == 10
-    assert len(list(tmp_dir.glob('tbl2/*.png'))) == 10
-    assert len(catalog.get_datatable(ds, 'tbl1').get_data()) == 10
-    assert len(catalog.get_datatable(ds, 'tbl2').get_data()) == 10
+    assert len(list(tmp_dir.glob("tbl1/*.png"))) == 10
+    assert len(list(tmp_dir.glob("tbl2/*.png"))) == 10
+    assert len(catalog.get_datatable(ds, "tbl1").get_data()) == 10
+    assert len(catalog.get_datatable(ds, "tbl2").get_data()) == 10
 
     # Delete some files from the folder
     for id in [0, 5, 7, 8, 9]:
-        (tmp_dir / 'tbl1' / f'im_{id}.png').unlink()
+        (tmp_dir / "tbl1" / f"im_{id}.png").unlink()
 
     run_steps(ds, steps)
 
-    assert len(list(tmp_dir.glob('tbl1/*.png'))) == 5
-    assert len(catalog.get_datatable(ds, 'tbl1').get_data()) == 5
-    assert len(catalog.get_datatable(ds, 'tbl1').get_metadata()) == 5
+    assert len(list(tmp_dir.glob("tbl1/*.png"))) == 5
+    assert len(catalog.get_datatable(ds, "tbl1").get_data()) == 5
+    assert len(catalog.get_datatable(ds, "tbl1").get_metadata()) == 5
 
-    assert len(list(tmp_dir.glob('tbl2/*.png'))) == 5
-    assert len(catalog.get_datatable(ds, 'tbl2').get_data()) == 5
-    assert len(catalog.get_datatable(ds, 'tbl2').get_metadata()) == 5
+    assert len(list(tmp_dir.glob("tbl2/*.png"))) == 5
+    assert len(catalog.get_datatable(ds, "tbl2").get_data()) == 5
+    assert len(catalog.get_datatable(ds, "tbl2").get_metadata()) == 5
diff --git a/tests/test_meta_info_in_datapipe_events.py b/tests/test_meta_info_in_datapipe_events.py
index a2890560..bf0a1e5b 100644
--- a/tests/test_meta_info_in_datapipe_events.py
+++ b/tests/test_meta_info_in_datapipe_events.py
@@ -10,80 +10,74 @@
 from datapipe.store.database import TableStoreDB
 
 TEST_SCHEMA = [
-    Column('pipeline_id', Integer(), primary_key=True),
-    Column('offer_id', Integer(), primary_key=True),
-    Column('test_field', JSON)
+    Column("pipeline_id", Integer(), primary_key=True),
+    Column("offer_id", Integer(), primary_key=True),
+    Column("test_field", JSON),
 ]
 
 
 def generate_data(value: int):
-    df_data = [{
-        "pipeline_id": value,
-        "offer_id": value,
-        "test_field": {"a": value}
-    }]
+    df_data = [{"pipeline_id": value, "offer_id": value, "test_field": {"a": value}}]
     yield pd.DataFrame(data=df_data)
 
 
 def update_data(df: pd.DataFrame, value: int) -> pd.DataFrame:
     df["test_field"].apply(lambda x: {**x, "b": value})
-    df.index = df.index.astype('str')
+    df.index = df.index.astype("str")
     return df
 
 
 def test_meta_info_in_datapipe_events(dbconn) -> None:
     ds = DataStore(dbconn, create_meta_table=True)
 
-    run_config = RunConfig(
-        filters={
-            "pipeline_id": 1
-        },
-        labels={
-            "pipeline_name": 'test_name',
-            "pipeline_id": 1
+    run_config = RunConfig(filters={"pipeline_id": 1}, labels={"pipeline_name": "test_name", "pipeline_id": 1})
+
+    catalog = Catalog(
+        {
+            "test_generate": Table(
+                store=TableStoreDB(
+                    dbconn,
+                    "test_generate_data",
+                    TEST_SCHEMA,
+                    create_table=True,
+                )
+            ),
+            "test_transform": Table(
+                store=TableStoreDB(
+                    dbconn,
+                    "test_transform_data",
+                    TEST_SCHEMA,
+                    create_table=True,
+                )
+            ),
         }
     )
 
-    catalog = Catalog({
-        'test_generate': Table(
-            store=TableStoreDB(
-                dbconn,
-                'test_generate_data',
-                TEST_SCHEMA,
-                create_table=True,
-            )
-        ),
-        'test_transform': Table(
-            store=TableStoreDB(
-                dbconn,
-                'test_transform_data',
-                TEST_SCHEMA,
-                create_table=True,
-            )
-        )
-    })
-
-    pipeline = Pipeline([
-        BatchGenerate(
-            generate_data,
-            outputs=["test_generate"],
-            kwargs=dict(
-                value=1,
+    pipeline = Pipeline(
+        [
+            BatchGenerate(
+                generate_data,
+                outputs=["test_generate"],
+                kwargs=dict(
+                    value=1,
+                ),
             ),
-        ),
-        BatchTransform(
-            update_data,
-            inputs=["test_generate"],
-            outputs=["test_transform"],
-            kwargs=dict(
-                value=2,
+            BatchTransform(
+                update_data,
+                inputs=["test_generate"],
+                outputs=["test_transform"],
+                kwargs=dict(
+                    value=2,
+                ),
             ),
-        )
-    ])
+        ]
+    )
 
     run_pipeline(ds, catalog, pipeline, run_config)
 
-    df_events = pd.read_sql_query(select(catalog.get_datatable(ds, 'test_generate').event_logger.events_table), dbconn.con)
+    df_events = pd.read_sql_query(
+        select(catalog.get_datatable(ds, "test_generate").event_logger.events_table), dbconn.con
+    )
 
     assert df_events.loc[0]["event"] == {
         "meta": {
@@ -94,15 +88,15 @@ def test_meta_info_in_datapipe_events(dbconn) -> None:
             },
             "filters": {
                 "pipeline_id": 1,
-            }
+            },
         },
         "data": {
             "table_name": "test_generate",
             "added_count": 1,
             "updated_count": 0,
             "deleted_count": 0,
-            "processed_count": 1
-        }
+            "processed_count": 1,
+        },
     }
 
     assert df_events.loc[1]["event"] == {
@@ -114,13 +108,13 @@ def test_meta_info_in_datapipe_events(dbconn) -> None:
             },
             "filters": {
                 "pipeline_id": 1,
-            }
+            },
         },
         "data": {
             "table_name": "test_transform",
             "added_count": 1,
             "updated_count": 0,
             "deleted_count": 0,
-            "processed_count": 1
-        }
+            "processed_count": 1,
+        },
     }
diff --git a/tests/test_metatable.py b/tests/test_metatable.py
index 53e0772b..b7923b88 100644
--- a/tests/test_metatable.py
+++ b/tests/test_metatable.py
@@ -6,7 +6,7 @@
 
 from datapipe.metastore import MetaTable
 from datapipe.store.database import DBConn, MetaKey
-from datapipe.types import DataDF, DataSchema, MetaSchema
+from datapipe.types import DataDF, DataSchema, IndexDF, MetaSchema
 
 import pytest
 from pytest_cases import parametrize_with_cases, parametrize
@@ -14,106 +14,91 @@
 
 
 class CasesTestDF:
-    @parametrize('N', [pytest.param(N) for N in [10, 100, 1000]])
+    @parametrize("N", [pytest.param(N) for N in [10, 100, 1000]])
     def case_single_idx(self, N):
         return (
-            ['id'],
+            ["id"],
             [
-                Column('id', Integer, primary_key=True),
+                Column("id", Integer, primary_key=True),
             ],
             [],
-            cast(DataDF, pd.DataFrame(
-                {
-                    'id': range(N),
-                    'a': range(N)
-                },
-            ))
+            cast(
+                DataDF,
+                pd.DataFrame(
+                    {"id": range(N), "a": range(N)},
+                ),
+            ),
         )
 
-    @parametrize('N', [pytest.param(N) for N in [10, 100, 1000]])
+    @parametrize("N", [pytest.param(N) for N in [10, 100, 1000]])
     def case_single_idx_with_meta(self, N):
         return (
-            ['id'],
+            ["id"],
             [
-                Column('id', Integer, primary_key=True),
+                Column("id", Integer, primary_key=True),
             ],
             [
-                Column('item_id', Integer, MetaKey()),
+                Column("item_id", Integer, MetaKey()),
             ],
-            cast(DataDF, pd.DataFrame(
-                {
-                    'id': range(N),
-                    'item_id': range(N),
-                    'a': range(N)
-                },
-            ))
+            cast(
+                DataDF,
+                pd.DataFrame(
+                    {"id": range(N), "item_id": range(N), "a": range(N)},
+                ),
+            ),
         )
 
-    @parametrize('N', [pytest.param(N) for N in [10, 100, 1000]])
+    @parametrize("N", [pytest.param(N) for N in [10, 100, 1000]])
     def case_multi_idx(self, N):
         return (
-            ['id1', 'id2'],
-            [
-                Column('id1', Integer, primary_key=True),
-                Column('id2', Integer, primary_key=True)
-            ],
+            ["id1", "id2"],
+            [Column("id1", Integer, primary_key=True), Column("id2", Integer, primary_key=True)],
             [],
-            cast(DataDF, pd.DataFrame(
-                {
-                    'id1': range(N),
-                    'id2': range(N),
-                    'a': range(N)
-                },
-            ))
+            cast(
+                DataDF,
+                pd.DataFrame(
+                    {"id1": range(N), "id2": range(N), "a": range(N)},
+                ),
+            ),
         )
 
-    @parametrize('N', [pytest.param(N) for N in [10, 100, 1000]])
+    @parametrize("N", [pytest.param(N) for N in [10, 100, 1000]])
     def case_multi_idx_with_meta(self, N):
         return (
-            ['id1', 'id2'],
-            [
-                Column('id1', Integer, primary_key=True),
-                Column('id2', Integer, primary_key=True)
-            ],
+            ["id1", "id2"],
+            [Column("id1", Integer, primary_key=True), Column("id2", Integer, primary_key=True)],
             [
-                Column('item_id', Integer, MetaKey()),
-                Column('product_id', Integer, MetaKey()),
+                Column("item_id", Integer, MetaKey()),
+                Column("product_id", Integer, MetaKey()),
             ],
-            cast(DataDF, pd.DataFrame(
-                {
-                    'id1': range(N),
-                    'id2': range(N),
-                    'item_id': range(N),
-                    'product_id': range(N),
-                    'a': range(N)
-                },
-            ))
+            cast(
+                DataDF,
+                pd.DataFrame(
+                    {"id1": range(N), "id2": range(N), "item_id": range(N), "product_id": range(N), "a": range(N)},
+                ),
+            ),
         )
 
 
-@parametrize_with_cases('index_cols,primary_schema,meta_schema,test_df', cases=CasesTestDF, import_fixtures=True)
+@parametrize_with_cases("index_cols,primary_schema,meta_schema,test_df", cases=CasesTestDF, import_fixtures=True)
 def test_insert_rows(
-    dbconn: DBConn,
-    index_cols: List[str],
-    primary_schema: DataSchema,
-    meta_schema: MetaSchema,
-    test_df: DataDF
+    dbconn: DBConn, index_cols: List[str], primary_schema: DataSchema, meta_schema: MetaSchema, test_df: DataDF
 ):
     mt = MetaTable(
-        name='test',
+        name="test",
         dbconn=dbconn,
         primary_schema=primary_schema,
         meta_schema=meta_schema,
         create_table=True,
     )
-    keys = set(mt.primary_keys) | set(mt.meta_keys.keys())
+    keys = list(set(mt.primary_keys) | set(mt.meta_keys.keys()))
 
     new_df, changed_df, new_meta_df, changed_meta_df = mt.get_changes_for_store_chunk(test_df)
     assert_df_equal(new_df, test_df, index_cols=index_cols)
-    assert(len(new_df) == len(test_df))
-    assert(len(new_meta_df) == len(test_df))
-    assert(len(changed_df) == 0)
-    assert(len(changed_meta_df) == 0)
+    assert len(new_df) == len(test_df)
+    assert len(new_meta_df) == len(test_df)
+    assert len(changed_df) == 0
+    assert len(changed_meta_df) == 0
 
     assert_df_equal(new_meta_df[index_cols], new_df[index_cols], index_cols=index_cols)
     assert_df_equal(new_meta_df[keys], new_df[keys], index_cols=index_cols)
@@ -126,19 +111,15 @@ def test_insert_rows(
     assert_df_equal(meta_df[index_cols], test_df[index_cols], index_cols=index_cols)
     assert_df_equal(meta_df[keys], test_df[keys], index_cols=index_cols)
 
-    assert(not meta_df['hash'].isna().any())
+    assert not meta_df["hash"].isna().any()
 
 
-@parametrize_with_cases('index_cols,primary_schema,meta_schema,test_df', cases=CasesTestDF, import_fixtures=True)
+@parametrize_with_cases("index_cols,primary_schema,meta_schema,test_df", cases=CasesTestDF, import_fixtures=True)
 def test_get_metadata(
-    dbconn: DBConn,
-    index_cols: List[str],
-    primary_schema: DataSchema,
-    meta_schema: MetaSchema,
-    test_df: DataDF
+    dbconn: DBConn, index_cols: List[str], primary_schema: DataSchema, meta_schema: MetaSchema, test_df: DataDF
 ):
     mt = MetaTable(
-        name='test',
+        name="test",
         dbconn=dbconn,
         primary_schema=primary_schema,
         meta_schema=meta_schema,
@@ -152,14 +133,6 @@ def test_get_metadata(
     part_idx = part_df[index_cols]
     keys = list(set(mt.primary_keys) | set(mt.meta_keys.keys()))
 
-    assert_df_equal(
-        mt.get_metadata(part_idx)[index_cols],
-        part_idx,
-        index_cols=index_cols
-    )
+    assert_df_equal(mt.get_metadata(cast(IndexDF, part_idx))[index_cols], part_idx, index_cols=index_cols)
 
-    assert_df_equal(
-        mt.get_metadata(part_idx)[keys],
-        part_df[keys],
-        index_cols=index_cols
-    )
+    assert_df_equal(mt.get_metadata(cast(IndexDF, part_idx))[keys], part_df[keys], index_cols=index_cols)
diff --git a/tests/test_table_store.py b/tests/test_table_store.py
index 86b07f74..c0a0d84c 100644
--- a/tests/test_table_store.py
+++ b/tests/test_table_store.py
@@ -318,9 +318,7 @@ def case_filedir_json(self, tmp_dir, df, fn_template, primary_schema):
         )
 
 
-@parametrize_with_cases(
-    "store,test_df", cases=CasesTableStore, has_tag="supports_get_schema"
-)
+@parametrize_with_cases("store,test_df", cases=CasesTableStore, has_tag="supports_get_schema")
 def test_get_schema(store: TableStore, test_df: pd.DataFrame) -> None:
     store.get_schema()
 
@@ -332,9 +330,7 @@ def test_write_read_rows(store: TableStore, test_df: pd.DataFrame) -> None:
     assert_ts_contains(store, test_df)
 
 
-@parametrize_with_cases(
-    "store,test_df", cases=CasesTableStore, has_tag="supports_read_all_rows"
-)
+@parametrize_with_cases("store,test_df", cases=CasesTableStore, has_tag="supports_read_all_rows")
 def test_write_read_full_rows(store: TableStore, test_df: pd.DataFrame) -> None:
     store.insert_rows(test_df)
 
@@ -342,9 +338,7 @@ def test_write_read_full_rows(store: TableStore, test_df: pd.DataFrame) -> None:
 
 
 @parametrize_with_cases("store,test_df", cases=CasesTableStore)
-def test_insert_identical_rows_twice_and_read_rows(
-    store: TableStore, test_df: pd.DataFrame
-) -> None:
+def test_insert_identical_rows_twice_and_read_rows(store: TableStore, test_df: pd.DataFrame) -> None:
     store.insert_rows(test_df)
 
     test_df_mod = test_df.copy()
@@ -355,9 +349,7 @@ def test_insert_identical_rows_twice_and_read_rows(
     assert_ts_contains(store, test_df_mod)
 
 
-@parametrize_with_cases(
-    "store,test_df", cases=CasesTableStore, has_tag="supports_read_nonexistent_rows"
-)
+@parametrize_with_cases("store,test_df", cases=CasesTableStore, has_tag="supports_read_nonexistent_rows")
 def test_read_non_existent_rows(store: TableStore, test_df: pd.DataFrame) -> None:
     test_df_to_store = test_df.drop(range(1, 5))
 
@@ -381,9 +373,7 @@ def test_read_empty_df(store: TableStore, test_df: pd.DataFrame) -> None:
     df_result[store.primary_keys]  # Empty df must have primary keys columns
 
 
-@parametrize_with_cases(
-    "store,test_df", cases=CasesTableStore, has_tag="supports_read_all_rows"
-)
+@parametrize_with_cases("store,test_df", cases=CasesTableStore, has_tag="supports_read_all_rows")
 def test_insert_empty_df(store: TableStore, test_df: pd.DataFrame) -> None:
     df_empty = pd.DataFrame()
     store.insert_rows(df_empty)
@@ -393,9 +383,7 @@ def test_insert_empty_df(store: TableStore, test_df: pd.DataFrame) -> None:
     df_result[store.primary_keys]  # Empty df must have primary keys columns
 
 
-@parametrize_with_cases(
-    "store,test_df", cases=CasesTableStore, has_tag="supports_read_all_rows"
-)
+@parametrize_with_cases("store,test_df", cases=CasesTableStore, has_tag="supports_read_all_rows")
 def test_update_empty_df(store: TableStore, test_df: pd.DataFrame) -> None:
     df_empty = pd.DataFrame()
     store.update_rows(df_empty)
@@ -447,7 +435,7 @@ def test_delete_rows(store: TableStore, test_df: pd.DataFrame) -> None:
         index_cols=store.primary_keys,
     )
 
-    store.delete_rows(test_df.loc[20:50, store.primary_keys])
+    store.delete_rows(cast(IndexDF, test_df.loc[20:50, store.primary_keys]))
 
     assert_df_equal(
         store.read_rows(),
@@ -456,9 +444,7 @@ def test_delete_rows(store: TableStore, test_df: pd.DataFrame) -> None:
     )
 
 
-@parametrize_with_cases(
-    "store,test_df", cases=CasesTableStore, has_tag="supports_read_meta_pseudo_df"
-)
+@parametrize_with_cases("store,test_df", cases=CasesTableStore, has_tag="supports_read_meta_pseudo_df")
 def test_read_rows_meta_pseudo_df(store: TableStore, test_df: pd.DataFrame) -> None:
     store.insert_rows(test_df)
 
@@ -470,12 +456,8 @@ def test_read_rows_meta_pseudo_df(store: TableStore, test_df: pd.DataFrame) -> N
     assert isinstance(next(pseudo_df_iter), DataDF)
 
 
-@parametrize_with_cases(
-    "store,test_df", cases=CasesTableStore, has_tag="supports_read_meta_pseudo_df"
-)
-def test_read_empty_rows_meta_pseudo_df(
-    store: TableStore, test_df: pd.DataFrame
-) -> None:
+@parametrize_with_cases("store,test_df", cases=CasesTableStore, has_tag="supports_read_meta_pseudo_df")
+def test_read_empty_rows_meta_pseudo_df(store: TableStore, test_df: pd.DataFrame) -> None:
     pseudo_df_iter = store.read_rows_meta_pseudo_df()
     assert isinstance(pseudo_df_iter, Iterable)
     for pseudo_df in pseudo_df_iter:
@@ -483,20 +465,14 @@ def test_read_empty_rows_meta_pseudo_df(
         pseudo_df[store.primary_keys]  # Empty df must have primary keys columns
 
 
-@parametrize_with_cases(
-    "store,test_df", cases=CasesTableStore, has_tag="supports_read_meta_pseudo_df"
-)
-def test_read_rows_meta_pseudo_df_with_runconfig(
-    store: TableStore, test_df: pd.DataFrame
-) -> None:
+@parametrize_with_cases("store,test_df", cases=CasesTableStore, has_tag="supports_read_meta_pseudo_df")
+def test_read_rows_meta_pseudo_df_with_runconfig(store: TableStore, test_df: pd.DataFrame) -> None:
     store.insert_rows(test_df)
 
     assert_ts_contains(store, test_df)
 
     # TODO проверять, что runconfig реально влияет на результирующие данные
-    pseudo_df_iter = store.read_rows_meta_pseudo_df(
-        run_config=RunConfig(filters={"a": 1})
-    )
+    pseudo_df_iter = store.read_rows_meta_pseudo_df(run_config=RunConfig(filters={"a": 1}))
     assert isinstance(pseudo_df_iter, Iterable)
     for pseudo_df in pseudo_df_iter:
         assert isinstance(pseudo_df, DataDF)
diff --git a/tests/test_transform_meta.py b/tests/test_transform_meta.py
new file mode 100644
index 00000000..e5cee0fd
--- /dev/null
+++ b/tests/test_transform_meta.py
@@ -0,0 +1,105 @@
+from typing import List
+
+import pytest
+from pytest_cases import parametrize
+from sqlalchemy import Column, Integer
+
+from datapipe.core_steps import BatchTransformStep
+from datapipe.metastore import MetaTable
+from datapipe.store.database import DBConn
+from datapipe.types import MetaSchema
+
+
+def make_mt(name, dbconn, schema_keys) -> MetaTable:
+    return MetaTable(
+        dbconn=dbconn,
+        name=name,
+        primary_schema=[
+            Column(key, Integer(), primary_key=True) for key in schema_keys
+        ],
+    )
+
+
+def assert_schema_equals(sch: MetaSchema, keys: List[str]):
+    assert sorted(col.name for col in sch) == sorted(keys)
+
+
+SUCCESS_CASES = [
+    (
+        [["a", "b"]],
+        [["a", "b"]],
+        None,
+        ["a", "b"],
+    ),
+    (
+        [["a", "b"]],
+        [["x", "b"]],
+        None,
+        ["b"],
+    ),
+    (
+        [["a", "b"]],
+        [["a", "b"]],
+        ["a"],
+        ["a"],
+    ),
+    (
+        [["bbox_id"], ["model_id"]],
+        [["bbox_id", "model_id"]],
+        ["bbox_id", "model_id"],
+        ["bbox_id", "model_id"],
+    ),
+]
+
+
+FAIL_CASES = [
+    (
+        [["a", "b"]],
+        [["x", "y"]],
+        None,
+    ),
+]
+
+
+@parametrize(
+    "input_keys_list,output_keys_list,transform_keys,expected_keys", SUCCESS_CASES
+)
+def test_compute_transform_schema_success(
+    dbconn: DBConn,
+    input_keys_list,
+    output_keys_list,
+    transform_keys,
+    expected_keys,
+):
+    inp_mts = [
+        make_mt(f"inp_{i}", dbconn, keys) for (i, keys) in enumerate(input_keys_list)
+    ]
+    out_mts = [
+        make_mt(f"out_{i}", dbconn, keys) for (i, keys) in enumerate(output_keys_list)
+    ]
+
+    _, sch = BatchTransformStep.compute_transform_schema(
+        inp_mts, out_mts, transform_keys=transform_keys
+    )
+
+    assert_schema_equals(sch, expected_keys)
+
+
+@parametrize("input_keys_list,output_keys_list,transform_keys", FAIL_CASES)
+def test_compute_transform_schema_fail(
+    dbconn: DBConn,
+    input_keys_list,
+    output_keys_list,
+    transform_keys,
+):
+    inp_mts = [
+        make_mt(f"inp_{i}", dbconn, keys) for (i, keys) in enumerate(input_keys_list)
+    ]
+    out_mts = [
+        make_mt(f"out_{i}", dbconn, keys) for (i, keys) in enumerate(output_keys_list)
+    ]
+
+    with pytest.raises(AssertionError):
+        BatchTransformStep.compute_transform_schema(
+            inp_mts, out_mts, transform_keys=transform_keys
+        )