Repository: kedro-org/kedro

Issue:
Eliminate the need of having all datasets defined in the catalog
## Description
Users struggle with ever increasing catalog files:
https://github.com/kedro-org/kedro/issues/891
https://github.com/kedro-org/kedro/issues/1847

Current "solutions":
- Use yaml anchors: https://www.educative.io/blog/advanced-yaml-syntax-cheatsheet#anchors
- The `TemplatedConfigLoader`
- Use the `TemplatedConfigLoader` with Jinja2: https://docs.kedro.org/en/stable/kedro_project_setup/configuration.html#jinja2-support

Instead of allowing anchors or loops in config let's just tackle the need of having so many datasets defined in the catalog in the first place. 

## Context
The new `OmegaConfigLoader` doesn't currently allow the use of yaml anchors (the omegaconf team discourages people to use this with omegaconf: https://github.com/omry/omegaconf/issues/93) or Jinja2. While Jinja2 is very powerful, we've always had the standpoint as a team that it's better to not use it, because it makes files hard to read and reason about. Jinja2 is meant for HTML templating, not for configuration. 

In order to make `OmegaConfigLoader` a suitable replacement for the `ConfigLoader` and `TemplatedConfigLoader` we need to find another solution to reduce the size of catalog files. 

## Possible Implementation
This is an idea from @idanov (written up by me, so I hope I have interpreted it correctly 😅 ) :

In every runner we have a method to create a default dataset for datasets that are used in the pipeline but not defined in the catalog:

https://github.com/kedro-org/kedro/blob/8acfb2a67049cce8f8d9126fc2373cb40e26b0d8/kedro/runner/sequential_runner.py#L32-L43

We can allow users to define what a "default" dataset should be so that in the catalog they only need to specify these default settings and Kedro would then create those without needing them to be defined in the catalog. The default dataset definition could then look like something like this:

```yaml
default_dataset: 
  type: spark.SparkDataSet
  file_format: parquet
  load_args:
    header: true
  save_args:
    mode: overwrite
```

And any dataset used in the pipeline, but not mentioned in the catalog, would get the above specifications. 

### Extra info
I collected several catalog files from internal teams to see what patterns I could find in there. I think these will be useful when designing our solution. https://miro.com/app/board/uXjVMZM7IFY=/?share_link_id=288914952503


Discussion:
Would you only allow a singular `default_dataset`?
> Would you only allow a singular `default_dataset`?

I think that's up for discussion. I'm going to go through config files and see what patterns I see and how this proposal would help. 
> We can allow users to define what a "default" dataset should be so that in the catalog they only need to specify these default settings and Kedro would then create those without needing them to be defined in the catalog.

Non-memory datasets need more information, like where you're going to store datasets and with what name. For example, Kubeflow Pipelines has the idea of `pipeline_root` ("The root path where this pipeline’s outputs are stored. This can be a MinIO, Google Cloud Storage, or Amazon Web Services S3 URI. You can override the pipeline root when you run the pipeline.") But if users want control over specifically what their file name is, they don't, and instead it becomes something like the catalog entry name.
@datajoely Having the option to have more than one default datasets is also on the table, e.g. one can imagine a pattern matching solution where if your dataset name starts with `xxxxx`, it creates `MemoryDataSet`, and if it starts with `yyyyy`, it creates `SparkDataSet`. This should ultimately be configurable in `settings.py` and potentially removed from the runner (although different runners will need different default datasets, e.g. `ParallelRunner` vs `SequentialRunner`) and moved to the `DataCatalog` as responsibility.

Currently we have  `CONFIG_LOADER_CLASS` and `CONFIG_LOADER_ARGS`, but we only have `DATA_CATALOG_CLASS` in the `settings.py`. We could get inspired from what we have in the `OmegaConfLoader` to specify file patterns for different config keys: 

https://github.com/kedro-org/kedro/blob/8acfb2a67049cce8f8d9126fc2373cb40e26b0d8/kedro/config/omegaconf_config.py#L102-L107
 
And allow a similar solution for our "default" datasets:

```python
# % settings.py

def create_spark_dataset(dataset_name: str, *chunks):
    # e.g. here chunks=["root_namespace", "something-instead-the-*", "spark"]
    return  SparkDataSet(filepath=f"data/{chunks[0]}/{chunks[1]}.parquet", file_format="parquet")

def create_memory_dataset(dataset_name: str, *chunks):
    return MemoryDataSet()

DATA_CATALOG_ARGS = {
    "datasets_factories": {
        "root_namespace.*@spark": create_spark_dataset,
        "root_namespace.*@memory": create_memory_dataset, 
    }
}
```

So when a dataset name matching some of these patterns is provided, the dataset is created accordingly. We might not necessarily have all of that in code, but it could reside somewhere in `catalog.yaml`, since you can have almost anything as key in yaml, if surrounded by "", e.g.:

```yaml
"{root_namespace}.{*}@{spark}":
  type: spark.SparkDataSet
  filepath: data/{chunks[0]}/{chunks[1]}.parquet
  file_format: parquet
```

We also need to decide on a pattern-matching language, simpler than regex, but powerful enough to be able to chunk up the dataset names and those chunks to be reused in the body of the configuration for the reasons pointed out by @deepyaman. The using of the chunks in the body could be done through either f-string replacement as in the above `yaml` example, `OmegaConf` resolvers or something else entirely (depending where it will happen, because we don't want to add dependencies on `OmegaConf` in the `DataCatalog`).

The `pipeline_root` problem is the easiest to solve, since it can be an environment variable, which can be included through `OmegaConf`, another `DATA_CATALOG_ARGS` key or something of the sorts.
```yaml
"{root_namespace}.{*}@{spark}":
  type: spark.SparkDataSet
  filepath: data/{chunks[0]}/{chunks[1]}.parquet
  file_format: parquet
```

@idanov  - this is a work of art, I'm sold.

What I would ask is that we consider introducing an [equivalent of](https://docs.getdbt.com/reference/commands/compile) `dbt compile` so the users have a mechanism for reviewing what is rendered at runtime versus what they configure at rest.
@AntonyMilneQB shared a nice alternative pattern-matching syntax here, which uses reverse Python f-strings format, which is definitely going to do the job and help us avoid creating a homebrew patter-matching language: https://github.com/r1chardj0n3s/parse

@datajoely we could do something like that, inspired from what @WaylonWalker created sometime ago: https://github.com/WaylonWalker/kedro-auto-catalog

Another option is a CLI command giving you the dataset definition per dataset name, something like `kedro catalog get <dataset-name-of-interest>`.

Just reading through this, so there are two things discussed here:
- Provide a flexible way to allow users to replace default datasets - I have seen this and am glad it is mentioned here
  - https://github.com/getindata/kedro-sagemaker/blob/develop/kedro_sagemaker/runner.py#L41-L67 This could be an example (1)
- A pattern-matching solution for replacing the YAML anchor (2)

What would the catalog looks like for (2) `catalog.yml`? Do the user still need to put all entry in the catalog (but shorter because you only define the top-level key), or do user still need `Jinja` to create entry dynamically (say a list of 1000 datasets)

Before we get carried away with using [`parse`](https://github.com/r1chardj0n3s/parse) to do the pattern matching we should definitely do some careful thinking about how this would work in practice. It seems like a great solution in theory (it's very lightweight and simple, gives you named chunks, everyone who knows Python should know the mini-language already), but when I've used it before it was a bit tricky to get right. Here's what the above example would look like:

```
"{root_namespace}.{dataset_name}@spark":
  type: spark.SparkDataSet
  filepath: data/{root_namespace}/{dataset_name}.parquet
  file_format: parquet

# would parse as follows
germany.companies@spark:
  type: spark.SparkDataSet
  filepath: data/germany/companies.parquet
  file_format: parquet
```

All good so far, but what happens in this case?
```
"{dataset_name}_{layer}":
  type: spark.SparkDataSet
  filepath: data/{root_namespace}/{dataset_name}.parquet
  layer: raw

# would parse as follows
big_companies_raw:
  type: spark.SparkDataSet
  filepath: data/companies_raw/big.parquet
  layer: companies_raw
```

...which is probably not what you wanted. The reason for this is that ["`parse()` will always match the shortest text necessary (from left to right)"](https://github.com/r1chardj0n3s/parse#potential-gotchas). This is fine but would need explaining.

We also need to be wary of how to handle multiple patterns that match. e.g. let's say you want to override the default. Then you should be able to do something like this:

```
"{root_namespace}.{dataset_name}@spark":
  type: spark.SparkDataSet
  filepath: data/{root_namespace}/{dataset_name}.parquet
  file_format: parquet

france.companies@spark:
  type: spark.SomethingElse
  filepath: other/data/france/companies.parquet
  file_format: parquet
```

Here `france.companies` should take precedence over the general pattern, so we can have a rule like "if two patterns match and there's no `{}` in one pattern, then that one is more specific and so wins". But how about in cases where multiple `{}` patterns match?

```
"{root_namespace}.{dataset_name}@spark":
  type: spark.SparkDataSet
  filepath: data/{root_namespace}/{dataset_name}.parquet
  file_format: parquet

"{dataset_name}@spark":
  type: spark.SomethingElse
  filepath: data/{dataset_name}.parquet
  file_format: parquet
```

Now `france.companies@spark` matches both entries, and it's not really clear which should win. Do we use the order the patterns are defined in the file to choose which wins? What about if the patterns come from a different file (since we merge multiple catalog.yml files together)?

All this is not to say that this solution is unworkable. I do think `parse` feels like a good fit here (after all, regex will suffer from similar problems), but that we really need to think about how it would work in practice. As soon as you start defining general patterns to match lots of things, you're going to want to override that for exceptions, and so we need to know exactly how that would work.
@AntonyMilneQB - thank you for this, it's really an excellent write-up. One push I would make is that we could manage this complexity by providing a live editor / prototyping space in Viz. 

I think it would massively improve the learning journey and help users get up and running quickly.
Something like this https://textfsm.nornir.tech/#
Thanks for diving a bit deeper into `parse` @AntonyMilneQB and writing up your thoughts!

I have a question about the final bit though. You're saying that if the catalog contains:
```
"{root_namespace}.{dataset_name}@spark":
  type: spark.SparkDataSet
  filepath: data/{root_namespace}/{dataset_name}.parquet
  file_format: parquet

"{dataset_name}@spark":
  type: spark.SomethingElse
  filepath: data/{dataset_name}.parquet
  file_format: parquet
```

`france.companies@spark` would match both. How come? Is the dot not a specific pattern that should be matched? 

@merelcht The `.` is not a specific pattern that should be matched - it's just a character in the string like anything else. Think of replacing it with the letter `A`:

```
In [2]: parse("{dataset_name}@spark", "france.companies@spark")
Out[2]: <Result () {'dataset_name': 'france.companies'}>

In [4]: parse("{dataset_name}@spark", "franceAcompanies@spark")
Out[4]: <Result () {'dataset_name': 'franceAcompanies'}>
```

One way around this is to add a [format specification](https://github.com/r1chardj0n3s/parse#format-specification) like this:

```
In [6]: parse("{dataset_name:w}@spark", "france.companies@spark")
# No matches because . doesn't come under format w, which means Letters, numbers and underscore

In [4]: parse("{dataset_name}@spark", "franceAcompanies@spark")
Out[4]: <Result () {'dataset_name': 'franceAcompanies'}>
# Still matches because the string is of format w
```

But, by default, if you don't specify a format then it will just match anything. This still feels better to me than regex but it is definitely not as simple as it might first appear.

@idanov also asked me to post the `make_like` example here - I'll do it in a second.
# Alternative proposal: [omegaconf custom resolvers](https://omegaconf.readthedocs.io/en/latest/custom_resolvers.html#custom-resolvers)

This is also originally @idanov's idea. The above example would look something like this:

```
# Define the "template" entry - needs to be an actual valid catalog entry I think (?)
germany.companies@spark:
  type: spark.SparkDataSet
  filepath: ${make_filepath:${..dataset_name}}  # this won't work - I don't know what the right syntax is or even if it's possible
  file_format: parquet

# Define other entries that follow the same pattern
france.companies@spark: ${make_like: ${..germany.companies@spark}} 
uk.companies@spark: ${make_like: ${..germany.companies@spark}} 
italy.companies@spark: ${make_like: ${..germany.companies@spark}} 
```

Somewhere (presumably user defined, but maybe we could build some common ones in?) the resolvers would be defined as something like:
```
OmegaConf.register_new_resolver("make_filepath", lambda dataset_name: f"data{dataset_name.split(".")[0]}/{dataset_name.split("."[1])}.parquet
OmegaConf.register_new_resolver("make_like", lambda catalog_entry: catalog_entry)
```

This is all pretty vague because I'm not immediately sure whether these things are possible using omegaconf resolvers...
* how to make the `germany.companies@spark` string accessible inside the catalog entry?
* how to get `make_like` to work?

Pros:
* no ambiguity with multiple pattern matching
* no need to introduce another paradigm for yaml manipulation on top of omegaconf which we are already using: we have one consistent mechanism for variable interpolation, environment variables, pattern matching, etc.

Cons:
* **doesn't actually reduce the number of catalog entries, just makes them much shorter** - you still need to define the france, italy, uk entries explicitly (is there some clever way in omegaconf to avoid this? I'm not sure since it seems you can only do variable interpolation in values, not keys?)
* not actually sure if it's possible - needs some further thought on how these resolvers would work!
* needs careful thought about implementation if we are to not make omegaconf a dependency of the data catalog

Pro or con depending on your perspective: more powerful and flexible than general pattern matching syntax because it allows for arbitrary Python functions, not just string parsing.
@AntonyMilneQB 

I don't think it's possible to pass a value that needs to be interpolated to a resolver. In this prototype example below:

```
# Define the "template" entry - needs to be an actual valid catalog entry I think (?)
germany.companies@spark:
  type: spark.SparkDataSet
  filepath: ${make_filepath:${..dataset_name}}  # this won't work - I don't know what the right syntax is or even if it's possible
  file_format: parquet

# Define other entries that follow the same pattern
france.companies@spark: ${make_like: ${..germany.companies@spark}} 
uk.companies@spark: ${make_like: ${..germany.companies@spark}} 
italy.companies@spark: ${make_like: ${..germany.companies@spark}} 
```

`${make_filepath:${..dataset_name}}` isn't possible as far as I found. Only if you add the `dataset_name` as a key in the file as well, but that would defeat the purpose of generalising the value, because you'd then just have the same file path value for all countries. 
Summary of the **technical design** discussions we had on 22/3 and 30/3:

Through the discussions we've identified there are in fact three problems that are all related.

## Problems to address

1. Users don't like working with large + hard to maintain catalog files.
2. Users want to be able to have another default than `MemoryDataSet` for datasets not defined in the catalog.
3. Users need a way to keep constant/global values in one place to minimise mistakes e.g. typos and pass these in configuration files. [Add templating functionality to allow "global" values in the catalog in OmegaConfigLoader](https://github.com/kedro-org/kedro/issues/2175)

## Proposed solutions:

1.[ Dataset factories](https://github.com/kedro-org/kedro/issues/2423#issuecomment-1470784359): addresses problem 1 & 2
2. Omegaconf "make-like" resolver: turns out that this isn't a feasible solution. 

## Problem 1: points discussed
- General agreement that large catalog files is a valid problem and not only for QuantumBlack vertical teams.
- Users are reluctant to make use of modular pipelines because it increases configuration.
- General impression is that when users start working with Kedro they prefer readable (more verbose) catalog files, but the more they get used the Kedro the greater the need for features like templating and other tools that make the catalog more compact. 
- Concern with any solution to make the catalog more compact is **readability**. With existing solutions like Jinja2 and yaml-anchors it's also hard to understand what the "compiled" catalog looks like that's used by Kedro.
   - **Solution:**  add a mechanism to compile the catalog. Could be a CLI command similar to [`dbt compile`](https://docs.getdbt.com/reference/commands/compile)
- Whatever solution we go for, we will always allow the full fledged dataset definitions in the catalog, so users do not have to use the more compact syntax. 


## Problem 2: points discussed
- @WaylonWalker created a tool that allows to change the default: https://github.com/WaylonWalker/kedro-auto-catalog
   - This solution saves the datasets into a separate "default" catalog file, so it addresses problem 2, but not problem 1 because datasets still end up in the catalog.
   - AFAIK you can only add 1 default
- The datasets factories solution would address this problem.   

## Problem 3
Not yet addressed. 

## Other comments
- Omegaconf resolvers do allow for some reduction of the catalog content just like `yaml` anchors. 
- With custom resolvers power users can find ways to reduce their catalogs more by themselves.
- Perhaps it's not worth introducing this extra syntax if the `OmegaConfigLoader` leverages all resolver syntax properly. 

## Next steps

- [x] Discuss problem 3 in the next technical design session: [Add templating functionality to allow "global" values in the catalog in OmegaConfigLoader](https://github.com/kedro-org/kedro/issues/2175)
- [x] Through some quick experimenting we found that variable interpolation can be used in place of yaml anchors. What needs to be done to make this work in `OmegaConfigLoader`? Same solution as for [Add templating functionality to allow "global" values in the catalog in OmegaConfigLoader](https://github.com/kedro-org/kedro/issues/2175)? To be done in #2507 

> **Note** We need to do user testing to verify any solution we decide on.



I know we don't want to - but I also think we need to ask, does our solution support dynamic pipelines? Our users, particularly those who care about this templating stuff are going to try that sort of thing very quickly.
|     |     |     |     |     |
| --- | --- | --- | --- | --- |
|     | Reduce the number Catalog Entry (How many top-level keys in the file)| Shorten Catalog Entry (How many lines for each Dataset) | Ability to Change Default DataSet | Constant/Global Value for all Configurations |
| DataSet Factory | ✅   | ✅   | ✅   | ❌   |
| OmegaConf Interpolation | ❌   | ✅   | ❌   | ？   |
| Environment Variable | ❌   | ❌   | ❌   | ✅   |
| OmegaConf Custom Resolver | ❌   | ✅   | ❌   | ？   |

(Feel free to edit, the ? are things that I am not sure about)
Updated at 2023-04-11
For a slightly 🌶️ comparison


|     |     |     |     |     |
| --- | --- | --- | --- | --- |
|     | Reduce the number Catalog Entry (How many top-level keys in the file)| Shorten Catalog Entry (How many lines for each Dataset) | Ability to Change Default DataSet | Constant/Global Value for all Configurations |
| Jinja2 (with .j2 template support) | ✅    | ✅   | ❌   |  ✅  |
@noklam I filled out two of the questions marks but don't understand what the last column means?

Also I think the dataset factory should have ✅ for "Reduce the number Catalog Entry", no? Since you would have a general pattern matching syntax rather than needing to define each entry explicitly.

Resolving patch:
diff --git a/kedro/io/data_catalog.py b/kedro/io/data_catalog.py
--- a/kedro/io/data_catalog.py
+++ b/kedro/io/data_catalog.py
@@ -11,7 +11,9 @@
 import logging
 import re
 from collections import defaultdict
-from typing import Any
+from typing import Any, Iterable
+
+from parse import parse
 
 from kedro.io.core import (
     AbstractDataSet,
@@ -136,11 +138,14 @@ class DataCatalog:
     to the underlying data sets.
     """
 
-    def __init__(
+    def __init__(  # pylint: disable=too-many-arguments
         self,
         data_sets: dict[str, AbstractDataSet] = None,
         feed_dict: dict[str, Any] = None,
         layers: dict[str, set[str]] = None,
+        dataset_patterns: dict[str, dict[str, Any]] = None,
+        load_versions: dict[str, str] = None,
+        save_version: str = None,
     ) -> None:
         """``DataCatalog`` stores instances of ``AbstractDataSet``
         implementations to provide ``load`` and ``save`` capabilities from
@@ -156,6 +161,15 @@ def __init__(
                 to a set of data set names, according to the
                 data engineering convention. For more details, see
                 https://docs.kedro.org/en/stable/resources/glossary.html#layers-data-engineering-convention
+            dataset_patterns: A dictionary of data set factory patterns
+                and corresponding data set configuration
+            load_versions: A mapping between data set names and versions
+                to load. Has no effect on data sets without enabled versioning.
+            save_version: Version string to be used for ``save`` operations
+                by all data sets with enabled versioning. It must: a) be a
+                case-insensitive string that conforms with operating system
+                filename limitations, b) always return the latest version when
+                sorted in lexicographical order.
 
         Example:
         ::
@@ -170,8 +184,12 @@ def __init__(
         self._data_sets = dict(data_sets or {})
         self.datasets = _FrozenDatasets(self._data_sets)
         self.layers = layers
+        # Keep a record of all patterns in the catalog.
+        # {dataset pattern name : dataset pattern body}
+        self._dataset_patterns = dataset_patterns or {}
+        self._load_versions = load_versions or {}
+        self._save_version = save_version
 
-        # import the feed dict
         if feed_dict:
             self.add_feed_dict(feed_dict)
 
@@ -181,7 +199,7 @@ def _logger(self):
 
     @classmethod
     def from_config(
-        cls: type,
+        cls,
         catalog: dict[str, dict[str, Any]] | None,
         credentials: dict[str, dict[str, Any]] = None,
         load_versions: dict[str, str] = None,
@@ -257,35 +275,124 @@ class to be loaded is specified with the key ``type`` and their
             >>> catalog.save("boats", df)
         """
         data_sets = {}
+        dataset_patterns = {}
         catalog = copy.deepcopy(catalog) or {}
         credentials = copy.deepcopy(credentials) or {}
         save_version = save_version or generate_timestamp()
         load_versions = copy.deepcopy(load_versions) or {}
+        layers: dict[str, set[str]] = defaultdict(set)
 
-        missing_keys = load_versions.keys() - catalog.keys()
+        for ds_name, ds_config in catalog.items():
+            ds_config = _resolve_credentials(ds_config, credentials)
+            if cls._is_pattern(ds_name):
+                # Add each factory to the dataset_patterns dict.
+                dataset_patterns[ds_name] = ds_config
+
+            else:
+                ds_layer = ds_config.pop("layer", None)
+                if ds_layer is not None:
+                    layers[ds_layer].add(ds_name)
+                data_sets[ds_name] = AbstractDataSet.from_config(
+                    ds_name, ds_config, load_versions.get(ds_name), save_version
+                )
+        dataset_layers = layers or None
+        sorted_patterns = cls._sort_patterns(dataset_patterns)
+        missing_keys = [
+            key
+            for key in load_versions.keys()
+            if not (cls._match_pattern(sorted_patterns, key) or key in catalog)
+        ]
         if missing_keys:
             raise DatasetNotFoundError(
                 f"'load_versions' keys [{', '.join(sorted(missing_keys))}] "
                 f"are not found in the catalog."
             )
 
-        layers: dict[str, set[str]] = defaultdict(set)
-        for ds_name, ds_config in catalog.items():
-            ds_layer = ds_config.pop("layer", None)
-            if ds_layer is not None:
-                layers[ds_layer].add(ds_name)
+        return cls(
+            data_sets=data_sets,
+            layers=dataset_layers,
+            dataset_patterns=sorted_patterns,
+            load_versions=load_versions,
+            save_version=save_version,
+        )
 
-            ds_config = _resolve_credentials(ds_config, credentials)
-            data_sets[ds_name] = AbstractDataSet.from_config(
-                ds_name, ds_config, load_versions.get(ds_name), save_version
-            )
+    @staticmethod
+    def _is_pattern(pattern: str):
+        """Check if a given string is a pattern. Assume that any name with '{' is a pattern."""
+        return "{" in pattern
+
+    @staticmethod
+    def _match_pattern(
+        data_set_patterns: dict[str, dict[str, Any]], data_set_name: str
+    ) -> str | None:
+        """Match a dataset name against patterns in a dictionary containing patterns"""
+        for pattern, _ in data_set_patterns.items():
+            result = parse(pattern, data_set_name)
+            if result:
+                return pattern
+        return None
 
-        dataset_layers = layers or None
-        return cls(data_sets=data_sets, layers=dataset_layers)
+    @classmethod
+    def _sort_patterns(
+        cls, data_set_patterns: dict[str, dict[str, Any]]
+    ) -> dict[str, dict[str, Any]]:
+        """Sort a dictionary of dataset patterns according to parsing rules -
+        1. Decreasing specificity (number of characters outside the curly brackets)
+        2. Decreasing number of placeholders (number of curly bracket pairs)
+        3. Alphabetically
+        """
+        sorted_keys = sorted(
+            data_set_patterns,
+            key=lambda pattern: (
+                -(cls._specificity(pattern)),
+                -pattern.count("{"),
+                pattern,
+            ),
+        )
+        sorted_patterns = {}
+        for key in sorted_keys:
+            sorted_patterns[key] = data_set_patterns[key]
+        return sorted_patterns
+
+    @staticmethod
+    def _specificity(pattern: str) -> int:
+        """Helper function to check the length of exactly matched characters not inside brackets
+        Example -
+        specificity("{namespace}.companies") = 10
+        specificity("{namespace}.{dataset}") = 1
+        specificity("france.companies") = 16
+        """
+        # Remove all the placeholders from the pattern and count the number of remaining chars
+        result = re.sub(r"\{.*?\}", "", pattern)
+        return len(result)
 
     def _get_dataset(
         self, data_set_name: str, version: Version = None, suggest: bool = True
     ) -> AbstractDataSet:
+        matched_pattern = self._match_pattern(self._dataset_patterns, data_set_name)
+        if data_set_name not in self._data_sets and matched_pattern:
+            # If the dataset is a patterned dataset, materialise it and add it to
+            # the catalog
+            data_set_config = self._resolve_config(data_set_name, matched_pattern)
+            ds_layer = data_set_config.pop("layer", None)
+            if ds_layer:
+                self.layers = self.layers or {}
+                self.layers.setdefault(ds_layer, set()).add(data_set_name)
+            data_set = AbstractDataSet.from_config(
+                data_set_name,
+                data_set_config,
+                self._load_versions.get(data_set_name),
+                self._save_version,
+            )
+            if self._specificity(matched_pattern) == 0:
+                self._logger.warning(
+                    "Config from the dataset factory pattern '%s' in the catalog will be used to "
+                    "override the default MemoryDataset creation for the dataset '%s'",
+                    matched_pattern,
+                    data_set_name,
+                )
+
+            self.add(data_set_name, data_set)
         if data_set_name not in self._data_sets:
             error_msg = f"Dataset '{data_set_name}' not found in the catalog"
 
@@ -298,9 +405,7 @@ def _get_dataset(
                 if matches:
                     suggestions = ", ".join(matches)
                     error_msg += f" - did you mean one of these instead: {suggestions}"
-
             raise DatasetNotFoundError(error_msg)
-
         data_set = self._data_sets[data_set_name]
         if version and isinstance(data_set, AbstractVersionedDataSet):
             # we only want to return a similar-looking dataset,
@@ -311,6 +416,35 @@ def _get_dataset(
 
         return data_set
 
+    def __contains__(self, data_set_name):
+        """Check if an item is in the catalog as a materialised dataset or pattern"""
+        matched_pattern = self._match_pattern(self._dataset_patterns, data_set_name)
+        if data_set_name in self._data_sets or matched_pattern:
+            return True
+        return False
+
+    def _resolve_config(
+        self,
+        data_set_name: str,
+        matched_pattern: str,
+    ) -> dict[str, Any]:
+        """Get resolved AbstractDataSet from a factory config"""
+        result = parse(matched_pattern, data_set_name)
+        config_copy = copy.deepcopy(self._dataset_patterns[matched_pattern])
+        # Resolve the factory config for the dataset
+        for key, value in config_copy.items():
+            if isinstance(value, Iterable) and "}" in value:
+                # result.named: gives access to all dict items in the match result.
+                # format_map fills in dict values into a string with {...} placeholders
+                # of the same key name.
+                try:
+                    config_copy[key] = str(value).format_map(result.named)
+                except KeyError as exc:
+                    raise DatasetError(
+                        f"Unable to resolve '{key}' for the pattern '{matched_pattern}'"
+                    ) from exc
+        return config_copy
+
     def load(self, name: str, version: str = None) -> Any:
         """Loads a registered data set.
 
@@ -573,10 +707,20 @@ def shallow_copy(self) -> DataCatalog:
         Returns:
             Copy of the current object.
         """
-        return DataCatalog(data_sets=self._data_sets, layers=self.layers)
+        return DataCatalog(
+            data_sets=self._data_sets,
+            layers=self.layers,
+            dataset_patterns=self._dataset_patterns,
+            load_versions=self._load_versions,
+            save_version=self._save_version,
+        )
 
     def __eq__(self, other):
-        return (self._data_sets, self.layers) == (other._data_sets, other.layers)
+        return (self._data_sets, self.layers, self._dataset_patterns) == (
+            other._data_sets,
+            other.layers,
+            other._dataset_patterns,
+        )
 
     def confirm(self, name: str) -> None:
         """Confirm a dataset by its name.
diff --git a/kedro/runner/runner.py b/kedro/runner/runner.py
--- a/kedro/runner/runner.py
+++ b/kedro/runner/runner.py
@@ -32,7 +32,7 @@ class AbstractRunner(ABC):
     """
 
     def __init__(self, is_async: bool = False):
-        """Instantiates the runner classs.
+        """Instantiates the runner class.
 
         Args:
             is_async: If True, the node inputs and outputs are loaded and saved
@@ -74,14 +74,25 @@ def run(
         hook_manager = hook_manager or _NullPluginManager()
         catalog = catalog.shallow_copy()
 
-        unsatisfied = pipeline.inputs() - set(catalog.list())
+        # Check which datasets used in the pipeline are in the catalog or match
+        # a pattern in the catalog
+        registered_ds = [ds for ds in pipeline.data_sets() if ds in catalog]
+
+        # Check if there are any input datasets that aren't in the catalog and
+        # don't match a pattern in the catalog.
+        unsatisfied = pipeline.inputs() - set(registered_ds)
+
         if unsatisfied:
             raise ValueError(
                 f"Pipeline input(s) {unsatisfied} not found in the DataCatalog"
             )
 
-        free_outputs = pipeline.outputs() - set(catalog.list())
-        unregistered_ds = pipeline.data_sets() - set(catalog.list())
+        # Check if there's any output datasets that aren't in the catalog and don't match a pattern
+        # in the catalog.
+        free_outputs = pipeline.outputs() - set(registered_ds)
+        unregistered_ds = pipeline.data_sets() - set(registered_ds)
+
+        # Create a default dataset for unregistered datasets
         for ds_name in unregistered_ds:
             catalog.add(ds_name, self.create_default_data_set(ds_name))
 
@@ -420,7 +431,7 @@ def _run_node_sequential(
     items: Iterable = outputs.items()
     # if all outputs are iterators, then the node is a generator node
     if all(isinstance(d, Iterator) for d in outputs.values()):
-        # Python dictionaries are ordered so we are sure
+        # Python dictionaries are ordered, so we are sure
         # the keys and the chunk streams are in the same order
         # [a, b, c]
         keys = list(outputs.keys())


Test patch:
diff --git a/tests/io/test_data_catalog.py b/tests/io/test_data_catalog.py
--- a/tests/io/test_data_catalog.py
+++ b/tests/io/test_data_catalog.py
@@ -87,6 +87,68 @@ def sane_config_with_tracking_ds(tmp_path):
     }
 
 
+@pytest.fixture
+def config_with_dataset_factories():
+    return {
+        "catalog": {
+            "{brand}_cars": {
+                "type": "pandas.CSVDataSet",
+                "filepath": "data/01_raw/{brand}_cars.csv",
+            },
+            "audi_cars": {
+                "type": "pandas.ParquetDataSet",
+                "filepath": "data/01_raw/audi_cars.pq",
+            },
+            "{type}_boats": {
+                "type": "pandas.CSVDataSet",
+                "filepath": "data/01_raw/{type}_boats.csv",
+            },
+        },
+    }
+
+
+@pytest.fixture
+def config_with_dataset_factories_with_default(config_with_dataset_factories):
+    config_with_dataset_factories["catalog"]["{default_dataset}"] = {
+        "type": "pandas.CSVDataSet",
+        "filepath": "data/01_raw/{default_dataset}.csv",
+    }
+    return config_with_dataset_factories
+
+
+@pytest.fixture
+def config_with_dataset_factories_bad_pattern(config_with_dataset_factories):
+    config_with_dataset_factories["catalog"]["{type}@planes"] = {
+        "type": "pandas.ParquetDataSet",
+        "filepath": "data/01_raw/{brand}_plane.pq",
+    }
+    return config_with_dataset_factories
+
+
+@pytest.fixture
+def config_with_dataset_factories_only_patterns():
+    return {
+        "catalog": {
+            "{default}": {
+                "type": "pandas.CSVDataSet",
+                "filepath": "data/01_raw/{default}.csv",
+            },
+            "{namespace}_{dataset}": {
+                "type": "pandas.CSVDataSet",
+                "filepath": "data/01_raw/{namespace}_{dataset}.pq",
+            },
+            "{country}_companies": {
+                "type": "pandas.CSVDataSet",
+                "filepath": "data/01_raw/{country}_companies.csv",
+            },
+            "{dataset}s": {
+                "type": "pandas.CSVDataSet",
+                "filepath": "data/01_raw/{dataset}s.csv",
+            },
+        },
+    }
+
+
 @pytest.fixture
 def data_set(filepath):
     return CSVDataSet(filepath=filepath, save_args={"index": False})
@@ -683,3 +745,151 @@ def test_no_versions_with_cloud_protocol(self):
         )
         with pytest.raises(DatasetError, match=pattern):
             versioned_dataset.load()
+
+
+class TestDataCatalogDatasetFactories:
+    def test_match_added_to_datasets_on_get(self, config_with_dataset_factories):
+        """Check that the datasets that match patterns are only added when fetched"""
+        catalog = DataCatalog.from_config(**config_with_dataset_factories)
+        assert "{brand}_cars" not in catalog._data_sets
+        assert "tesla_cars" not in catalog._data_sets
+        assert "{brand}_cars" in catalog._dataset_patterns
+
+        tesla_cars = catalog._get_dataset("tesla_cars")
+        assert isinstance(tesla_cars, CSVDataSet)
+        assert "tesla_cars" in catalog._data_sets
+
+    @pytest.mark.parametrize(
+        "dataset_name, expected",
+        [
+            ("audi_cars", True),
+            ("tesla_cars", True),
+            ("row_boats", True),
+            ("boats", False),
+            ("tesla_card", False),
+        ],
+    )
+    def test_exists_in_catalog_config(
+        self, config_with_dataset_factories, dataset_name, expected
+    ):
+        """Check that the dataset exists in catalog when it matches a pattern
+        or is in the catalog"""
+        catalog = DataCatalog.from_config(**config_with_dataset_factories)
+        assert (dataset_name in catalog) == expected
+
+    def test_patterns_not_in_catalog_datasets(self, config_with_dataset_factories):
+        """Check that the pattern is not in the catalog datasets"""
+        catalog = DataCatalog.from_config(**config_with_dataset_factories)
+        assert "audi_cars" in catalog._data_sets
+        assert "{brand}_cars" not in catalog._data_sets
+        assert "audi_cars" not in catalog._dataset_patterns
+        assert "{brand}_cars" in catalog._dataset_patterns
+
+    def test_explicit_entry_not_overwritten(self, config_with_dataset_factories):
+        """Check that the existing catalog entry is not overwritten by config in pattern"""
+        catalog = DataCatalog.from_config(**config_with_dataset_factories)
+        audi_cars = catalog._get_dataset("audi_cars")
+        assert isinstance(audi_cars, ParquetDataSet)
+
+    @pytest.mark.parametrize(
+        "dataset_name,pattern",
+        [
+            ("missing", "Dataset 'missing' not found in the catalog"),
+            ("tesla@cars", "Dataset 'tesla@cars' not found in the catalog"),
+        ],
+    )
+    def test_dataset_not_in_catalog_when_no_pattern_match(
+        self, config_with_dataset_factories, dataset_name, pattern
+    ):
+        """Check that the dataset is not added to the catalog when there is no pattern"""
+        catalog = DataCatalog.from_config(**config_with_dataset_factories)
+        with pytest.raises(DatasetError, match=re.escape(pattern)):
+            catalog._get_dataset(dataset_name)
+
+    def test_sorting_order_patterns(self, config_with_dataset_factories_only_patterns):
+        """Check that the sorted order of the patterns is correct according
+        to parsing rules"""
+        catalog = DataCatalog.from_config(**config_with_dataset_factories_only_patterns)
+        sorted_keys_expected = [
+            "{country}_companies",
+            "{namespace}_{dataset}",
+            "{dataset}s",
+            "{default}",
+        ]
+        assert list(catalog._dataset_patterns.keys()) == sorted_keys_expected
+
+    def test_default_dataset(self, config_with_dataset_factories_with_default, caplog):
+        """Check that default dataset is used when no other pattern matches"""
+        catalog = DataCatalog.from_config(**config_with_dataset_factories_with_default)
+        assert "jet@planes" not in catalog._data_sets
+        jet_dataset = catalog._get_dataset("jet@planes")
+        log_record = caplog.records[0]
+        assert log_record.levelname == "WARNING"
+        assert (
+            "Config from the dataset factory pattern '{default_dataset}' "
+            "in the catalog will be used to override the default "
+            "MemoryDataset creation for the dataset 'jet@planes'" in log_record.message
+        )
+        assert isinstance(jet_dataset, CSVDataSet)
+
+    def test_unmatched_key_error_when_parsing_config(
+        self, config_with_dataset_factories_bad_pattern
+    ):
+        """Check error raised when key mentioned in the config is not in pattern name"""
+        catalog = DataCatalog.from_config(**config_with_dataset_factories_bad_pattern)
+        pattern = "Unable to resolve 'filepath' for the pattern '{type}@planes'"
+        with pytest.raises(DatasetError, match=re.escape(pattern)):
+            catalog._get_dataset("jet@planes")
+
+    def test_factory_layer(self, config_with_dataset_factories):
+        """Check that layer is correctly processed for patterned datasets"""
+        config_with_dataset_factories["catalog"]["{brand}_cars"]["layer"] = "raw"
+        catalog = DataCatalog.from_config(**config_with_dataset_factories)
+        _ = catalog._get_dataset("tesla_cars")
+        assert catalog.layers["raw"] == {"tesla_cars"}
+
+    def test_factory_config_versioned(
+        self, config_with_dataset_factories, filepath, dummy_dataframe
+    ):
+        """Test load and save of versioned data sets from config"""
+        config_with_dataset_factories["catalog"]["{brand}_cars"]["versioned"] = True
+        config_with_dataset_factories["catalog"]["{brand}_cars"]["filepath"] = filepath
+
+        assert "tesla_cars" not in config_with_dataset_factories
+
+        # Decompose `generate_timestamp` to keep `current_ts` reference.
+        current_ts = datetime.now(tz=timezone.utc)
+        fmt = (
+            "{d.year:04d}-{d.month:02d}-{d.day:02d}T{d.hour:02d}"
+            ".{d.minute:02d}.{d.second:02d}.{ms:03d}Z"
+        )
+        version = fmt.format(d=current_ts, ms=current_ts.microsecond // 1000)
+
+        catalog = DataCatalog.from_config(
+            **config_with_dataset_factories,
+            load_versions={"tesla_cars": version},
+            save_version=version,
+        )
+
+        catalog.save("tesla_cars", dummy_dataframe)
+        path = Path(
+            config_with_dataset_factories["catalog"]["{brand}_cars"]["filepath"]
+        )
+        path = path / version / path.name
+        assert path.is_file()
+
+        reloaded_df = catalog.load("tesla_cars")
+        assert_frame_equal(reloaded_df, dummy_dataframe)
+
+        reloaded_df_version = catalog.load("tesla_cars", version=version)
+        assert_frame_equal(reloaded_df_version, dummy_dataframe)
+
+        # Verify that `VERSION_FORMAT` can help regenerate `current_ts`.
+        actual_timestamp = datetime.strptime(
+            catalog.datasets.tesla_cars.resolve_load_version(),  # pylint: disable=no-member
+            VERSION_FORMAT,
+        )
+        expected_timestamp = current_ts.replace(
+            microsecond=current_ts.microsecond // 1000 * 1000, tzinfo=None
+        )
+        assert actual_timestamp == expected_timestamp