Add *LCEL* Transformers
Labels: name: area: doc loader, name: auto:enhancement, name: size:XXL

**Description:**

Langchain introduces the `BaseDocumentTransformer` interface for document transformation, featuring two abstract methods:

- `def transform_documents(self, documents: Sequence[Document], **kwargs: Any) -> Sequence[Document]`
- `async def atransform_documents(self, documents: Sequence[Document], **kwargs: Any) -> Sequence[Document]`

However, this interface lacks compatibility with **LCEL**.

Document transformation can vary in speed, from quick processes like text splitting to more extensive processes such as generating document summaries, translation via `DoctranTextTranslator`, and creating questions. Introducing an asynchronous lazy approach for transformations would be advantageous, facilitating use in streams and reducing memory consumption.

Furthermore, making all this compatible with **LCEL** presents an opportunity. The proposed solution involves an incremental approach. The initial step is to introduce a new, asynchronous, and lazy LCEL-compatible interface. Subsequently, all existing transformations (21 classes) would migrate to this new interface.

The `RunnableGeneratorDocumentTransformer` interface, derived from `RunnableSerializable` and `BaseDocumentTransformer`, offers two new methods:

- `lazy_transform_documents()`
- `alazy_transform_documents()`

This class aims to replace `BaseDocumentTransformer`.

These methods are generators, possibly asynchronous (`AsyncIterator[Document]`). The default implementations of `transform_documents()` and `atransform_documents()` use the lazy versions internally to maintain compatibility with streams.

A new version of `TextSplitter` and `CharacterTextSplitter` is proposed to demonstrate modifying the current transformer code to accommodate the new interface. (See `unit_test/document_transformers/new_text_splitter.py` and `unit_test/document_transformers/test_new_text_splitter.py`).

The update enables a series of transformations:

```python
runnable = (CharacterTextSplitter(...) | TokenTextSplitter(...))
transformed = list(runnable.invoke([doc1, doc2]))
```

The addition of the `+` operator allows multiple transformations for the same inputs:

```python
runnable = (CharacterTextSplitter(...) + TokenTextSplitter(...))
transformed = list(runnable.invoke([doc1, doc2]))
```

Operators can be combined:

```python
runnable = (UpperTransformer(...) + CharacterTextSplitter(...) |
            TokenTextSplitter(...))
transformed = list(runnable.invoke([doc1, doc2]))
```

Currently, only the unit tests provide examples of transformations.

If this proposal is accepted, the plan is to refactor all *Transformer* implementations to make them compatible with a lazy, asynchronous, and *LCEL-compatible* approach. This will be undertaken post-integration into the **master** branch. Refer to the tests `test_lcel_transform_documents()` and `test_alcel_transform_documents()`.

**Why is the lazy approach important?**

Consider an API retrieving a URL from a client, responsible for fetching all documents and importing them into a vector store. Applying three transformations to each fragment (via `ParentDocumentRetriever`), the current code will consume excessive memory. In contrast, a lazy approach significantly reduces memory requirements, making it feasible to respond to multiple users simultaneously without a memory-intensive instance.

To improve the proximity of each piece, the 
code applies three transformations to each fragment (via `ParentDocumentRetriever`): 
    - a summary, 
    - the generation of 3 questions, 
    - and translation into French.

With the current langchain API, the code will :
- retrieve 10 PDF files of 200 pages, 
- split them into 10-page fragments (i.e. 200+20*10 pages in memory)
- Produce 5 versions of each fragment (i.e. 200+(20*10)*4= 1000 pages in memory)
- then save the fragments in the vector store 1000 pages + 1000 texts to import

The same scenario with a lazy approach translates as follows:
- Retrieve a lazy file one file at a time (Loader.lazy_load(); 200 pages in memory)
- Split the file into lazy files ( 200 pages + 1 fragment)
- For each fragment, produce 5 versions (200 pages + 1 fragment + 1 variation)
- Then save the fragments in the vectorstore
The maximum memory footprint is : 1 document, 1 fragment, 1 variation.

**Transformers benefiting from the async lazy approach:**

- `DoctranQATransformer`
- `DoctranTextTranslator`
- `OpenAIMetadataTagger`
- `DoctranPropertyExtractor`
- `NucliaTextTransformer`
- `GoogleTranslateTransformer`

Transformers less suited at this stage are those primarily working in memory:

- `LongContextReorder`
- `EmbeddingsRedundantFilter`
- `EmbeddingsClusteringFilter`
- `Html2TextTransformer`
- All `TextSplitter`

**Preliminary Step for Another Pull-Request:**

This proposal serves as a preliminary step for another pull-request, offering a better implementation of `ParentDocumentRetriever`. Refer to [langchain-rag](https://github.com/pprados/langchain-rag).

**Issue:**

- `BaseDocumentTransformer` is incompatible with a stream.
- `BaseDocumentTransformer` is not LCEL-compatible.
- The current API consumes excessive memory in many online scenarios.

**Dependencies:**

No dependencies.

**Tag maintainer:**

@baskaryan

**Twitter handle:**

Twitter ID: pprados