Neural Search in Action

Representing,transiting &searching
multimodal data
Berlin · Beijing · Shenzhen

Han Xiao, Founder of Jina AI@hxiao@JinaAI_About me & Jina AI
Han Xiao, Founder & CEO of Jina AI. Based in Berlin, Germany.
- ML PhD in 2014 TU Munich; Zalando Research; Tencent AI Lab; Creatorof Fashion-MNIST.
Jina AI- Founded in 2020, focus on multimodal AI search & create- Opensource contributor: Jina, DocArray (Linux Foundation),
CLIP-as-service, …
- 60 people, HQ in Berlin. Offices in Beijing, Shenzhen.

Jina AI Tech SpectrumPrompt tuning

the process of crafting and refining the input
prompts in order to guide its output towards
specific, desired responses.

the deployment of fine-tuned models in a
production environment, usually requiring
substantial resources such as GPU hosting.
MLOps, emphasizing the serving of mid-size tolarge models in a scalable, efficient, and reliable
manner.

Model serving

Prompt serving
wrapping and serving prompts through an API,
without hosting heavy models. The API calls apublic large language model service and handles
the orchestration of inputs and outputs in a chainof operations.

Also known as fine-tuning, involves adjusting theparameters of a pre-trained model on a new,often task-specific dataset to improve its
performance and adapt it to a specificapplication.

Model tuningFaster time2market

Prompt tuning
PromptPerfect

LLMSearch

DevGPT

EnsembleThinkGPTArtiBannerAgentChain

SceneX

Model serving
Inference API

Jina+DocArray

Prompt serving

LC-serve

JcloudDiscoArt

OpenGPT

DalleFlow

Rationale

CLIP-as-service
DocsQA

Finetuner

Long-term invest

Model tuning

Agenda- Preliminary: multimodal AI- Opensource package: DocArray-Motivation
Representing data
Transiting data
Storing data
Retrieving data

- Multimodal at scale in production
This tutorial may require technical knowledge. Familiarity with Python 3.7+
concepts like data classes could be helpful.

Preliminary:from unimodal to multimodal

Multimodal

Neural search

Unimodal AI
Before 2021

FutureCreative AI
2022

Cross-modalFrom unimodal to multimodal"modality" roughly means "data type".
- Unimodal AI refers to applying AI to one specific type of data.
- Most early machine learning works fall into this category.
- Even today, when you open any machine learning literature,
unimodal AI is still the majority of the content.

Unimodal - NLP
LDA was the 2010's transformer

Unimodal tasks in NLP
Adhoc methods for NLP problems

Unimodal - CV
Fashion-MNIST, 2017

Unimodal tasks in CV

Unimodal tasks in speech & audio

Unimodal know-how are hardly transferable●Tasks are specific to just one modality (e.g.
textual, visual, acoustic, etc).

●

Knowledge is learned from and applied toonly one modality (i.e. a visual algorithm canonly learn from and be applied to images).

A detour: cross-modal model
NIPS 2010, Cross-LDA

Erase the boundary between modalities- Tasks are shared and transferred between multiple modalities (soone algorithm can work with images and text and audio).
- Knowledge is learned from and applied to multiple modalities (so analgorithm can learn from textual data and apply that to visual data).

Paradigm shift from unimodal to multimodal
The rise of multimodal AI can be attributed to advances in two machinelearning techniques: Representation learning and transfer learning.
- Representation learning lets models create common representations
for all modalities.
- Transfer learning lets models first learn fundamental knowledge, andthen fine-tune on specific domains.

CLIP, DALLE, BLIP, Bark, GPT4
We will see more and more AI
applications move beyond
one data modality andleverage relationships
between different modalities

“An artificial intelligence systemtrained on words and sentences
alone will never approximate
human understanding.”
Y. Lecun in 2022 in AI And The Limits Of Language

Multimodal AI is the future,
but the ML ecosystem is not yet
suited for it.Agenda- Preliminary: multimodal AI- Opensource package: DocArray-Motivation
Representing data
Transiting data
Storing data
Retrieving data

- Multimodal at scale in production
This tutorial may require technical knowledge. Familiarity with Python 3.7+
concepts like data classes could be helpful.

DocArray for
representing, transiting,storing, searching
multimodal data

Representing multimodal data is a pain
- Lack of common interface for different modalities makes it difficult towork with multiple modalities at the same time.
- No easy way to represent unstructured and nested multimodal data.

Lack of common interface
Preprocess

ComputeStorage

AudioVideoImage
Text
3D mesh…

…

…

…

No easy way to represent
unstructured nested multimodal data
● Unstructured
document● Nested content● Different modalities(text, image, …)

DocArray way of representing multimodal
dataFrequent data transfer over network is
expensive
Multimodal data is processed by multiple models and models are usually deployed in a
distributed way.

Performant serialization is important
DocArray is designed to be “ready-to-wire” at anytime.

Binary serialization optimized for in-transit &at-rest

Binary serialization optimized for in-transit &at-rest

Storing nested data with databases iscomplicated- Complex and nested schema are not directly supported indatabases- Explosion in numbers of vector databases with different APIs but nouniversal client

DocArray way of storing data

DocArray way of storing data

Vector Search via a consistent API

Vector Search via a consistent API

Quick Recap●●●●

It’s like JSON, but for intensive computation.
It’s like numpy.ndarray, but for unstructured data.
It’s like pandas.DataFrame, but for nested and mixed media data with embeddings.
It’s like Protobuf, but for data scientists and deep learning engineers.

Quick Recap●●●●

It’s like JSON, but for intensive computation.
It’s like numpy.ndarray, but for unstructured data.
It’s like pandas.DataFrame, but for nested and mixed media data with embeddings.
It’s like Protobuf, but for data scientists and deep learning engineers.

Hands-on DocArrayInstall DocArray
To install DocArray (0.33), you can use the following command:
pip install "docarray[full]"

https://docs.docarray.org/

For old DocArray, more compatibility and features
pip install "docarray[full]"==0.21

Representing data - Document
At the heart of DocArray lies the concept of BaseDoc.

The following Python code deﬁnes a BannerDoc class that can be used to represent the data of a website banner:

Representing data - Document

Representing multimodal data with nested
structure
Let's say you want to represent a YouTube video in your application, perhaps to build a search system for YouTube
videos.
A YouTube video is not only composed of a video, but also has a title, description, thumbnail (and more, but let's
keep it simple).
All of these elements are from different modalities:
the title and description are text,
the thumbnail is an image,and the video itself is, well, a video.
DocArray lets you represent all of this multimodal data
in a single object.

Representing multimodal data with nested
structureRepresenting multimodal data with nested
structureRepresenting multimodal data with nested
structureRepresenting multimodal data with nested
structure
You see here that ImageDoc and VideoDoc are alsoBaseDoc, and they are later used inside another
BaseDoc`. This is what we call nested data
representation.
BaseDoc can be nested to represent any kind of data
hierarchy.

Representing multimodal data with nested
structure
You see here that ImageDoc and VideoDoc are also
BaseDoc, and they are later used inside another
BaseDoc`. This is what we call nested data
representation.
BaseDoc can be nested to represent any kind of data
hierarchy.

This representation can be used to send or
store data. You can even use it directly totrain a machine learning Pytorch model on
this representation.

Recap: representing multimodal data
- "Dataclass" look and feel, for defining the structure- Strong typing, for defining modality-Python built-in types
Numpy types
URI types
-

Text
Image
AudioVideo
Mesh3D
PointCloud3D-Tensor types-

ImageTensor
AudioTensor
VideoTensor
Embedding
Optional[]Representing an array of multimodal data
The fundamental building block of DocArray is the BaseDoc class which represents a single document, a single
datapoint.
However, in machine learning we often need to work with an array of documents, and an array of data points.

We introduce●DocList which is a Python list of BaseDocs●DocVec which is a column-based representation of BaseDocs

Example of DocList

Example of DocList

Example of DocList
DocList and DocVec are both AnyDocArrays. The following section will use DocList as an example, but the same applies to DocVec.

Example of DocList

Accessing member attribute at array level

Accessing member attribute at array level

DocList[DocType] syntax
DocList[DocType] creates a custom DocList that can only contain DocTypeDocuments.
Non-typing DocList forheterogeneous data

Strong-typing DocList forhomogeneous data

DocList vs DocVec
DocList is based on Python Lists. You can append, extend, insert, pop, and so on. In DocList, data is individually

owned by each BaseDoc collect just different Document references.
Use DocList when you want to be able to rearrange or re-rank your data. One ﬂaw of DocList is that none of thedata is contiguous in memory, so you cannot leverage functions that require contiguous data without ﬁrst copying
the data in a continuous array.
DocVec is a columnar data structure. DocVec is always an array of homogeneous Documents. The idea is thatevery attribute of the BaseDoc will be stored in a contiguous array: a column.

DocList vs DocVec
Let's say you want to embed a batch of Images:

def embed(image: NdArray['batch_size', 3, 224, 224]):...

DocList vs DocVecDocList vs DocVec

embed
embed

DocList vs DocVec

Access the view of Document in DocVec
If you access a document inside a DocVec you will get a document view. A document view is a view of the columnar
data structure which looks and behaves like a BaseDoc instance. It is a BaseDoc instance but with a different way toaccess the data.

Access the view of Document in DocVec
If you access a document inside a DocVec you will get a document view. A document view is a view of the columnar
data structure which looks and behaves like a BaseDoc instance. It is a BaseDoc instance but with a different way toaccess the data.

you should use DocVec when you need to work with
contiguous data, and you should use DocList whenyou need to rearrange or extend your data.

Storing & retrieving via Vector Database

Storing & retrieving via Vector DatabaseDocument Index: ORM for vector DBs
Document Index provides a uniﬁed interface to a number of vector databases.
You can think of Document Index as an ORM for vector databases.
Currently, DocArray supports the following vector databases:●●●●

Weaviate | Docs
Qdrant | Docs
Elasticsearch v7 and v8 | Docs
HNSWlib | Docs

*Old DocArray v0.21 supports Milvus, Redis, OpensearchConstruct a HNSWDocumentIndex
To use HnswDocumentIndex, you need to install extra dependencies with the following command: pip install "docarray[hnswlib]"Construct a HNSWDocumentIndex
To use HnswDocumentIndex, you need to install extra dependencies with the following command: pip install "docarray[hnswlib]"

In this code snippet, HnswDocumentIndex takes aschema of the form of MyDoc. The Document Index
then creates a column for each ﬁeld in MyDoc.

Construct a HNSWDocumentIndex
To use HnswDocumentIndex, you need to install extra dependencies with the following command: pip install "docarray[hnswlib]"

In this code snippet, HnswDocumentIndex takes aschema of the form of MyDoc. The Document Index
then creates a column for each ﬁeld in MyDoc.
The column types in the backend database aredetermined by the type hints of the document's
ﬁelds. Optionally, you can customize the databasetypes for every ﬁeld.

Construct a HNSWDocumentIndex
To use HnswDocumentIndex, you need to install extra dependencies with the following command: pip install "docarray[hnswlib]"

In this code snippet, HnswDocumentIndex takes aschema of the form of MyDoc. The Document Index
then creates a column for each ﬁeld in MyDoc.
The column types in the backend database aredetermined by the type hints of the document'sﬁelds. Optionally, you can customize the databasetypes for every ﬁeld.
Most vector databases need to know thedimensionality of the vectors that will be stored.
Here, that is automatically inferred from the type hintof the embedding ﬁeld: NdArray[128] means that thedatabase will store vectors with 128 dimensions.

Index dataIndex data

As you can see, DocList[MyDoc] andHnswDocumentIndex[MyDoc] are bothparameterized with MyDoc. This meansthat they share the same schema, and ingeneral, the schema of a Document
Index and the data that you want tostore need to have compatible schemas

Vector search

Vector search

Vector search

Vector searchHybrid search through the query builder
Document Index supports atomic operations for vector similarity search, text search and ﬁlter search.
To combine these operations into a single, hybrid search query, you can use the query builder that is accessible
through build_query():Customize vector DB configuration

Indexing andsearching
multimodal data
In the following example you can see acomplex schema that contains nestedDocuments. The YouTubeVideoDoc contains a
VideoDoc and an ImageDoc, alongside some"basic" ﬁelds:Indexing and searching multimodal data
You can perform search on any nesting level by using the dunder operator to specify the ﬁeld deﬁned in thenested data.

Nested DocList withsubindex
Documents can be nested by containing a DocList of otherdocuments, which is a slightly more complicated scenario
than the previous one.

In this case, the nested DocList will be represented as a new
sub-index (or table, collection, etc., depending on the
database backend), that is linked with the parent index(table, collection, ...).

Search by subindex

Transiting data over network
Sending via REST API/JSON -> Backend: FastAPI

Sending via gRPC/ws -> Backend: Jina microservice

Transiting data over network
Sending via REST API/JSON -> Backend: FastAPISending via gRPC/ws -> Backend: Jina microservice

Transiting data over network
Sending via gRPC/ws -> Backend: Jina microservice

Transiting data over network
Sending via gRPC/ws -> Backend: Jina microservice

Agenda- Preliminary: multimodal AI- Opensource package: DocArray-Motivation
Representing data
Transiting data
Storing data
Retrieving data

- Multimodal at scale in production
This tutorial may require technical knowledge. Familiarity with Python 3.7+
concepts like data classes could be helpful.

An end to end example
https://docs.docarray.org/how_to/multimodal_training_and_serving/Thanks foryourattention
jina.ai@JinaAI_han.xiao@jina.ai
Berlin · Beijing · Shenzhen