Package: timescale-vector
Timescale VectorPostgreSQL++ for AI Applications.Signup for Timescale
Vector:
Get 90 days free to try Timescale Vector on the Timescale cloud data
platform. There is no self-managed version at this time.Documentation: Learn the
key features of Timescale Vector and how to use them.Getting Started
Tutorial:
Learn how to use Timescale Vector for semantic search on a real-world
dataset.Learn
more:
Learn more about Timescale Vector, how it works and why we built it.If you prefer to use an LLM development or data framework, see Timescale
Vector’s integrations withLangChainandLlamaIndexInstallTo install the main library use:pipinstalltimescale_vectorWe also usedotenvin our examples for passing around secrets and
keys. You can install that with:pipinstallpython-dotenvBasic usageFirst, import all the necessary libraries:fromdotenvimportload_dotenv,find_dotenvimportosfromtimescale_vectorimportclientimportuuidfromdatetimeimportdatetime,timedeltaLoad up your PostgreSQL credentials. Safest way is with a .env file:_=load_dotenv(find_dotenv(),override=True)service_url=os.environ['TIMESCALE_SERVICE_URL']Next, create the client. In this tutorial, we will use the sync client.
But we have an async client as well (with an identical interface that
uses async functions).The client constructor takes three required arguments:namedescriptionservice_urlTimescale service URL / connection stringtable_nameName of the table to use for storing the embeddings. Think of this as the collection namenum_dimensionsNumber of dimensions in the vectorvec=client.Sync(service_url,"my_data",2)Next, create the tables for the collection:vec.create_tables()Next, insert some data. The data record contains:A UUID to uniquely identify the embeddingA JSON blob of metadata about the embeddingThe text the embedding representsThe embedding itselfBecause this data includes UUIDs which become primary keys, we ingest
with upserts.vec.upsert([\(uuid.uuid1(),{"animal":"fox"},"the brown fox",[1.0,1.3]),\(uuid.uuid1(),{"animal":"fox","action":"jump"},"jumped over the",[1.0,10.8]),\])You can now create a vector index to speed up similarity search:vec.create_embedding_index(client.TimescaleVectorIndex())Now, you can query for similar items:vec.search([1.0,9.0])[[UUID('73d05df0-84c1-11ee-98da-6ee10b77fd08'),
  {'action': 'jump', 'animal': 'fox'},
  'jumped over the',
  array([ 1. , 10.8], dtype=float32),
  0.00016793422934946456],
 [UUID('73d05d6e-84c1-11ee-98da-6ee10b77fd08'),
  {'animal': 'fox'},
  'the brown fox',
  array([1. , 1.3], dtype=float32),
  0.14489260377438218]]There are many search options which we will cover below in theAdvanced searchsection.As one example, we will return one item using a similarity search
constrained by a metadata filter.vec.search([1.0,9.0],limit=1,filter={"action":"jump"})[[UUID('73d05df0-84c1-11ee-98da-6ee10b77fd08'),
  {'action': 'jump', 'animal': 'fox'},
  'jumped over the',
  array([ 1. , 10.8], dtype=float32),
  0.00016793422934946456]]The returned records contain 5 fields:namedescriptionidThe UUID of the recordmetadataThe JSON metadata associated with the recordcontentsthe text content that was embeddedembeddingThe vector embeddingdistanceThe distance between the query embedding and the vectorYou can access the fields by simply using the record as a dictionary
keyed on the field name:records=vec.search([1.0,9.0],limit=1,filter={"action":"jump"})(records[0]["id"],records[0]["metadata"],records[0]["contents"],records[0]["embedding"],records[0]["distance"])(UUID('73d05df0-84c1-11ee-98da-6ee10b77fd08'),
 {'action': 'jump', 'animal': 'fox'},
 'jumped over the',
 array([ 1. , 10.8], dtype=float32),
 0.00016793422934946456)You can delete by ID:vec.delete_by_ids([records[0]["id"]])Or you can delete by metadata filters:vec.delete_by_metadata({"action":"jump"})To delete all records use:vec.delete_all()Advanced usageIn this section, we will go into more detail about our feature. We will
cover:Search filter options - how to narrow your search by additional
constraintsIndexing - how to speed up your similarity queriesTime-based partitioning - how to optimize similarity queries that
filter on timeSetting different distance types to use in distance calculationsSearch optionsThesearchfunction is very versatile and allows you to search for the
right vector in a wide variety of ways. We’ll describe the search option
in 3 parts:We’ll cover basic similarity search.Then, we’ll describe how to filter your search based on the
associated metadata.Finally, we’ll talk about filtering on time when time-partitioning
is enabled.Let’s use the following data for our example:vec.upsert([\(uuid.uuid1(),{"animal":"fox","action":"sit","times":1},"the brown fox",[1.0,1.3]),\(uuid.uuid1(),{"animal":"fox","action":"jump","times":100},"jumped over the",[1.0,10.8]),\])The basic query looks like:vec.search([1.0,9.0])[[UUID('7487af96-84c1-11ee-98da-6ee10b77fd08'),
  {'times': 100, 'action': 'jump', 'animal': 'fox'},
  'jumped over the',
  array([ 1. , 10.8], dtype=float32),
  0.00016793422934946456],
 [UUID('7487af14-84c1-11ee-98da-6ee10b77fd08'),
  {'times': 1, 'action': 'sit', 'animal': 'fox'},
  'the brown fox',
  array([1. , 1.3], dtype=float32),
  0.14489260377438218]]You could provide a limit for the number of items returned:vec.search([1.0,9.0],limit=1)[[UUID('7487af96-84c1-11ee-98da-6ee10b77fd08'),
  {'times': 100, 'action': 'jump', 'animal': 'fox'},
  'jumped over the',
  array([ 1. , 10.8], dtype=float32),
  0.00016793422934946456]]Narrowing your search by metadataWe have two main ways to filter results by metadata: -filtersfor
equality matches on metadata. -predicatesfor complex conditions on
metadata.Filters are more likely to be performant but are more limited in what
they can express, so we suggest using those if your use case allows it.FiltersYou could specify a match on the metadata as a dictionary where all keys
have to match the provided values (keys not in the filter are
unconstrained):vec.search([1.0,9.0],limit=1,filter={"action":"sit"})[[UUID('7487af14-84c1-11ee-98da-6ee10b77fd08'),
  {'times': 1, 'action': 'sit', 'animal': 'fox'},
  'the brown fox',
  array([1. , 1.3], dtype=float32),
  0.14489260377438218]]You can also specify a list of filter dictionaries, where an item is
returned if it matches any dict:vec.search([1.0,9.0],limit=2,filter=[{"action":"jump"},{"animal":"fox"}])[[UUID('7487af96-84c1-11ee-98da-6ee10b77fd08'),
  {'times': 100, 'action': 'jump', 'animal': 'fox'},
  'jumped over the',
  array([ 1. , 10.8], dtype=float32),
  0.00016793422934946456],
 [UUID('7487af14-84c1-11ee-98da-6ee10b77fd08'),
  {'times': 1, 'action': 'sit', 'animal': 'fox'},
  'the brown fox',
  array([1. , 1.3], dtype=float32),
  0.14489260377438218]]PredicatesPredicates allow for more complex search conditions. For example, you
could use greater than and less than conditions on numeric values.vec.search([1.0,9.0],limit=2,predicates=client.Predicates("times",">",1))[[UUID('7487af96-84c1-11ee-98da-6ee10b77fd08'),
  {'times': 100, 'action': 'jump', 'animal': 'fox'},
  'jumped over the',
  array([ 1. , 10.8], dtype=float32),
  0.00016793422934946456]]Predicatesobjects are defined by the name of the metadata key, an operator, and a
value.The supported operators are:==,!=,<,<=,>,>=The type of the values determines the type of comparison to perform. For
example, passing in"Sam"(a string) will do a string comparison while
a10(an int) will perform an integer comparison while a10.0(float) will do a float comparison. It is important to note that using a
value of"10"will do a string comparison as well so it’s important to
use the right type. Supported Python types are:str,int, andfloat. One more example with a string comparison:vec.search([1.0,9.0],limit=2,predicates=client.Predicates("action","==","jump"))[[UUID('7487af96-84c1-11ee-98da-6ee10b77fd08'),
  {'times': 100, 'action': 'jump', 'animal': 'fox'},
  'jumped over the',
  array([ 1. , 10.8], dtype=float32),
  0.00016793422934946456]]The real power of predicates is that they can also be combined using the&operator (for combining predicates with AND semantics) and|(for
combining using OR semantic). So you can do:vec.search([1.0,9.0],limit=2,predicates=client.Predicates("action","==","jump")&client.Predicates("times",">",1))[[UUID('7487af96-84c1-11ee-98da-6ee10b77fd08'),
  {'times': 100, 'action': 'jump', 'animal': 'fox'},
  'jumped over the',
  array([ 1. , 10.8], dtype=float32),
  0.00016793422934946456]]Just for sanity, let’s show a case where no results are returned because
or predicates:vec.search([1.0,9.0],limit=2,predicates=client.Predicates("action","==","jump")&client.Predicates("times","==",1))[]And one more example where we define the predicates as a variable and
use grouping with parenthesis:my_predicates=client.Predicates("action","==","jump")&(client.Predicates("times","==",1)|client.Predicates("times",">",1))vec.search([1.0,9.0],limit=2,predicates=my_predicates)[[UUID('7487af96-84c1-11ee-98da-6ee10b77fd08'),
  {'times': 100, 'action': 'jump', 'animal': 'fox'},
  'jumped over the',
  array([ 1. , 10.8], dtype=float32),
  0.00016793422934946456]]We also have some semantic sugar for combining many predicates with AND
semantics. You can pass in multiple 3-tuples toPredicates:vec.search([1.0,9.0],limit=2,predicates=client.Predicates(("action","==","jump"),("times",">",10)))[[UUID('7487af96-84c1-11ee-98da-6ee10b77fd08'),
  {'times': 100, 'action': 'jump', 'animal': 'fox'},
  'jumped over the',
  array([ 1. , 10.8], dtype=float32),
  0.00016793422934946456]]Filter your search by timeWhen usingtime-partitioning(see below). You can very efficiently
filter your search by time. Time-partitioning makes a timestamp embedded
as part of the UUID-based ID associated with an embedding. Let us first
create a collection with time partitioning and insert some data (one
item from January 2018 and another in January 2019):tpvec=client.Sync(service_url,"time_partitioned_table",2,time_partition_interval=timedelta(hours=6))tpvec.create_tables()specific_datetime=datetime(2018,1,1,12,0,0)tpvec.upsert([\(client.uuid_from_time(specific_datetime),{"animal":"fox","action":"sit","times":1},"the brown fox",[1.0,1.3]),\(client.uuid_from_time(specific_datetime+timedelta(days=365)),{"animal":"fox","action":"jump","times":100},"jumped over the",[1.0,10.8]),\])Then, you can filter using the timestamps by specifing auuid_time_filter:tpvec.search([1.0,9.0],limit=4,uuid_time_filter=client.UUIDTimeRange(specific_datetime,specific_datetime+timedelta(days=1)))[[UUID('33c52800-ef15-11e7-be03-4f1f9a1bde5a'),
  {'times': 1, 'action': 'sit', 'animal': 'fox'},
  'the brown fox',
  array([1. , 1.3], dtype=float32),
  0.14489260377438218]]AUUIDTimeRangecan specify a start_date or end_date or both(as in the example above).
Specifying only the start_date or end_date leaves the other end
unconstrained.tpvec.search([1.0,9.0],limit=4,uuid_time_filter=client.UUIDTimeRange(start_date=specific_datetime))[[UUID('ac8be800-0de6-11e9-889a-5eec84ba8a7b'),
  {'times': 100, 'action': 'jump', 'animal': 'fox'},
  'jumped over the',
  array([ 1. , 10.8], dtype=float32),
  0.00016793422934946456],
 [UUID('33c52800-ef15-11e7-be03-4f1f9a1bde5a'),
  {'times': 1, 'action': 'sit', 'animal': 'fox'},
  'the brown fox',
  array([1. , 1.3], dtype=float32),
  0.14489260377438218]]You have the option to define the inclusivity of the start and end dates
with thestart_inclusiveandend_inclusiveparameters. Settingstart_inclusiveto true results in comparisons using the>=operator, whereas setting it to false applies the>operator. By
default, the start date is inclusive, while the end date is exclusive.
One example:tpvec.search([1.0,9.0],limit=4,uuid_time_filter=client.UUIDTimeRange(start_date=specific_datetime,start_inclusive=False))[[UUID('ac8be800-0de6-11e9-889a-5eec84ba8a7b'),
  {'times': 100, 'action': 'jump', 'animal': 'fox'},
  'jumped over the',
  array([ 1. , 10.8], dtype=float32),
  0.00016793422934946456]]Notice how the results are different when we use thestart_inclusive=Falseoption because the first row has the exact
timestamp specified bystart_date.We’ve also made it easy to integrate time filters using thefilterandpredicatesparameters described above using special reserved key names
to make it appear that the timestamps are part of your metadata. We
found this useful when integrating with other systems that just want to
specify a set of filters (often these are “auto retriever” type
systems). The reserved key names are__start_dateand__end_datefor
filters and__uuid_timestampfor predicates. Some examples below:tpvec.search([1.0,9.0],limit=4,filter={"__start_date":specific_datetime,"__end_date":specific_datetime+timedelta(days=1)})[[UUID('33c52800-ef15-11e7-be03-4f1f9a1bde5a'),
  {'times': 1, 'action': 'sit', 'animal': 'fox'},
  'the brown fox',
  array([1. , 1.3], dtype=float32),
  0.14489260377438218]]tpvec.search([1.0,9.0],limit=4,predicates=client.Predicates("__uuid_timestamp",">",specific_datetime)&client.Predicates("__uuid_timestamp","<",specific_datetime+timedelta(days=1)))[[UUID('33c52800-ef15-11e7-be03-4f1f9a1bde5a'),
  {'times': 1, 'action': 'sit', 'animal': 'fox'},
  'the brown fox',
  array([1. , 1.3], dtype=float32),
  0.14489260377438218]]IndexingIndexing speeds up queries over your data. By default, we set up indexes
to query your data by the UUID and the metadata.But to speed up similarity search based on the embeddings, you have to
create additional indexes.Note that if performing a query without an index, you will always get an
exact result, but the query will be slow (it has to read all of the data
you store for every query). With an index, your queries will be
order-of-magnitude faster, but the results are approximate (because
there are no known indexing techniques that are exact).Nevertheless, there are excellent approximate algorithms. There are 3
different indexing algorithms available on the Timescale platform:
Timescale Vector index, pgvector HNSW, and pgvector ivfflat. Below are
the trade-offs between these algorithms:AlgorithmBuild speedQuery speedNeed to rebuild after updatestimescale vectorSlowFastestNopgvector hnswSlowestFasterNopgvector ivfflatFastestSlowestYesYou can seebenchmarkson our blog.We recommend using the Timescale Vector index for most use cases. This
can be created with:vec.create_embedding_index(client.TimescaleVectorIndex())Indexes are created for a particular distance metric type. So it is
important that the same distance metric is set on the client during
index creation as it is during queries. See thedistance typesection
below.Each of these indexes has a set of build-time options for controlling
the speed/accuracy trade-off when creating the index and an additional
query-time option for controlling accuracy during a particular query. We
have smart defaults for all of these options but will also describe the
details below so that you can adjust these options manually.Timescale Vector indexThe Timescale Vector index is a graph-based algorithm that uses theDiskANNalgorithm. You can read
more about it on ourblogannouncing its release.To create this index, run:vec.create_embedding_index(client.TimescaleVectorIndex())The above command will create the index using smart defaults. There are
a number of parameters you could tune to adjust the accuracy/speed
trade-off.The parameters you can set at index build time are:Parameter nameDescriptionDefault valuenum_neighborsSets the maximum number of neighbors per node. Higher values increase accuracy but make the graph traversal slower.50search_list_sizeThis is the S parameter used in the greedy search algorithm used during construction. Higher values improve graph quality at the cost of slower index builds.100max_alphaIs the alpha parameter in the algorithm. Higher values improve graph quality at the cost of slower index builds.1.0To set these parameters, you could run:vec.create_embedding_index(client.TimescaleVectorIndex(num_neighbors=50,search_list_size=100,max_alpha=1.0))You can also set a parameter to control the accuracy vs. query speed
trade-off at query time. The parameter is set in thesearch()function
using thequery_paramsargment. You can set thesearch_list_size(default: 100). This is the number of additional
candidates considered during the graph search at query time. Higher
values improve query accuracy while making the query slower.You can specify this value during search as follows:# vec.search([1.0, 9.0], limit=4, query_params=TimescaleVectorIndexParams(search_list_size=10))To drop the index, run:vec.drop_embedding_index()pgvector HNSW indexPgvector provides a graph-based indexing algorithm based on the popularHNSW algorithm.To create this index, run:vec.create_embedding_index(client.HNSWIndex())The above command will create the index using smart defaults. There are
a number of parameters you could tune to adjust the accuracy/speed
trade-off.The parameters you can set at index build time are:Parameter nameDescriptionDefault valuemRepresents the maximum number of connections per layer. Think of these connections as edges created for each node during graph construction. Increasing m increases accuracy but also increases index build time and size.16ef_constructionRepresents the size of the dynamic candidate list for constructing the graph. It influences the trade-off between index quality and construction speed. Increasing ef_construction enables more accurate search results at the expense of lengthier index build times.64To set these parameters, you could run:vec.create_embedding_index(client.HNSWIndex(m=16,ef_construction=64))You can also set a parameter to control the accuracy vs. query speed
trade-off at query time. The parameter is set in thesearch()function
using thequery_paramsargument. You can set theef_search(default:
40). This parameter specifies the size of the dynamic candidate list
used during search. Higher values improve query accuracy while making
the query slower.You can specify this value during search as follows:# vec.search([1.0, 9.0], limit=4, query_params=HNSWIndexParams(ef_search=10))To drop the index run:vec.drop_embedding_index()pgvector ivfflat indexPgvector provides a clustering-based indexing algorithm. Ourblog
postdescribes how it works in detail. It provides the fastest index-build
speed but the slowest query speeds of any indexing algorithm.To create this index, run:vec.create_embedding_index(client.IvfflatIndex())Note:ivfflat should never be created on empty tablesbecause it needs
to cluster data, and that only happens when an index is first created,
not when new rows are inserted or modified. Also, if your table
undergoes a lot of modifications, you will need to rebuild this index
occasionally to maintain good accuracy. See ourblog
postfor details.Pgvector ivfflat has alistsindex parameter that is automatically set
with a smart default based on the number of rows in your table. If you
know that you’ll have a different table size, you can specify the number
of records to use for calculating thelistsparameter as follows:vec.create_embedding_index(client.IvfflatIndex(num_records=1000000))You can also set thelistsparameter directly:vec.create_embedding_index(client.IvfflatIndex(num_lists=100))You can also set a parameter to control the accuracy vs. query speed
trade-off at query time. The parameter is set in thesearch()function
using thequery_paramsargument. You can set theprobes. This
parameter specifies the number of clusters searched during a query. It
is recommended to set this parameter tosqrt(lists)where lists is thenum_listparameter used above during index creation. Higher values
improve query accuracy while making the query slower.You can specify this value during search as follows:# vec.search([1.0, 9.0], limit=4, query_params=IvfflatIndexParams(probes=10))To drop the index, run:vec.drop_embedding_index()Time partitioningIn many use cases where you have many embeddings, time is an important
component associated with the embeddings. For example, when embedding
news stories, you often search by time as well as similarity (e.g.,
stories related to Bitcoin in the past week or stories about Clinton in
November 2016).Yet, traditionally, searching by two components “similarity” and “time”
is challenging for Approximate Nearest Neighbor (ANN) indexes and makes
the similarity-search index less effective.One approach to solving this is partitioning the data by time and
creating ANN indexes on each partition individually. Then, during
search, you can:Step 1: filter our partitions that don’t match the time predicate.Step 2: perform the similarity search on all matching partitions.Step 3: combine all the results from each partition in step 2, rerank,
and filter out results by time.Step 1 makes the search a lot more efficient by filtering out whole
swaths of data in one go.Timescale-vector supports time partitioning using TimescaleDB’s
hypertables. To use this feature, simply indicate the length of time for
each partition when creating the client:fromdatetimeimporttimedeltafromdatetimeimportdatetimevec=client.Async(service_url,"my_data_with_time_partition",2,time_partition_interval=timedelta(hours=6))awaitvec.create_tables()Then, insert data where the IDs use UUIDs v1 and the time component of
the UUID specifies the time of the embedding. For example, to create an
embedding for the current time, simply do:id=uuid.uuid1()awaitvec.upsert([(id,{"key":"val"},"the brown fox",[1.0,1.2])])To insert data for a specific time in the past, create the UUID using
ouruuid_from_timefunctionspecific_datetime=datetime(2018,8,10,15,30,0)awaitvec.upsert([(client.uuid_from_time(specific_datetime),{"key":"val"},"the brown fox",[1.0,1.2])])You can then query the data by specifying auuid_time_filterin the
search call:rec=awaitvec.search([1.0,2.0],limit=4,uuid_time_filter=client.UUIDTimeRange(specific_datetime-timedelta(days=7),specific_datetime+timedelta(days=7)))Distance metricsBy default, we use cosine distance to measure how similarly an embedding
is to a given query. In addition to cosine distance, we also support
Euclidean/L2 distance. The distance type is set when creating the client
using thedistance_typeparameter. For example, to use the Euclidean
distance metric, you can create the client with:vec=client.Sync(service_url,"my_data",2,distance_type="euclidean")Valid values fordistance_typearecosineandeuclidean.It is important to note that you should use consistent distance types on
clients that create indexes and perform queries. That is because an
index is only valid for one particular type of distance measure.Please note the Timescale Vector index only supports cosine distance at
this time.LangChain integrationLangChainis a popular framework for
development applications powered by LLMs. Timescale Vector has a native
LangChain integration, enabling you to use Timescale Vector as a
vectorstore and leverage all its capabilities in your applications built
with LangChain.Here are resources about using Timescale Vector with LangChain:Getting started with LangChain and Timescale
Vector:
You’ll learn how to use Timescale Vector for (1) semantic search, (2)
time-based vector search, (3) self-querying, and (4) how to create
indexes to speed up queries.PostgreSQL Self
Querying:
Learn how to use Timescale Vector with self-querying in LangChain.LangChain template: RAG with conversational
retrieval:
This template is used for conversational retrieval, which is one of
the most popular LLM use-cases. It passes both a conversation history
and retrieved documents into an LLM for synthesis.LangChain template: RAG with time-based search and self-query
retrieval:This
template shows how to use timescale-vector with the self-query
retriver to perform hybrid search on similarity and time. This is
useful any time your data has a strong time-based component.Learn more about Timescale Vector and
LangChainLlamaIndex integration[LlamaIndex] is a popular data framework for connecting custom data
sources to large language models (LLMs). Timescale Vector has a native
LlamaIndex integration, enabling you to use Timescale Vector as a
vectorstore and leverage all its capabilities in your applications built
with LlamaIndex.Here are resources about using Timescale Vector with LlamaIndex:Getting started with LlamaIndex and Timescale
Vector:
You’ll learn how to use Timescale Vector for (1) similarity
search, (2) time-based vector search, (3) faster search with indexes,
and (4) retrieval and query engine.Time-based
retrieval: Learn
how to power RAG applications with time-based retrieval.Llama Pack: Auto Retrieval with time-based
search:
This pack demonstrates performing auto-retrieval for hybrid search
based on both similarity and time, using the timescale-vector
(PostgreSQL) vectorstore.Learn more about Timescale Vector and
LlamaIndexPgVectorizePgVectorize enables you to create vector embeddings from any data that
you already have stored in PostgreSQL. You can get more background
information in ourblog
postannouncing this feature, as well as a“how we built
in”post going into the details of the design.To create vector embeddings, simply attach PgVectorize to any PostgreSQL
table, and it will automatically sync that table’s data with a set of
embeddings stored in Timescale Vector. For example, let’s say you have a
blog table defined in the following way:importpsycopg2fromlangchain.docstore.documentimportDocumentfromlangchain.text_splitterimportCharacterTextSplitterfromtimescale_vectorimportclient,pgvectorizerfromlangchain.embeddings.openaiimportOpenAIEmbeddingsfromlangchain.vectorstores.timescalevectorimportTimescaleVectorfromdatetimeimporttimedeltawithpsycopg2.connect(service_url)asconn:withconn.cursor()ascursor:cursor.execute('''CREATE TABLE IF NOT EXISTS blog (id              SERIAL PRIMARY KEY NOT NULL,title           TEXT NOT NULL,author          TEXT NOT NULL,contents        TEXT NOT NULL,category        TEXT NOT NULL,published_time  TIMESTAMPTZ NULL --NULL if not yet published);''')You can insert some data as follows:withpsycopg2.connect(service_url)asconn:withconn.cursor()ascursor:cursor.execute('''INSERT INTO blog (title, author, contents, category, published_time) VALUES ('First Post', 'Matvey Arye', 'some super interesting content about cats.', 'AI', '2021-01-01');''')Now, say you want to embed these blogs in Timescale Vector. First, you
need to define anembed_and_writefunction that takes a set of blog
posts, creates the embeddings, and writes them into TimescaleVector. For
example, if using LangChain, it could look something like the following.defget_document(blog):text_splitter=CharacterTextSplitter(chunk_size=1000,chunk_overlap=200,)docs=[]forchunkintext_splitter.split_text(blog['contents']):content=f"Author{blog['author']}, title:{blog['title']}, contents:{chunk}"metadata={"id":str(client.uuid_from_time(blog['published_time'])),"blog_id":blog['id'],"author":blog['author'],"category":blog['category'],"published_time":blog['published_time'].isoformat(),}docs.append(Document(page_content=content,metadata=metadata))returndocsdefembed_and_write(blog_instances,vectorizer):embedding=OpenAIEmbeddings()vector_store=TimescaleVector(collection_name="blog_embedding",service_url=service_url,embedding=embedding,time_partition_interval=timedelta(days=30),)# delete old embeddings for all ids in the work queue. locked_id is a special column that is set to the primary key of the table being# embedded. For items that are deleted, it is the only key that is set.metadata_for_delete=[{"blog_id":blog['locked_id']}forbloginblog_instances]vector_store.delete_by_metadata(metadata_for_delete)documents=[]forbloginblog_instances:# skip blogs that are not published yet, or are deleted (in which case it will be NULL)ifblog['published_time']!=None:documents.extend(get_document(blog))iflen(documents)==0:returntexts=[d.page_contentfordindocuments]metadatas=[d.metadatafordindocuments]ids=[d.metadata["id"]fordindocuments]vector_store.add_texts(texts,metadatas,ids)Then, all you have to do is run the following code in a scheduled job
(cron job, Lambda job, etc):# this job should be run on a schedulevectorizer=pgvectorizer.Vectorize(service_url,'blog')whilevectorizer.process(embed_and_write)>0:passEvery time that job runs, it will sync the table with your embeddings.
It will sync all inserts, updates, and deletes to an embeddings table
calledblog_embedding.Now, you can simply search the embeddings as follows (again, using
LangChain in the example):embedding=OpenAIEmbeddings()vector_store=TimescaleVector(collection_name="blog_embedding",service_url=service_url,embedding=embedding,time_partition_interval=timedelta(days=30),)res=vector_store.similarity_search_with_score("Blogs about cats")res[(Document(page_content='Author Matvey Arye, title: First Post, contents:some super interesting content about cats.', metadata={'id': '4a784000-4bc4-11eb-855a-06302dbc8ce7', 'author': 'Matvey Arye', 'blog_id': 1, 'category': 'AI', 'published_time': '2021-01-01T00:00:00+00:00'}),
  0.12595687795193833)]DevelopmentThis project is developed withnbdev. Please
see that website for the development process.