Package: sinker
Sinker: Synchronize Postgres to ElasticsearchWhat are yousinking about?What is Sinker?Sinker is middleware that synchronizes relational data from a Postgres database to Elasticsearch.
It is simple to operate, requires minimal RAM, and handles arbitrarily complex schemas.
Sinker is built byParadigm, and is licensed under the Apache and MIT licenses.For ExampleIn Postgres, you might have a normalized schema like this:A Student and a Teacher refer to a PersonA Course is taught by a TeacherStudents have and belong to many Courses through the Enrollment join tableIn Elasticsearch, you might want to index the Course data in an index calledcourseslike this:{"name":"Reth","description":"How to build a modern Ethereum node","teacher":{"salary":100000.0,"person":{"name":"Prof Georgios"}},"enrollments":[{"grade":3.14,"student":{"gpa":3.99,"person":{"name":"Loren"}}},{"grade":3.50,"student":{"gpa":4.00,"person":{"name":"Abigail"}}}]}Now you can easily query Elasticsearch for courses taught by Prof Georgios, or students with high GPAs named Loren.
To do this, you need to do two things reliably:Denormalize the normalized data from the five Postgres tables into a single Elasticsearch document with the Course as
the parent and the other four tables nested appropriately inside it.Keep the Elasticsearch document in sync with the Postgres data, so that if Abigail changes her name in the database
to Abby, it's reflected in theCourse->Enrollments->Student->Person.namefield.How it WorksSinker transforms the normalized Postgres data into JSON documents stored in a simple key-value materialized view where
the key is the Elasticsearch document ID and the value is the JSON document to be stored in Elasticsearch.Sinker creates triggers on the Postgres tables that you want to synchronize (e.g., the five tables in the example
above). When a row is inserted, updated, or deleted in any of these tables, the trigger schedules the materialized view
to be refreshed at the next interval.The changes to the materialized view are sent to a logical replication slot. Sinker reads from this slot and indexes the
documents in Elasticsearch.You define the query behind the materialized view, so you can denormalize the data however you want, filter out unwanted
documents, transform some fields, etc. If you can express it in SQL, you can build your materialized view around it.You also configure the Elasticsearch index settings and mappings however you like.InstallationpipinstallsinkerEnvironment VariablesHere are some of the environment variables that you'll want to set:Environment VariableValueSINKER_DEFINITIONS_PATH.SINKER_SCHEMApublicSINKER_POLL_INTERVAL10SINKER_LOG_LEVELDEBUGPGPASSWORDsecret!PGHOSTlocalhostPGUSERdevPGDATABASEdev_dbELASTICSEARCH_HOSTlocalhostELASTICSEARCH_SCHEMEhttpSee sinker/settings.py for the full list.ConfigurationSinker's main configuration fileviews_to_indices.jsonspecifies the mapping between the root Postgres materialized
view names and the Elasticsearch indexes that will get populated by them, e.g.:{"person_mv":"people","course_mv":"courses"}This tells Sinker to define a Postgres materialized view calledperson_mvbased on the query in theperson_mv.sqlfile and an Elasticsearch index calledpeoplebased on the settings and mappings in thepeople.jsonfile. It will
then populate thepeopleindex with the documents from theperson_mvmaterialized view. It will then do the same for
thecourse_mvmaterialized view and thecoursesindex.Materialized View ConfigurationTheperson_mvmaterialized view is defined by the SQL in theperson_mv.sqlfile, e.g.:selectid,json_build_object('name',"name")as"person"from"person"Thecourse_mvSQL is more complex, but you can see how it denormalizes the data from the five tables into a single
JSON document:selectid,json_build_object('name',"name",'description',"description",'teacher',(selectjson_build_object('salary',"salary",'person',(selectjson_build_object('name',"name")frompersonwhereperson.id=person_id))fromteacherwhereteacher.id=teacher_id),'enrollments',(selectjson_agg(json_build_object('grade',"grade",'student',(selectjson_build_object('gpa',"gpa",'person',(selectjson_build_object('name',"name")frompersonwhereperson.id=person_id))fromstudentwherestudent.id=student_id)))fromenrollmentwhereenrollment.course_id=course.id))as"course"from"course";Index ConfigurationThe Elasticsearch index configurations are stored in thepeople.jsonandcourses.jsonfiles, e.g.:{"mappings":{"dynamic":"strict","properties":{"name":{"type":"keyword"}}},"settings":{"index":{"number_of_shards":"1","number_of_replicas":"0"}}}Usingstrictmappings helps ensure the JSON document structure from the materialized view matches what Elasticsearch
expects in the index.{"mappings":{"dynamic":"strict","properties":{"description":{"type":"text"},"enrollments":{"properties":{"grade":{"type":"float"},"student":{"properties":{"gpa":{"type":"float"},"person":{"properties":{"name":{"type":"text"}}}}}}},"name":{"type":"text"},"teacher":{"properties":{"person":{"properties":{"name":{"type":"text"}}},"salary":{"type":"float"}}}}},"settings":{"index":{"number_of_shards":"1","number_of_replicas":"0"}}}Dry RunBefore running sinker, do a dry run on one of your view-index mappings to verify everything is defined correctly:Execute the SQL query withLIMIT 1in the materialized view definition file to see what a sample JSON record will
look like.Create the destination Elasticsearch index with the settings and mappings in the index configuration file.In Kibana or via cURL,PUTthe sample JSON record into the Elasticsearch index.Verify the record is indexed as you expect in Elasticsearch.RunningOnce you have the environment variables and configuration files set up, you can run Sinker with:sinkerPerformanceOnce you have Sinker running, you may well want it to run faster. Here are some things you can do to improve
performance:DecreaseSINKER_POLL_INTERVAL. This will make Sinker refresh the materialized views more
frequently (and thus keep Elasticsearch in closer sync), but it will also increase the load on the database. Note
that the materialized views are only refreshed when one of the underlying tables has changed, so this won't increase
the load on the database if there are no changes.Increase thePGCHUNK_SIZE. This will make Sinker read more rows from the logical replication slot at a time, which
will reduce the number of round trips to the database. However, it will also increase the memory usage of Sinker.Increase theELASTICSEARCH_CHUNK_SIZE. This will make Sinker index more documents in a single Elasticsearch
bulk request, which will reduce the number of round trips to Elasticsearch. However, it will also increase the memory
usage of Sinker and the CPU load on the Elasticsearch cluster.RunEXPLAIN ANALYZEon your materialized view queries to see if you can optimize them (e.g., by adding indexes on
the foreign keys).DevelopingBefore getting started, make sure you havepoetryanddocker-composeinstalled.%docker-compose--version
DockerComposeversionv2.15.1
%poetry--version
Poetry(version1.4.0)Clone the repo:gitclonegit@github.com:paradigmxyz/sinker.gitcdsinkerInstall dependencies:poetryinstallSpin up Postgres and Elasticsearch:docker-compose--env-file=.env.testup-dRun tests:poetryrunpytest-sOperationsDockerTo run Sinker as a Docker container, you can use thedocker/files in this repo as a starting point.
Bundle your views and indices configuration files into the Docker image, set up your environment variables, and deploy.Monitoring and AlertingSetting up monitoring and alerting will give you confidence that Sinker is functioning properly.
For instance, you can periodically check that the row counts in Postgres match the expected document counts in
Elasticsearch.ContributingContributions are welcome! Please open an issue or submit a pull request.AcknowledgementsSinker was inspired bypgsyncanddebezium. Each project
takes a different approach to the problem, so check them out to see which one is best for you.