Package: meilisync
meilisyncIntroductionRealtime sync data from MySQL/PostgreSQL/MongoDB to Meilisearch.There is also a web admin dashboard for meilisyncmeilisync-admin.InstallJust install from pypi:pipinstallmeilisyncUse docker (Recommended)You can use docker to runmeilisync:version:"3"services:meilisync:image:long2ice/meilisyncvolumes:-./config.yml:/meilisync/config.ymlrestart:alwaysPrerequisitesMySQL:binlog_format = ROW, use binary log.PostgreSQL:wal_level = logicaland installwal2jsonextension, use logical replication.MongoDB: enable replica set mode, use change stream.Quick StartIf you runmeilisyncwithout any arguments, it will try to load the configuration fromconfig.ymlin the current
directory.❯meilisync--helpUsage:meilisync[OPTIONS]COMMAND[ARGS]...

╭─Options──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│--config-cTEXTConfigfilepath[default:config.yml]│
│--install-completionInstallcompletionforthecurrentshell.│
│--show-completionShowcompletionforthecurrentshell,tocopyitorcustomizetheinstallation.│
│--helpShowthismessageandexit.│
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─Commands─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│checkCheckwhetherthedatainthedatabaseisconsistentwiththedatainMeilisearch│
│refreshRefreshalldatabyswapindex│
│startStartmeilisync│
│versionShowmeilisyncversion│
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯Start syncStart sync data from MySQL to Meilisearch:❯meilisyncstart2023-03-0708:37:25.656|INFO|meilisync.main:_:86-Startincrementsyncdatafrom"mysql"toMeilisearch...Refresh syncRefresh all data by swap index:❯meilisyncrefresh-ttestBefore refresh, you need stop the sync process first to avoid data inconsistency.Check syncCheck whether the data count in the database is consistent with the data in Meilisearch:❯meilisynccheck-ttestConfigurationHere is an example configuration file:debug:trueplugins:-meilisync.plugin.Pluginprogress:type:filesource:type:mysqlhost:192.168.123.205port:3306user:rootpassword:"123456"database:beautymeilisearch:api_url:http://192.168.123.205:7700api_key:insert_size:1000insert_interval:10sync:-table:collectionindex:beauty-collectionsplugins:-meilisync.plugin.Pluginfull:truefields:id:title:description:category:-table:pictureindex:beauty-picturesfull:truefields:id:description:category:sentry:dsn:""environment:"production"debug (optional)Enable debug mode, default isfalse, if you want to see more logs, you can set it totrue.plugins (optional)The plugins are used to customize the data before or after insert to Meilisearch and the plugins is a list of python
modules.Which is a python class withpre_eventandpost_eventmethods, thepre_eventmethod is called before insert to
Meilisearch, thepost_eventmethod is called after insert to Meilisearch.classPlugin:is_global=Falseasyncdefpre_event(self,event:Event):logger.debug(f"pre_event:{event}, is_global:{self.is_global}")returneventasyncdefpost_event(self,event:Event):logger.debug(f"post_event:{event}, is_global:{self.is_global}")returneventTheis_globalis used to indicate whether the plugin instance is global, if set toTrue, the plugin instance will be
created only once, otherwise, the plugin instance will be created for each event.progressThe progress is used to record the last sync position, such as binlog position for MySQL.type:fileorredis, if set to file, another optionpathis required.path: the file path to store the progress, default isprogress.json.key: the redis key to store the progress, default ismeilisync:progress.dsn: the redis dsn, default isredis://localhost:6379/0.sourceSource database configuration, currently only support MySQL and PostgreSQL and MongoDB.type:mysqlorpostgresormongo.server_id: the server id for MySQL binlog, default is1.database: the database name.other keys: the database connection arguments, MySQL seeasyncmy, PostgreSQL
seepsycopg2, MongoDB seemotor.meilisearchMeilisearch configuration.api_url: the Meilisearch API URL.api_key: the Meilisearch API key.insert_size: insert after collecting this many documents, optional.insert_interval: insert after this many seconds have passed, optional.If netherinsert_sizenorinsert_intervalis set, it will insert each document immediately.If you prefer performance, just set and increaseinsert_sizeandinsert_interval. The insert will be made as long as
one of the conditions is met.syncThe sync configuration, you can add multiple sync tasks.table: the database table name or collection name.index: the Meilisearch index name, if not set, it will use the table name.full: whether to do a full sync, default isfalse.fields: the fields to sync, if not set, it will sync all fields. The key is table field name, the value is the
Meilisearch field name, if not set, it will use the table field name.plugins: the table level plugins, optional.sentry (optional)Sentry configuration.dsn: the sentry dsn.environment: the sentry environment, default isproduction.LicenseThis project is licensed under theApache-2.0License.