Project: PolarDB (App Definition and Development / Database)
File: quickstart-development.md

This document explains the development process of PolarDB-X, covering code compilation, deployment and other processes. Note: This document is mainly for CentOS 7 and Ubuntu 20, other Linux distributions have similar processes. On your computer, get the source code of PolarDB-X from github repositories: Check out the `main` branch of each repository. Install dependencies on CentOS7 ```bash yum install cmake3 ln -s /usr/bin/cmake3 /usr/bin/cmake yum install centos-release-scl yum install devtoolset-7-gcc devtoolset-7-gcc-c++ devtoolset-7-binutils echo "source /opt/rh/devtoolset-7/enable" >>/etc/profile yum install make automake git openssl-devel ncurses-devel bison libaio-devel ``` Install dependencies on Ubuntu20 ```bash apt install -y gcc-7 g++-7 update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-7 60 \ --slave /usr/bin/g++ g++ /usr/bin/g++-7 update-alternatives --config gcc gcc --version g++ --version apt install make automake cmake git bison libaio-dev libncurses-dev libsasl2-dev libldap2-dev libssl-dev pkg-config ``` Compile && Install ```bash cd polardbx-engine wget https://boostorg.jfrog.io/artifactory/main/release/1.70.0/source/boost170_0.tar.gz mkdir extra/boost cp boost170_0.tar.gz extra/boost/ cmake . \ -DFORCEINSOURCEBUILD=ON \ -DCMAKEBUILDTYPE="Debug" \ -DSYSCONFDIR="/u01/mysql" \ -DCMAKEINSTALLPREFIX="/u01/mysql" \ -DMYSQL_DATADIR="/u01/mysql/data" \ -DWITHBOOST="./extra/boost/boost1700.tar.gz" make -j8 make install ``` ```shell git submodule update --init cd polardbx-sql/ mvn install -D maven.test.skip=true -D env=release tar zxvf target/polardbx-server-*-SNAPSHOT.tar.gz ``` ```shell cd polardbx-cdc mvn install -D maven.test.skip=true -D env=release tar zxvf polardbx-cdc-assemble/target/polardbx-binlog*.tar.gz ``` NOTE: PolarDB-X Engine is a derivative work of MySQL, the following will use mysql and polardbx-engine without distinction This step starts a mysql process that acts as the metadb and DN Refer to the mysql configuration file (my.cnf) in the appendix to modify it accordingly. By default, 4886 is used as mysql port and 32886 as private protocol port By default, /u01/my3306 is used as mysql data directory, you can change it to other directory NOTEYou should start DN under account other than root Start MySQL ```shell mkdir -p /u01/my3306/{data,log,run,tmp,mysql} /u01/mysql/bin/mysqld --defaults-file=my.cnf --initialize-insecure /u01/mysql/bin/mysqld --defaults-file=my.cnf ``` Once the mysql process is started, PolarDB-X can be initialized and the following configurations need to be prepared: metadb user`my_polarx` metadb database: `polardbxmetadb_polardbx` dnPasswordKey `asdf1234ghjk5678` PolarDB-X default root user`polardbx_root` PolarDB-X default password`123456`, you can reset it by `-S` NOTEYou should start CN under account other than root update conf/server.properties: ```text serverPort=8527 rpcPort=9090 metaDbAddr=127.0.0.1:4886 metaDbXprotoPort=32886 metaDbUser=my_polarx metaDbName=polardbxmetadb_polardbx instanceId=polardbx-polardbx ``` Initialize PolarDB-X: -I: Initialization mode -P: dnPasswordKey -d: DataNode's address list, in standalone mode, is the port and address of the previously started mysql process -r: metadb password -u: PolarDB-X root user -S: PolarDB-X root user password ```shell bin/startup.sh \ -I \ -P asdf1234ghjk5678 \ -d 127.0.0.1:4886:32886 \ -r "" \ -u polardbx_root \ -S "123456" ``` This step generates the internal and encrypted passwords, which need to be filled in the configuration file conf/server.properties for subsequent access: ```sql Generate password for user: my_polarx && M8%V5%K9^$5%oY0%yC0+&1!J7@8+R6) Encrypted password: DB84u4UkU/OYlMzu3aj9NFdknvxYgedFiW9z59bVnoc= Root user for polarx with password: polardbx_root && 123456 Encrypted password for polarx: H1AzXc2NmCs61dNjH5nMvA== ======== Paste following configurations to

! ======= metaDbPasswd=HMqvkvXZtT7XedA6t2IWY8+D7fJWIJir/mIY1Nf1b58= ``` Final Step, Run PolarDB-X: ```shell bin/startup.sh -P asdf1234ghjk5678 ``` Connect to PolarDB-X to verify, if you can connect, it means the database is started successfully, try some SQLs. ```sql mysql -h127.1 -P8527 -upolardbx_root ``` After the PolarDB-X is running, the PolarDB-X CDC component can be initialized, and the following configurations need to be prepared. metadb usersame as before `my_polarx` metadb database: same as before `polardbxmetadb_polardbx` metadb passwordsame as before metadb portsame as before `3306` dnPasswordKeysame as before `asdf1234ghjk5678` PolarDB-X usersame as before `polardbx_root` PolarDB-X passwordencrypted password of polarx `H1AzXc2NmCs61dNjH5nMvA==` PolarDB-X Portsame as before `8527` Max allocatable memory for cdcthe size, in MB, stands for memory limitation of cdc, see as `16000` in example below NOTEYou should start CDC under account other than root Modify the configuration file conf/config.properties and replace ${HOME} in the following example with the current user's home directory, e.g. /home/mysql ```shell useEncryptedPassword=true polardbx.instance.id=polardbx-polardbx mem_size=16000 metaDburl=jdbc:mysql://127.0.0.1:4886/polardbxmetadbpolardbx?useSSL=false metaDbusername=mypolarx metaDbPasswd=HMqvkvXZtT7XedA6t2IWY8+D7fJWIJir/mIY1Nf1b58= polarxurl=jdbc:mysql://127.0.0.1:8527/cdc_ polarxusername=polardbxroot polarx_password=H1AzXc2NmCs61dNjH5nMvA== dnPasswordKey=asdf1234ghjk5678 storage.persistBasePath=${HOME}/logs/rocksdb binlog.dir.path=${HOME}/binlog/ ``` Next, you can start the CDC daemon process with the command shown below. After starting, check the process status by `jps` command, CDC will have 3 processes, namely DaemonBootStrap, TaskBootStrap and DumperBootStrap. The system log of CDC will be output to ${HOME}/logs directory, and the global binary log will be output to directory configured by binlog.dir.path. ```shell bin/daemon.sh start ``` Connect to PolarDB-X, do some DDL or DML operations, then execute `SHOW BINARY LOGS` and `SHOW BINLOG EVENTS` commands to verify the global binlog, enjoy it! ```yaml [mysqld] socket = /u01/my3306/run/mysql.sock datadir = /u01/my3306/data tmpdir = /u01/my3306/tmp log-bin = /u01/my3306/mysql/mysql-bin.log log-bin-index = /u01/my3306/mysql/mysql-bin.index relay-log = /u01/my3306/mysql/slave-relay.log relay-log-info-file = /u01/my3306/mysql/slave-relay-log.info relay-log-index = /u01/my3306/mysql/slave-relay-log.index master-info-file = /u01/my3306/mysql/master.info slowquerylogfile = /u01/my3306/mysql/slowquery.log innodbdatahome_dir = /u01/my3306/mysql innodbloggrouphomedir = /u01/my3306/mysql port = 4886 loosepolarxport = 32886 loosegalaxyxport = 32886 loosepolarxmax_connections = 5000 looseserverid = 476984231 loose_cluster-info = 127.0.0.1:14886@1 loose_cluster-id = 5431 looseenablegts = 1 looseinnodbundo_retention=1800 core-file looselogsql_info=1 looselogsqlinfoindex=1 loose_indexstat=1 loose_tablestat=1 defaultauthenticationplugin=mysqlnativepassword binlog_checksum=CRC32 logbinusev1row_events=on explicitdefaultsfor_timestamp=OFF binlogrowimage=FULL binlogrowsquerylogevents=ON binlogstmtcache_size=32768 innodbdatafile_path=ibdata1:100M;ibdata2:200M:autoextend innodbbufferpool_instances=8 innodblogfilesingroup=4 innodblogfile_size=200M innodblogbuffer_size=200M innodbflushlogattrx_commit=1 innodbmaxdirtypagespct=60 innodbiocapacity_max=10000 innodbiocapacity=6000 innodbthreadconcurrency=64 innodbreadio_threads=8 innodbwriteio_threads=8 innodbopenfiles=615350 innodbfileper_table=1 innodbflushmethod=O_DIRECT innodbchangebuffering=none innodbadaptiveflushing=1 innodboldblocks_time=1000 innodbstatson_metadata=0 innodbusenative_aio=1 innodblockwait_timeout=50 innodbrollbackon_timeout=0 innodbpurgethreads=1 innodbstrictmode=1 innodbdisablesortfilecache=ON innodblruscan_depth=2048 innodbflushneighbors=0 innodbsyncarray_size=16 innodbprintall_deadlocks innodbchecksumalgorithm=CRC32 innodbmaxdirtypagespct_lwm=10 innodbbufferpool_size=500M concurrent_insert=2 delayedinserttimeout=300 slavetypeconversions="ALLNONLOSSY" slavenettimeout=4 skip-slave-start=OFF syncmasterinfo=10000 syncrelaylog_info=1 masterinforepository=TABLE relayloginfo_repository=TABLE relaylogrecovery=0 slaveexecmode=STRICT slaveparalleltype=LOGICAL_CLOCK looseslavepr_mode=TABLE slave-parallel-workers=32 server_id=193317851 binlogcachesize=32K maxbinlogcache_size=2147483648 looseconsensuslarge_trx=ON maxbinlogsize=500M maxrelaylog_size=500M relaylogpurge=OFF binlog-format=ROW sync_binlog=1 syncrelaylog=1 log-slave-updates=0 expirelogsdays=0 rplstopslave_timeout=300 slavecheckpointgroup=1024 slavecheckpointperiod=300 slavependingjobssizemax=1073741824 slaverowssearchalgorithms='TABLESCAN,INDEX_SCAN' slavesqlverify_checksum=OFF masterverifychecksum=OFF binlogtransactiondependency_tracking = WRITESET transactionwriteset_extraction = XXHASH64 gtid_mode=OFF enforcegtidconsistency=OFF looseconsensus-io-threadcnt=8 looseconsensus-worker-threadcnt=8 looseconsensusmaxdelayindex=10000 loose_consensus-election-timeout=10000 looseconsensusmaxpacketsize=131072 looseconsensusmaxlogsize=20M looseconsensusautoleadertransfer=ON looseconsensuslogcachesize=536870912 looseconsensusprefetchcachesize=268435456 looseconsensusprefetchwindowsize=100 looseconsensusautoresetmatch_index=ON loose_cluster-mts-recover-use-index=ON looseasynccommitthreadcount=128 loose_replicate-same-server-id=on loosecommitlockdonecount=1 loosebinlogorder_commits=OFF loose_cluster-log-type-node=OFF default-storage-engine=INNODB character-set-server=utf8 lowercasetable_names=1 skip-external-locking openfileslimit=615350 safe-user-create local-infile=1 sqlmode='NOENGINE_SUBSTITUTION' performance_schema=0 logslowadmin_statements=1 looselogslow_verbosity=full longquerytime=1 slowquerylog=0 general_log=0 looserdscheckcorefile_enabled=ON tabledefinitioncache=32768 eqrangeindexdivelimit=200 tableopencache_instances=16 tableopencache=32768 thread_stack=1024k binlogcachesize=32K netbufferlength=16384 threadcachesize=256 readrndbuffer_size=128K sortbuffersize=256K joinbuffersize=128K readbuffersize=128K max_connections=36000 maxuserconnections=35000 maxconnecterrors=65536 maxallowedpacket=1073741824 connect_timeout=8 netreadtimeout=30 netwritetimeout=60 back_log=1024 looseboostpk_access=1 logqueriesnotusingindexes=0 log_timestamps=SYSTEM innodbreadahead_threshold=0 looseiostate=1 looseusemyfs=0 loosedaemonmemcachedvaluesdelimiter=':;:' loosedaemonmemcached_option="-t 32 -c 8000 -p15506" innodb_doublewrite=1 ```