Package: pandas-upsert-to-mysql
Strongly suggest usingpangres, which is much more efficient and elegant than this package.pandas_upsert_to_mysqlEnhancedto_sqlmethod in pandas DataFrame, for MySQL database only. It provides a relatively convenientupsert(insert or update) feature inplementation through temporary table.Whether a record needs to be update or not is determined by primary key or unique constraintThe MySQL database table structure requires to be well designed (need to use SQLAlchemy)The primary key must be MySQL auto-increment. Not allow composite primary key mixing auto-increment and other columnsInstallationpipinstallpandas_upsert_to_mysqlUsage and ExampleLet's use an order table as instance. Here therow_idis the auto-incremented primary key.order_idandproduct_idmake up of the unique contraint (a single order can have more than one kind of product).-- Here we use native SQL to create the table for illustration convenience. In the actual pratice we need to use SQLAlchemy.CREATETABLE`order_info`(`row_id`int(11)NOTNULLAUTO_INCREMENTCOMMENT'auto_incremented_ID',`order_id`varchar(5)NOTNULLDEFAULT'-9999'COMMENT'order_id',`product_id`varchar(5)NOTNULLDEFAULT'-9999'COMMENT'product_id',`qty`int(11)DEFAULTNULLCOMMENT'purchase_quantity',`refund_qty`int(11)DEFAULTNULLCOMMENT'refund_quantity',`update_time`timestampNOTNULLDEFAULTCURRENT_TIMESTAMPONUPDATECURRENT_TIMESTAMPCOMMENT'last_update_time',`create_time`timestampNOTNULLDEFAULTCURRENT_TIMESTAMPCOMMENT'first_create_time',PRIMARYKEY(`row_id`),UNIQUEKEY`main`(`order_id`,`product_id`))ENGINE=InnoDBDEFAULTCHARSET=utf8COMMENT='Order Info'As time moving, this table needs two operations as below:New orders: insert new recordsExisted orders apply refund: update therefund_qty('qty' for quantity) field of the specific ordersWe insert the base data first. This step can be easily done with pandas official native methodto_sqlrow_idorder_idproduct_idqtyrefund_qtyupdate_timecreate_time1A0001PD1001002020-06-26 11:11:552020-06-26 11:11:552A0002PD2002002020-06-26 11:11:552020-06-26 11:11:553A0002PD2012202020-06-26 11:11:552020-06-26 11:11:55Assume the database already has the data above. Now we need to upsert the new data below. Pay attention that A0002-PD201 now has two refund_qty, and we have the new record A0003-PD300.order_idproduct_idqtyrefund_qtyA0001PD100100A0002PD200200A0002PD201222A0003PD300300# 'table' is a module containing pre-defined SQLAlchemy ORM table structure classes# 'engine' is a sqlalchemy.engine.Engine which needs you to define yourself. We use it to connect to the target MySQL database. It has the same requirements with the paramater 'con' in pandas.DataFrame.to_sql# Both 'table' and 'engine' here is just the instance in package. You should define them by yourself in actual usageimportpandas_upsert_to_mysql.tableastablefrompandas_upsert_to_mysqlimportUpsertfromconnectionimportConnectorengine=Connector(schema='dev').get_engine()Upsert(engine=engine).to_mysql(df=table.ExampleOrderTable.new_df,target_table=table.Order,temp_table=table.OrderTemp,if_record_exists='update')Then we can get the result. Be aware that theupdate_timeonly changed in the record whose row_id=3, completely as expected.row_idorder_idproduct_idqtyrefund_qtyupdate_timecreate_time1A0001PD1001002020-06-26 11:11:552020-06-26 11:11:552A0002PD2002002020-06-26 11:11:552020-06-26 11:11:553A0002PD2012222020-06-26 11:13:192020-06-26 11:11:554A0003PD3003002020-06-26 11:13:192020-06-26 11:13:19CaveatsAny conlumn in the unique constraint must not have null value, otherwise the update will violate the constraint and cause duplicates. This is considered as a bug#8173of MySQL with long history, which means before we write the DataFrame to the database, we should assign a default value for each column in unique constraint to replace the possible null values.Implement usingsession scopeof SQLAlchemy. But since the author is not a professional engineer, I still can not gurantee the reliability in concurrent conditionOnly tested in pandas >= 1.0.3 and MySQL 5.7 innodb environment. The transaction isolation level is the default REPEATABLE-READOriginPandas official (up to 1.0.5 version)to_sqlmethod does not implement upsert feature. Its parameterif_existhas avaliable values as below:if_exists: {‘fail’, ‘replace’, ‘append’}, default ‘fail’How to behave if the table already exists.fail: Raise a ValueError.replace: Drop the table before inserting new values.append: Insert new values to the existing table.Notice that thereplacehere takes effect on the wholetablerather than each specific row, which means the native method can only truncate the whole table and re-insert the entire DataFrame.Demand for the upsert feature has been discussed in the official repo long before, seeissue #14553. But the discussion has lasted for almost 4 years (up to 1.0.5 version), and the officials still can not deliver this feature in a stable release. For now it is only called "may be fixedby#29636"In those discussion, the major consideration of the officials are:Consistent support for multiple databases, but different databases can have huge difference on their native upsert praticeAlso due to the disparity of databases, they would only support upsert by primary key in the first plan, not by unique constraintsThe conciseness of APITherefore it can be expected that, the officials may be conservative on their first implementation for the balance of simplicity ("to protect users that have a poorly designed database") and function abundance: only support upsert by primary key for example. However, since the auto-increment has to be (part of) the primary key in MySQL, you won't be able to distinguish those duplicate records which need to be updated once you define an auto-increment. One possible solution is that the primary key contains only  an auto-increment, other columns to decide a unique record serve as the unique constraint. For this goal, this repo gives an unofficial solution only for MySQL, and it is highly probably not a best pratice.Up to the last commit of this repo, pandas 1.0.5 version's nativeto_sqlmethod still does not supportupsert.Referencepangres: support multiple databases; upsert by primary key, require to set the primary key as the DataFrame index (In fact it is completely compatible with unique constraint); implement without temporary table, much faster.pandabase: support multiple databases; upsert by primary key, require to set the primary key as the DataFrame index; seems not supporting MySQLpandas-to-mysqlpandas-sqlPandas-to_sql-upsert