Package: pd2ml
What is it?pd2ml(pandasDataFrametoMySQLLoader) is a Python package that provides an efficient way to upload PandasDataFrametoMySQLand download from the database table into a DataFrame. The application of MySQL statementsLOAD DATA LOCAL INFILEandSELECT INTO OUTILEare the most essential reason for such efficient upload and download.It turns out that the advantages of pd2ml will gradually become apparent as the amount of data increases, so it will be a good assistant when dealing with massive amounts of data.To make a digression, the name pd2ml can be pronounced as pudumal (/'pudum蓴:/), which is a homonym for purdy cat in Chinese, and this is also the inspiration for the logo of this package.How to get itYou can install using the pip package manager by runningpipinstallpd2mlNote that it is necessary for pd2ml to run in thePython3environment.DependenciespandasNumPySQLAlchemyExamplesIt is veary easy to get started and easy to use.Firstly, start by reading a csv file into your pandas DataFrame, e.g. by usingimportpandasaspdimportsqlalchemyassaengine=sa.create_engine('mysql+pymysql://username:pwd@localhost:3306/db?charset=utf8&local_infile=1')df=pd.read_csv('stock.csv')To uploaddfto database, runfrompd2mlimportLoaderLoader(engine).load_to(df,'stock')To downloaddffrom databse, runfrompd2mlimportLoaderdf=Loader(engine).load_from('stock')print(df)In particular, it supports pandas extension, so you can also use it like thisimportpd2ml# Call it directly just like any other DataFrame's native methods.# load to dbdf.load_to('stock',engine)# load from dbdf=pd.DataFrame()df=df.load_from('stock',engine)print(df)What's more, it also works well with multi-processing or multi-threading# from multiprocessing.pool import ThreadPool as Poolfrommultiprocessing.poolimportPoolfromutilsimportsplit_dataframe,load_to,load_from_# load to dbwithLoader(engine):fordinsplit_dataframe(df,chunk_size):pool.apply_async(load_to,(d,'stock'))# load from dbwithLoader(engine):fortableintables:result=pool.apply_async(load_from_,(table,get_engine,))result_arr.append(result)For more details about examples, please seehere.TipsTo ensure that pd2ml works well, here are some tips and suggestionsIt is essential to add parametersinfile_local=1when connecting to the databaseTo make sure pd2ml works, it must be setsecure-file-priv=""in MySQL configuration filemy.iniormy.cnfMeanwhile, to maximize efficiency, the valueinnodb_buffer_pool_sizemay be adjusted appropriately in configuration filePerformanceTo estimate the performance of pd2ml, I did a preliminary rough test on my laptop.Test EnvironmentMySQL 5.7Windows 10ThinkPad of 4 cores and 8G RAMTest ItemsI created a table named stock, this table has 6 fields, of which 2 primary keys, 3 indexes. ThenUse pd2ml and multi-value insert SQL (INSERT INTO ... VALUES (), () ...) to insert 10000/100000/1000000 records 10 times into the empty table respectively, the average elapsed time is shown in the figureUse p2dml and multi-value replace SQL (REPLACE INTO ... VALUES (), () ...) to insert 10000/100000/1000000 records 10 times into the non-empty table, the average elapsed time is shown in the figureUse pd2ml and pandas native methodread_sql_tableto read 10000/100000/1000000 records 10 times from the table as DATAFRAME, the average elapsed time is shown in the figureConclusionAs the amount of data increases, the speed advantage of pd2ml becomes more and more obvious. The efficiency of writing into the database is at least 20% faster, and reading from the databse is improved three to four times.