Package: bcpandas
bcpandasHigh-level wrapper around BCP for high performance data transfers between pandas and SQL Server. No
knowledge of BCP required!! (pronouncedBEE-CEE-Pandas)QuickstartIMPORTANT - Read vs. WriteBenchmarksto_sqlWhy not just use the new pandasmethod='multi'?read_sqlRequirementsDatabasePython UserInstallationUsageCredential/Connection objectRecommended UsageKnown IssuesTroubleshootingAll quote characters appear in the dataAll delimiter characters appear in the dataWrite to database fails due to spaces in columnsBackgroundExisting SolutionsDesign and ScopeTestingTesting RequirementsWhat Is Tested?Testing ImplementationCI/CDContributingGitHub Stars TrendQuickstartIn[1]:importpandasaspd...:importnumpyasnp...:...:frombcpandasimportSqlCreds,to_sqlIn[2]:creds=SqlCreds(...:'my_server',...:'my_db',...:'my_username',...:'my_password'...:)In[3]:df=pd.DataFrame(...:data=np.ndarray(shape=(10,6),dtype=int),...:columns=[f"col_{x}"forxinrange(6)]...:)In[4]:dfOut[4]:col_0col_1col_2col_3col_4col_50412886060293753801155557065266192517536754148497567536751445655271435297471201701246726029433688135768813907274595655371033424333661922877333586029427648816263571046553710475367377077980642263375367327602281294922156357104701245167503057536741734012472746106734014162260367274612707799968813876029428766192436226041688137865537107209065602941586881378655371072090657536743727458866192489622603072090656619231688138072746123014770In[5]:to_sql(df,'my_test_table',creds,index=False,if_exists='replace')In[6]:df2=pd.read_sql_table(table_name='my_test_table',con=creds.engine)In[7]:df2Out[7]:col_0col_1col_2col_3col_4col_50412886060293753801155557065266192517536754148497567536751445655271435297471201701246726029433688135768813907274595655371033424333661922877333586029427648816263571046553710475367377077980642263375367327602281294922156357104701245167503057536741734012472746106734014162260367274612707799968813876029428766192436226041688137865537107209065602941586881378655371072090657536743727458866192489622603072090656619231688138072746123014770IMPORTANT - Read vs. WriteThe big speedup benefit of bcpandas is in theto_sqlfunction, as the benchmarks below show.
However, the bcpandasread_sqlfunction actually performsslowerthan the pandas equivalent.
Therefore, the bcpandasread_sqlfunction was deprecated in v5.0 and has now been removed in
v6.0+. To read datafromSQL to pandas, use the native pandas methodpd.read_sql_tableorpd.read_sql_query.BenchmarksSee figures below. All code is in the/benchmarksdirectory. To run the benchmarks, from the root
directory of this repository, runpython benchmarks/benchmark.py main --helpand fill in the
command line options that are presented.Running this will outputPNG image of the graphJSON file of the benchmark dataJSON file with the environment details of the machine that was used to generate itto_sqlI didn't bother including the pandas non-multiinsertversion here because it just takes way too
longWhy not just use the new pandasmethod='multi'?Because it is still much slowerBecause you are forced to set thechunksizeparameter to a very small number for it to work -
generally a bit less then2100/<number of columns>. This is because SQL Server can only accept
up to 2100 parameters in a query. Seehereandherefor more discussion on this, and
the recommendation to use a bulk insert tool such as BCP. It seems that SQL Server simply didn't
design the regularINSERTstatement to support huge amounts of data.read_sqlAs you can see, pandas native clearly wins hereRequirementsDatabaseAny version of Microsoft SQL Server. Can be installed on-prem, in the cloud, on a VM, or one of the
Azure versions.Python UserBCPUtilityMicrosoft ODBC Driver11, 13, 13.1, or 17for SQL Server.
(Microsoft Docs)
See thepyodbc docsfor more on different driver versions.Python >= 3.8pandas>= 1.5sqlalchemy>= 1.4pyodbcas thesupported DBAPIInstallationSourceCommandPyPIpip install bcpandasCondaconda install -c conda-forge bcpandasUsageCreate creds (see next section)Replace anydf.to_sql(...)in your code withbcpandas.to_sql(df, ...)That's it!Credential/Connection objectBcpandas requires abcpandas.SqlCredsobject in order to use it, and also asqlalchemy.Engine.
The user has 2 options when constructing it.Create the bcpandasSqlCredsobject with just the minimum attributes needed (server, database,
username, password), and bcpandas will create a fullEngineobject from this. It will usepyodbc,sqlalchemy, and the Microsoft ODBC Driver for SQL Server, and will store it in the.engineattribute.In[1]:frombcpandasimportSqlCredsIn[2]:creds=SqlCreds('my_server','my_db','my_username','my_password')In[3]:creds.engineOut[3]:Engine(mssql+pyodbc:///?odbc_connect=Driver={ODBCDriver17forSQLServer};Server=tcp:my_server,1433;Database=my_db;UID=my_username;PWD=my_password)Pass a fullEngineobject to the bcpandasSqlCredsobject, and bcpandas will attempt to
parse out the server, database, username, and password to pass to the command line utilities. If
a DSN is used, this will fail.(continuing example above)In[4]:creds2=SqlCreds.from_engine(creds.engine)In[5]:creds2.engineOut[5]:Engine(mssql+pyodbc:///?odbc_connect=Driver={ODBCDriver17forSQLServer};Server=tcp:my_server,1433;Database=my_db;UID=my_username;PWD=my_password)In[6]:creds2Out[6]:SqlCreds(server='my_server',database='my_db',username='my_username',with_krb_auth=False,engine=Engine(mssql+pyodbc:///?odbc_connect=Driver={ODBCDriver17forSQLServer};Server=tcp:my_server,1433;Database=my_db;UID=my_username;PWD=my_password),password=[REDACTED])Recommended UsageFeaturePandas nativeBCPandasSuper speed:x::white_check_mark:Good for simple data types like numbers and dates:x::white_check_mark:Handle messy string data:white_check_mark::x:built with the help ofhttps://www.tablesgenerator.com/markdown_tables#andhttps://gist.github.com/rxaviers/7360908Known IssuesHere are some caveats and limitations of bcpandas.Bcpandas has been tested with all ASCII characters 32-127. Unicode characters beyond that range
have not been tested.An empty string ("") in the dataframe becomesNULLin the SQL database instead of remaining
an empty string.Because bcpandas first outputs to CSV, it needs to use several specific characters to create the
CSV, including adelimiterand aquote character. Bcpandas attempts to use characters that
are not present in the dataframe for this, going through the possilbe delimiters and quote
characters specified inconstants.py. If all possible characters are present in the dataframe
and bcpandas cannot find both a delimiter and quote character to use, it will throw an error.The BCP utility doesnotignore delimiter characters when surrounded by quotes, unlike
CSVs - seeherein the Microsoft docs.If there is a NaN/Null in the last column of the dataframe it will throw an error. This is due
to a BCP issue. See my issue with Microsoft about thishere.This doesn't seem to be a
problem based on the tests.TroubleshootingAll quote characters appear in the dataIf you encounter the error:bcpandas.constants.BCPandasValueError: Data contains all of the possible quote characters ('"', "'", '`', '~'),
cannot use BCP to import it. Replace one of the possible quote characters in
your data, or use another method besides bcpandas.And want to still use BCPandas, you will need to pick a quote character and remove all instances of it from the dataframe.  Note
that you are modifying your data and take care that replacing e.g., the~character will not have undesired consequences.In this case we are looking to remove~, replacing it with-.  Hunt for its presence in a column:my_df['some_text_column'].str.contains('~').sum()If that returns a value greater than zero, you can perform replacement in that column like this:my_df['some_text_column'] = my_df['some_text_column'].str.replace('~','-')Then use the first command to confirm that the number of occurences is now 0.All delimiter characters appear in the dataVery similar to above, but with the error message:bcpandas.constants.BCPandasValueError: Data contains all of the possible delimiter characters (',', '|', '\t'),
cannot use BCP to import it. Replace one of the possible delimiter characters in
your data, or use another method besides bcpandas.Approach this as is described above for quote characters.  If you target the|character for replacement, note that it
must be escaped in a regular expression with a backslash.  So the relevant commands would be (here the pipe|is replaced with a front slash):my_df['some_text_column'] = my_df['some_text_column'].str.replace('\|','/')
my_df['some_text_column'].str.contains('\|').sum()Write to database fails due to spaces in columnsIf you get this error message when writing to the database:Error = [Microsoft][ODBC Driver 17 for SQL Server]Incorrect host-column number found in BCP format-fileTry replacing any space characters in your column names, with a command likemy_df.columns = my_df.columns.str.replace(' ','_')(source).BackgroundWriting data from pandas DataFrames to a SQL database is very slow using the built-into_sqlmethod, even with the newly introducedexecute_manyoption. For Microsoft SQL Server, a far far faster method is to use the BCP utility provided by
Microsoft. This utility is a command line tool that transfers data to/from the database and flat
text files.This package is a wrapper for seamlessly using the bcp utility from Python using a pandas
DataFrame. Despite the IO hits, the fastest option by far is saving the data to a CSV file in the
file system and using the bcp utility to transfer the CSV file to SQL Server.Best of all, you
don't need to know anything about using BCP at all!Existing SolutionsMuch credit is due tobcpyfor the original idea and for some of the code that was adopted and
changed.bcpybcpyhas several flaws:No support for reading from SQL, only writing to SQLA convoluted, overly class-based internal designScope a bit too broad - deals with pandas as well as flat files This repository aims to fix and
improve onbcpyand the above issues by making the design choices described earlier.Design and ScopeTheonlyscope ofbcpandasis to read and write between a pandas DataFrame and a Microsoft
SQL Server database. That's it. We donotconcern ourselves with reading existing flat files
to/from SQL - that introduceswayto much complexity in trying to parse and decode the various
parts of the file, like delimiters, quote characters, and line endings. Instead, to read/write an
exiting flat file, just import it via pandas into a DataFrame, and then usebcpandas.The big benefit of this is that we get to precicely control all the finicky parts of the text file
when we write/read it to a local file and then in the BCP utility. This lets us set library-wide
defaults (maybe configurable in the future) and work with those.For now, we are using the non-XML BCP format file type. In the future, XML format files may be
added.TestingTesting RequirementsDocker Desktop installed, either of the Linux or Windows runtimes, doesn't matterpytesthypothesispytest-cov (coverage.py)What Is Tested?We take testing very seriously here. In order to rely on a library like this in production, itMUSTbe ruthlessly tested, which thankfully it is. Here is a partial list of what has been
tested so far. Pull Requests welcome!Data types: All ASCII characters 32-127 (using the Hypothesis library, see below). Unicode
characters beyond that range have not been tested.numpy.NaN,Nonenumpy.inf(fails, as expected)Empty dataframe (nothing happens, database not modified)Duplicate column names (raises error)Database columns that are missing from the dataframe, are out of order, or both (passes)Extra dataframe columns that aren't in database, whenif_exists="append"specified (fails)Testing ImplementationTesting usespytest.To test for all possible data types, we use thehypothesislibrary, instead of trying to come
up with every single case on our own.pytest-cov(which usescoverage.pyunder the hood) is used to measure code coverage. This is
then uploaded tocodecov.ioas part of the
CI/CD process (see below).In order to spin up a local SQL Server during testing, we use Docker. Specifically, we run one of
the images that Microsoft provides that already have SQL Server fully installed, all we have to
do is use the image to run a container. Here are the links to theLinux versionsand the Windows versions -ExpressandDeveloper.When running the tests, we can specify a specific Docker image to use, by invoking the custom
command line option called--mssql-docker-image. For example:pytestbcpandas/tests--mssql-docker-imagemcr.microsoft.com/mssql/server:2019-latestInstead of using thesubprocesslibrary to control Docker manually, we use the elegantdocker-pylibrary which works very nicely. ADockerDBPython class is defined inbcpandas/tests/utils.pyand it wraps up all the Docker commands and functionality needed to use
SQL Server into one class. This class is used inconftest.pyin the core bcpandas tests, and in
thebenchmarks/directory for both the benchmarks code as well as the legacy tests forread_sql.CI/CDGithub Actions is used for CI/CD, although it is still somewhat a work in progress.ContributingPlease, all contributions are very welcome!I will attempt to use thepandasdocstring style as detailedhere.GitHub Stars Trend