Package: vertica-ml-python
Vertica-ML-PythonThe documentation is available at:https://github.com/vertica/Vertica-ML-Python/blob/master/documentation.pdfOr directly in the Wiki at:https://github.com/vertica/Vertica-ML-Python/wiki(c) Copyright [2018-2020] Micro Focus or one of its affiliates.
Licensed under the Apache License, Version 2.0 (the "License");
You may not use this file except in compliance with the License.
You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.⚠ If you want to contribute, send a mail tobadr.ouali@microfocus.comVertica-ML-Python is a Python library that exposes sci-kit like functionality to conduct data science projects on data stored in Vertica, thus taking advantage Vertica’s speed and built-in analytics and machine learning capabilities. It supports the entire data science life cycle, uses a ‘pipeline’ mechanism to sequentialize data transformation operation (called Virtual Dataframe), and offers multiple graphical rendering possibilities.The 'Big Data' (Tb of data) is now one of the main topics in the Data Science World. Data Scientists are now very important for any organisation. Becoming Data-Driven is mandatory to survive. Vertica is the first real analytic columnar Database and is still the fastest in the market. However, SQL is not enough flexible to be very popular for Data Scientists. Python flexibility is priceless and provides to any user a very nice experience. The level of abstraction is so high that it is enough to think about a function to notice that it already exists. Many Data Science APIs were created during the last 15 years and were directly adopted by the Data Science community (examples: pandas and scikit-learn). However, Python is only working in-memory for a single node process. Even if some famous highly distributed programming languages exist to face this challenge, they are still in-memory and most of the time they can not process on all the data. Besides, moving the data can become very expensive. Data Scientists must also find a way to deploy their data preparation and their models. We are far away from easiness and the entire process can become time expensive.The idea behind VERTICA ML PYTHON is simple: Combining the Scalability of VERTICA with the Flexibility of Python to give to the community what they needBringing the logic to the data and not the opposite. This version 1.0 is the work of 3 years of new ideas and improvement.Main Advantages:easy Data Exploration.easy Data Preparation.easy Data Modeling.easy Model Evaluation.easy Model Deployment.most of what pandas.Dataframe can do, vertica_ml_python.vDataframe can do (and even much more)easy ML model creation and evaluation.many scikit functions and algorithms are available (and scalable!).All information related to the API can be found at:https://github.com/vertica/Vertica-ML-Python/Python Versionvertica-ml-pythonworks with at least:Vertica:=> 9.1 (with previous versions, some functions and algorithms may not be available)Python Version:=> 3.6 - [version 3.5 may works]Python Modules needed for Rendering Capabilities:matplotlib (=> 3.0) - [other versions of matplotlib may work], numpy (=> 1.9) - [other versions of numpy may work]Other Python Modules:Except to get rendering capabilities, VERTICA ML Python uses only built-in libraries (statistics, random, math, time and os)Standard Librariesvertica-ml-pythonlibrary is only using the standard Python libraries such asmatplotlib,numpy...
Other libraries can be used asanytreefor tree visualization orsqlparsefor SQL indentation but they are optional.InstallationTo installvertica-ml-python, you can use the pip command:root@ubuntu:~$pip3installvertica_ml_pythonOr you can get a copy of the source by cloning from the Vertica-ML-Python github project and install with:root@ubuntu:~$python3setup.pyinstallYou can also drag and drop thevertica_ml_pythonfolder in thesite-packagefolder of the Python framework. In the MAC environment, you can find it in:/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packagesAnother way is to call the library from where it is located.You can then import each library element using the usual Python syntax.# to import the vDataframefromvertica_ml_pythonimportvDataframe# to import the Logistic Regressionfromvertica_ml_python.learn.linear_modelimportLogisticRegressionEverything is well detailed in the following documentation.Connection to the DatabaseThis step is useless ifvertica-pythonorpyodbcis already installed and you have a DSN in your machine. With this configuration, you do not need to manually create a cursor. It is possible to create a vDataframe using directly the DSN (dsnparameter of the vDataframe).ODBCTo connect to the database, the user can use an ODBC connection to the Vertica database.vertica-pythonandpyodbcprovide a cursor that will point to the database. It will be used by thevertica-ml-pythonto create all the different objects.## vertica_python#importvertica_python# Connection using all the DSN informationconn_info={'host':"10.211.55.14",'port':5433,'user':"dbadmin",'password':"XxX",'database':"testdb"}cur=vertica_python.connect(**conn_info).cursor()# Connection using directly the DSNfromvertica_ml_python.utilitiesimportto_vertica_python_format# This function will parse the odbc.ini filedsn="VerticaDSN"cur=vertica_python.connect(**to_vertica_python_format(dsn)).cursor()## pyodbc#importpyodbc# Connection using all the DSN informationdriver="/Library/Vertica/ODBC/lib/libverticaodbc.dylib"server="10.211.55.14"database="testdb"port="5433"uid="dbadmin"pwd="XxX"dsn=("DRIVER={}; SERVER={}; DATABASE={}; PORT={}; UID={}; PWD={};").format(driver,server,database,port,uid,pwd)cur=pyodbc.connect(dsn).cursor()# Connection using directly the DSNdsn=("DSN=VerticaDSN")cur=pyodbc.connect(dsn).cursor()JDBCThe user can also use a JDBC connection to the Vertica Database.importjaydebeapi# Vertica Server Detailsdatabase="testdb"hostname="10.211.55.14"port="5433"uid="dbadmin"pwd="XxX"# Vertica JDBC class namejdbc_driver_name="com.vertica.jdbc.Driver"# Vertica JDBC driver pathjdbc_driver_loc="/Library/Vertica/JDBC/vertica-jdbc-9.3.1-0.jar"# JDBC connection stringconnection_string='jdbc:vertica://'+hostname+':'+port+'/'+databaseurl='{}:user={};password={}'.format(connection_string,uid,pwd)conn=jaydebeapi.connect(jdbc_driver_name,connection_string,{'user':uid,'password':pwd},jars=jdbc_driver_loc)cur=conn.cursor()Quick StartInstall the library using thepipcommand:root@ubuntu:~$pip3installvertica_ml_pythonInstallvertica_pythonorpyodbcto build a DB cursor:root@ubuntu:~$pip3installvertica_pythonCreate a vertica cursorfromvertica_ml_python.utilitiesimportvertica_cursorcur=vertica_cursor("VerticaDSN")Create the Virtual Dataframe of your relation:fromvertica_ml_pythonimportvDataframevdf=vDataframe("my_relation",cursor=cur)If you don't have data to play, you can easily load well known datasetsfromvertica_ml_python.learn.datasetsimportload_titanicvdf=load_titanic(cursor=cur)You can now play with the data...vdf.describe()# Outputmin25%50%75%age0.3321.028.039.0body1.079.25160.5257.5fare0.07.895814.454231.3875parch0.00.00.00.0pclass1.01.03.03.0sibsp0.00.00.01.0survived0.00.00.01.0maxuniqueage80.096body328.0118fare512.3292277parch9.08pclass3.03sibsp8.07survived1.02You can also print the SQL code generation using thesql_on_offmethod.vdf.sql_on_off()vdf.describe()# Output## Compute the descriptive statistics of all the numerical columns ##SELECTSUMMARIZE_NUMCOL("age","body","survived","pclass","parch","fare","sibsp")OVER()FROMpublic.titanicWith Vertica ML Python, it is now possible to solve a ML problem with four lines of code (two if we don't consider the libraries loading).fromvertica_ml_python.learn.model_selectionimportcross_validatefromvertica_ml_python.learn.ensembleimportRandomForestClassifier# Data Preparationvdf["sex"].label_encode()["boat"].fillna(method="0ifnull")["name"].str_extract(' ([A-Za-z]+)\.').eval("family_size",expr="parch + sibsp + 1").drop(columns=["cabin","body","ticket","home.dest"])["fare"].fill_outliers().fillna().to_db("titanic_clean")# Model Evaluationcross_validate(RandomForestClassifier("rf_titanic",cur,max_leaf_nodes=100,n_estimators=30),"titanic_clean",["age","family_size","sex","pclass","fare","boat"],"survived",cutoff=0.35)# Outputaucprc_auc1-fold0.98771144278606910.95304659150393392-fold0.99655550146056420.76764853514257213-fold0.99272392165493010.6419135521132449avg0.9923302886340.787536226253std0.003621284640930.12779562393accuracylog_loss1-fold0.9712918660287080.05020525412238712-fold0.9832535885167460.02981677517984573-fold0.9648241206030150.0392745694400433avg0.9731231917160.0397655329141std0.00763442367290.00833079837099precisionrecall1-fold0.960.962-fold0.95569620253164561.03-fold0.96478873239436620.9383561643835616avg0.9601616449750.966118721461std0.003713769123110.025535200301f1-scoremcc1-fold0.96872592820828840.93761194029850752-fold0.98671726755218210.96469710108784693-fold0.95880202873090970.9240569687684576avg0.971415074830.942122003385std0.01155389607530.0168949813163informednessmarkedness1-fold0.93761194029850750.93761194029850752-fold0.97378277153558070.95569620253164563-fold0.91851489454229180.9296324823943662avg0.9433032021250.940980208408std0.02291909542610.0109037699717csi1-fold0.92307692307692312-fold0.95569620253164563-fold0.9072847682119205avg0.928685964607std0.0201579224026Happy Playing ! 😜