Package: verticapy
:star: 2023-12-01: VerticaPy secures 200 stars.:loudspeaker: 2020-06-27: Vertica-ML-Python has been renamed to VerticaPy.:warning: The old website has been relocated tohttps://www.vertica.com/python/old/and will be discontinued soon. We strongly recommend upgrading to the new major release before August 2024.:warning: The following README is for VerticaPy 1.0.x and onwards, and so some of the elements may not be present in the previous versions.:scroll: Some basic syntax can be found inthe cheat sheet.📰 Check out the latest newsletterhere.VerticaPyVerticaPy is a Python library with scikit-like functionality used to conduct data science projects on data stored in Vertica, taking advantage of Vertica’s speed and built-in analytics and machine learning features. VerticaPy offers robust support for the entire data science life cycle, uses a 'pipeline' mechanism to sequentialize data transformation operations, and offers beautiful graphical options.Table of ContentsIntroductionInstallationConnecting to the DatabaseDocumentationUse-casesHighlighted FeaturesThemes - Dark | LightSQL MagicSQL PlotsDiverse Database ConnectionsPython and SQL ComboChartsComplete ML pipelineQuickstartHelp and SupportContributingCommunicationIntroductionVertica was the first real analytic columnar database and is still the fastest in the market. However, SQL alone isn't flexible enough to meet the needs of data scientists.Python has quickly become the most popular tool in this domain, owing much of its flexibility to its high-level of abstraction and impressively large and ever-growing set of libraries. Its accessibility has led to the development of popular and perfomant APIs, like pandas and scikit-learn, and a dedicated community of data scientists. Unfortunately, Python only works in-memory as a single-node process. This problem has led to the rise of distributed programming languages, but they too, are limited as in-memory processes and, as such, will never be able to process all of your data in this era, and moving data for processing is prohobitively expensive. On top of all of this, data scientists must also find convenient ways to deploy their data and models. The whole process is time consuming.VerticaPy aims to solve all of these problems. The idea is simple: instead of moving data around for processing, VerticaPy brings the logic to the data.3+ years in the making, we're proud to bring you VerticaPy.Main Advantages:Easy Data Exploration.Fast Data Preparation.In-Database Machine Learning.Easy Model Evaluation.Easy Model Deployment.Flexibility of using either Python or SQL.:arrow_up: Back to TOCInstallationTo installVerticaPywith pip:# Latest release versionroot@ubuntu:~$pip3installverticapy[all]# Latest commit on master branchroot@ubuntu:~$pip3installgit+https://github.com/vertica/verticapy.git@masterTo installVerticaPyfrom source, run the following command from the root directory:root@ubuntu:~$python3setup.pyinstallA detailed installation guide is available at:https://www.vertica.com/python/documentation/installation.html:arrow_up: Back to TOCConnecting to the DatabaseVerticaPy is compatible with several clients. For details, see theconnection page.:arrow_up: Back to TOCDocumentationThe easiest and most accurate way to find documentation for a particular function is to use the help function:importverticapyasvphelp(vp.vDataFrame)Official documentation is available at:https://www.vertica.com/python/documentation/:arrow_up: Back to TOCUse-casesExamples and case-studies:https://www.vertica.com/python/examples/:arrow_up: Back to TOCHighlighted FeaturesThemesVerticaPy, offers users the flexibility to customize their coding experience with two visually appealing themes:DarkandLight.Dark mode, ideal for night-time coding sessions, features a sleek and stylish dark color scheme, providing a comfortable and eye-friendly environment.On the other hand, Light mode serves as the default theme, offering a clean and bright interface for users who prefer a traditional coding ambiance.Theme can be easily switched by:importverticapyasvpvp.set_option("theme","dark")# can be switched 'light'.VerticaPy's theme-switching option ensures that users can tailor their experience to their preferences, making data exploration and analysis a more personalized and enjoyable journey.:arrow_up: Back to TOCSQL MagicYou can use VerticaPy to execute SQL queries directly from a Jupyter notebook. For details, seeSQL Magic:ExampleLoad the SQL extension.%load_extverticapy.sqlExecute your SQL queries.%%sqlSELECTversion();#Output#VerticaAnalyticDatabasev11.0.1-0:arrow_up: Back to TOCSQL PlotsYou can create interactive, professional plots directly from SQL.To create plots, simply provide the type of plot along with the SQL command.Example%load_extverticapy.jupyter.extensions.chart_magic%chart-kpie-c"SELECT pclass, AVG(age) AS av_avg FROM titanic GROUP BY 1;":arrow_up: Back to TOCMultiple Database Connection using DBLINKIn a single platform, multiple databases (e.g. PostgreSQL, Vertica, MySQL, In-memory) can be accessed using SQL and python.Example%%sql/* Fetch TAIL_NUMBER and CITY after Joining the flight_vertica table with airports table in MySQL database. */SELECTflight_vertica.TAIL_NUMBER,airports.CITYASDeparting_CityFROMflight_verticaINNERJOIN&&&airports&&&ONflight_vertica.ORIGIN_AIRPORT=airports.IATA_CODE;In the example above, the 'flight_vertica' table is stored in Vertica, whereas the 'airports' table is stored in MySQL. We can associate special symbols "&&&" to the different databases to fetch the data. The best part is that all the aggregation is pushed to the databases (i.e. it is not done in memory)!For more details on how to setup DBLINK, please visit thegithub repo. To learn about using DBLINK in VerticaPy, check out thedocumentation page.:arrow_up: Back to TOCPython and SQL ComboVerticaPy has a unique place in the market because it allows users to use Python and SQL in the same environment.Exampleimportverticapyasvpselected_titanic=vp.vDataFrame("(SELECT pclass, embarked, AVG(survived) FROM public.titanic GROUP BY 1, 2) x")selected_titanic.groupby(columns=["pclass"],expr=["AVG(AVG)"]):arrow_up: Back to TOCChartsVerticapy comes integrated with three popular plotting libraries: matplotlib, highcharts, and plotly.A gallery of VerticaPy-generated charts is available at:https://www.vertica.com/python/documentation/chart.html:arrow_up: Back to TOCComplete Machine Learning PipelineData IngestionVerticaPy allows users to ingest data from a diverse range of sources, such as AVRO, Parquet, CSV, JSON etc. With a simple command "read_file", VerticaPy automatically infers the source type and the data type.importverticapyasvpread_file("/home/laliga/2012.json",table_name="laliga",)Note: Not all columns are displayed in the screenshot above because of width restriction here.As shown above, it has created a nested structure for the complex data. The actual file structure is below:We can even see the SQL underneath every VerticaPy command by turning on the genSQL option:importverticapyasvpread_file("/home/laliga/2012.json",table_name="laliga",genSQL=True)CREATELOCALTEMPORARYTABLE"laliga"("away_score"INT,"away_team"ROW("away_team_gender"VARCHAR,"away_team_group"VARCHAR,"away_team_id"INT,...ROW("id"INT,"name"VARCHAR)),"competition"ROW("competition_id"INT,"competition_name"VARCHAR,"country_name"VARCHAR),"competition_stage"ROW("id"INT,"name"VARCHAR),"home_score"INT,"home_team"ROW("country"ROW("id"INT,"name"VARCHAR),"home_team_gender"VARCHAR,"home_team_group"VARCHAR,"home_team_id"INT,...),"kick_off"TIME,"last_updated"DATE,"match_DATE"DATE,"match_id"INT,...ROW("data_version"DATE,"shot_fidelity_version"INT,"xy_fidelity_version"INT),"season"ROW("season_id"INT,"season_name"VARCHAR))ONCOMMITPRESERVEROWSCOPY"v_temp_schema"."laliga"FROM'/home/laliga/2012.json'PARSERFJsonParser()VerticaPy provides functions for importing other specific file types, such asread_jsonandread_csv. Since these functions focus on a particular file type, they offer more options for tackling the data. For example,read_jsonhas a "flatten_arrays" parameter that allows you to flatten nested JSON arrays.Data ExplorationThere are many options for descriptive and visual exploration.fromverticapy.datasetsimportload_irisiris_data=load_iris()iris_data.scatter(["SepalWidthCm","SepalLengthCm","PetalLengthCm"],by="Species",max_nb_points=30)TheCorrelation Matrixis also very fast and convenient to compute. Users can choose from a wide variety of correaltions, including cramer, spearman, pearson etc.fromverticapy.datasetsimportload_titanictitanic=load_titanic()titanic.corr(method="spearman")By turning on the SQL print option, users can see and copy SQL queries:fromverticapyimportset_optionset_option("sql_on",True)SELECT/*+LABEL('vDataframe._aggregate_matrix')*/CORR_MATRIX("pclass","survived","age","sibsp","parch","fare","body")OVER()FROM(SELECTRANK()OVER(ORDERBY"pclass")AS"pclass",RANK()OVER(ORDERBY"survived")AS"survived",RANK()OVER(ORDERBY"age")AS"age",RANK()OVER(ORDERBY"sibsp")AS"sibsp",RANK()OVER(ORDERBY"parch")AS"parch",RANK()OVER(ORDERBY"fare")AS"fare",RANK()OVER(ORDERBY"body")AS"body"FROM"public"."titanic")spearman_tableVerticaPy allows users to calculate a focused correlation using the "focus" parameter:titanic.corr(method="spearman",focus="survived")Data PreparationWhether you arejoining multiple tables,encoding, orfilling missing values, VerticaPy has everything and more in one package.importrandomimportverticapyasvpdata=vp.vDataFrame({"Heights":[random.randint(10,60)for_inrange(40)]+[100]})data.outliers_plot(columns="Heights")Machine LearningML is the strongest suite of VerticaPy as it capitalizes on the speed of in-database training and prediction by using SQL in the background to interact with the database. ML for VerticaPy covers a vast array of tools, includingtime series forecasting,clustering, andclassification.# titanic_vd is already loaded# Logistic Regression model is already loadedstepwise_result=stepwise(model,input_relation=titanic_vd,X=["age","fare","parch","pclass",],y="survived",direction="backward",height=600,width=800,):arrow_up: Back to TOCLoading Predefined DatasetsVerticaPy provides some predefined datasets that can be easily loaded. These datasets include the iris dataset, titanic dataset, amazon, and more.There are two ways to access the provided datasets:(1) Use the standard python method:fromverticapy.datasetsimportload_irisiris_data=load_iris()(2) Use the standard name of the dataset from the public schema:iris_data=vp.vDataFrame(input_relation="public.iris"):arrow_up: Back to TOCQuickstartThe following example follows theVerticaPy quickstart guide.Install the library using withpip.root@ubuntu:~$pip3installverticapy[all]Create a new Vertica connection:importverticapyasvpvp.new_connection({"host":"10.211.55.14","port":"5433","database":"testdb","password":"XxX","user":"dbadmin"},name="Vertica_New_Connection")Use the newly created connection:vp.connect("Vertica_New_Connection")Create a VerticaPy schema for native VerticaPy models (that is, models available in VerticaPy, but not Vertica itself):vp.create_verticapy_schema()Create a vDataFrame of your relation:fromverticapyimportvDataFramevdf=vDataFrame("my_relation")Load a sample dataset:fromverticapy.datasetsimportload_titanicvdf=load_titanic()Examine your data:vdf.describe()Print the SQL query withset_option:set_option("sql_on",True)vdf.describe()# Output## Compute the descriptive statistics of all the numerical columns ##SELECTSUMMARIZE_NUMCOL("pclass","survived","age","sibsp","parch","fare","body")OVER()FROMpublic.titanicWith VerticaPy, it is now possible to solve a ML problem with few lines of code.fromverticapy.machine_learning.model_selection.model_validationimportcross_validatefromverticapy.machine_learning.verticaimportRandomForestClassifier# Data Preparationvdf["sex"].label_encode()["boat"].fillna(method="0ifnull")["name"].str_extract(" ([A-Za-z]+)\.").eval("family_size",expr="parch + sibsp + 1").drop(columns=["cabin","body","ticket","home.dest"])["fare"].fill_outliers().fillna()# Model Evaluationcross_validate(RandomForestClassifier("rf_titanic",max_leaf_nodes=100,n_estimators=30),vdf,["age","family_size","sex","pclass","fare","boat"],"survived",cutoff=0.35,)# Features importancemodel.fit(vdf,["age","family_size","sex","pclass","fare","boat"],"survived")model.features_importance()# ROC Curvemodel=RandomForestClassifier(name="public.RF_titanic",n_estimators=20,max_features="auto",max_leaf_nodes=32,sample=0.7,max_depth=3,min_samples_leaf=5,min_info_gain=0.0,nbins=32)model.fit("public.titanic",# input relation["age","fare","sex"],# predictors"survived"# response)# Roc Curvemodel.roc_curve()Enjoy!:arrow_up: Back to TOCHelp and SupportContributingFor a short guide on contribution standards, see theContribution Guidelines.CommunicationLinkedIn:https://www.linkedin.com/company/verticapy/Announcements and Discussion:https://github.com/vertica/VerticaPy/discussions:arrow_up: Back to TOC