Package: pdexplorer
"...succinctness is power... we take the trouble to develop high-level languages...
so that we can say (and more importantly, think) in 10 lines of a high-level language what would require 1000 lines of machine language... -Paul Graham, Succinctness is PowerInstallationpdexploreris available onPyPI. Runpipto install:pip install pdexplorerThen import thepdexplorercommands withfrompdexplorerimport*pdexplorerpdexploreris a Stata emulator for Python/pandas. In contrast to raw Python/pandas, Stata syntax achieves succinctness by:Using spaces and commas rather than parentheses, brackets, curly braces, and quotes (where possible)Specifying a set of concise commands on the "current" dataset rather than cluttering the namespace with multiple datasetsBeing verbose by default i.e., displaying output that represents the results of the commandHaving sensible defaults that cover the majority of use cases and demonstrate common usageAllowing for namespace abbreviations for both commands and variable namesEmploying two types of column names: Variable name are concise and used for programming. Variable labels are verbose
and used for presentation.Packages are imported lazily e.g.,import torchis loaded only when it's first used by a command. This ensures thatfrom pdexplorer import *runs quickly.ExamplesLoad Stata dataset and perform exploratory data analysiswebuse('auto')browse()Seehttps://www.stata.com/manuals/dwebuse.pdfSummarize Data by Subgroupswebuse('auto')withby('foreign'):summarize('mpg weight')Seehttps://www.stata.com/manuals/rsummarize.pdfOrdinary Least Squares (OLS) Regressionwebuse('auto')regress('mpg weight foreign')ereturnlist()Return ValuesIn the last example, note the use ofereturnlist(), corresponding to the Stata commandereturn list. Additionally, a Python object may also be available as the command's return value. For example,webuse('auto')results=regress('mpg weight foreign')Here,resultsis aRegressionResultsWrapperobject from thestatsmodelspackage.Similarly,results=regress('mpg weight foreign',library='scikit-learn')Now,resultsis aLinearRegressionobject from thescikit-learnpackage.Finally,results=regress('mpg weight foreign',library='pytorch')Here,resultsis atorch.nn.Linearobject from thePyTorchpackage.LLM Fine Tuning ExamplesSentiment Analysis using HuggingFacefrompdexplorerimport*frompdexplorer.tests.fixturesimportyelp_reviewsdf=pd.DataFrame.from_records(yelp_reviews)use(df)# load examples into pdexplorerfthuggingface("stars text",task="sentiment-analysis",model_name="distilbert-base-uncased")# slowaskhuggingface("I absolutely loved Burgerosity!",task="sentiment-analysis")Next Word Prediction using HuggingFacefrompdexplorerimport*frompdexplorer.tests.fixturesimporteli5df=pd.DataFrame.from_records(eli5)use(df)# load examples into pdexplorerfthuggingface("text",task="text-generation",model_name="distilgpt2")# slowaskhuggingface("A poem about Mickey Mouse in iambic pentameter:\n",task="text-generation")Next Word Prediction using OpenAI (gpt-3.5-turbo)frompdexplorerimport*frompdexplorer.tests.test_ftgptimportdfuse(df)ftgpt("assistant user system")# slow; requires OPENAI_API_KEY environment variableaskgpt("A poem about Mickey Mouse in iambic pentameter:\n")Why not use Stata instead of pandas?Stata is great, but Python/pandas is free and easier to integrate with other applications. For example, you can
build a web server in Python, but not Stata; You can run Python in AWS SageMmaker, but not Stata.Additionally, even for devout Stata users, there is utility in being able to run Stata commands through a Python stack for
comparison purposes.Howpdexplorerfulfills theZen of Python(relative to pandas)YESNOBeautiful is better than ugly.Explicit is better than implicit.Simple is better than complex.In the face of ambiguity, refuse the temptation to guess.Flat is better than nested.Readability counts.Although practicality beats purity.There should be one-- and preferably only one --obvious way to do it.Now is better than never.Howpdexplorerdiffers from StataCommands are implemented as Python functions and hence require at least one set of parenthesespdexploreruses Python libraries under the hood. (The result of a command reflects the output of those libraries. See above.)There is no support formata. Under the hood,pdexploreris just the Python data stack.The API for producing charts is based onAltair, not Stata.pdexploreradds commands for machine learning (using sklearn, PyTorch, and huggingface)Syntax summaryWith few exceptions, the basic Stata language syntax (as documentedhere) is[byvarlist:] command [varlist] [=exp] [if exp] [inrange] [weight] [, options]where square brackets distinguish optional qualifiers and options from required ones. In this diagram,
varlist denotes a list of variable names, command denotes a Stata command, exp denotes an algebraic
expression, range denotes an observation range, weight denotes a weighting expression, and options
denotes a list of options.Theby varlist:prefix causes Stata to repeat a command for each subset of the data for which the
values of the variables in varlist are equal. When prefixed with by varlist:, the result of the command
will be the same as if you had formed separate datasets for each group of observations, saved them,
and then gave the command on each dataset separately. The data must already be sorted by varlist,
although by has a sort option.In pdexplorer, this gets translated towithby('varlist'):command("[varlist] [=exp] [if exp] [in range] [weight] [, options]",*args,**kwargs)where*args, and**kwargsrepresent additional arguments that are available in apdexplorercommand but
not in the equivalent Stata command.Sometimes, Stata commands are two words. In such cases, thepdexplorercommand is a concatenation of the two words. For example,labeldata"label"becomeslabeldata("label")Module DependenciesFile locationDescriptionDependencies/*.pycommands that are native to Stata related to data wrangling or statisticspandas/_altair_mapper.pycommands that are native to Altair for chartingaltairshortcuts/*.pyshortcut commands related to data wrangling, statistics, or chartingall of the abovefinance/*.pycommands that are specific to financial applicationsall of the aboveml/*.pycommands that use machine learning techniques (and are outside the scope of Stata)scikit-learnnn/*.pycommands that use neutral networks (primarily built using PyTorch)PyTorchdata/*.pypython scripts that collect data from various sourcesData suppliersexperimental/*.pycommands that are current under development and not yet stableN/ACommand Dependenciespdexplorercommandpackage dependencycfydata-profilingorsweetvizbrowsedtaleregressstatsmodelsorscikit-learnorPyTorchChartspdexplorerdeparts somewhat from Stata in producing charts. Rather than emulating Stata's chart syntax,pdexplorerusesAltairwith some syntactic sugar.
Take the example"Simple Scatter Plot with Tooltips":importwebbrowserimportaltairasaltfromvega_datasetsimportdatasource=data.cars()chart=alt.Chart(source).mark_circle(size=60).encode(x="Horsepower",y="Miles_per_Gallon",color='Origin',tooltip=["Name","Origin","Horsepower","Miles_per_Gallon"],)chart.save('mychart.html')webbrowser.open('mychart.html')Inpdexplorer, this becomes:frompdexplorerimport*webuse("cars","vega")circlechart("miles_per_gallon horsepower,\color(origin) tooltip(name origin horsepower miles_per_gallon)",size=60,)In other words,pdexplorersupports avarlistparameter fory/xencodings. Additional encodings are specified via the
Stata options syntax.In the above example,circlechartautomatically displays the chart in a web browser. However, sometimes it's necessary to add features to thealt.ChartAltair object. To use thealt.Chartobject, we can usecirclechart_instead ofcirclechart. For example,importwebbrowserfrompdexplorerimport*webuse("cars","vega")altair_chart_object=circlechart_("miles_per_gallon horsepower, color(origin)\tooltip(name origin horsepower miles_per_gallon)",size=60,)altair_chart_object.configure_legend(strokeColor='gray',fillColor='#EEEEEE',padding=10,cornerRadius=10,orient='top-right')# See https://altair-viz.github.io/user_guide/configuration.html#legend-configurationaltair_chart_object.save('mychart.html')webbrowser.open('mychart.html')Instead of saving the chart, explicitly, we can also writealtair_chart_object()The()at the end tells Altair to open the chart in a web browser. This method is not available inalt.Chartsitself,
but we monkey patched this into the class for convenience.Similarly, the following two statements are identical,circlechart_("miles_per_gallon horsepower")()# the object-oriented stylecirclechart("miles_per_gallon horsepower")# the imperative styleIn this example,circlechart_itself simply returns aalt.Chartobject and()indicates that the chart should be displayed in
a web browser.Since we can access thealt.Chartobject, we can also specify encodings explicitly using Altair'sencodemethod e.g.,frompdexplorerimport*webuse("cars","vega")# note that names are forced to be lower case by defaultcirclechart_(size=60).encode(x="horsepower",y="miles_per_gallon",color="origin",tooltip=["name","origin","horsepower","miles_per_gallon"],)()# () indicates that the chart is complete and should be opened in a web browserSincepdexplorercharts are justalt.Chartobjects,Layered and Multi-View Chartsare also supported e.g.,frompdexplorerimport*webuse("cars","vega")chartA=circlechart_("miles_per_gallon weight_in_lbs")chartB=circlechart_("horsepower weight_in_lbs")(chartA&chartB)()# Vertically concatenate chartA and chartBFinally,pdexploreralso offers syntactic sugar for charting multiplex/yvariables. More specifically, the previous block can be
written asfrompdexplorerimport*webuse("cars","vega")circlechart("miles_per_gallon horsepower weight_in_lbs",stacked=True)Note that Stata'svarlistinterpretation is used here by default i.e.,var1 var2 var3is assumed to representyvar1 yvar2 xvar. We can change this interpretation with the optional argumentyX.frompdexplorerimport*webuse("cars","vega")circlechart("miles_per_gallon horsepower weight_in_lbs",yX=True,stacked=True)Nowvar1 var2 var3is assumed to representyvar xvar1 xvar2as it would be for theregresscommand.The Stata default is to layer all variables onto a single chart. Thestacked=Trueoption allows the graphs to be shown on
separate grids. Ifstacked=False, the charts are all shown on the same grid i.e.,frompdexplorerimport*webuse("cars","vega")circlechart("miles_per_gallon horsepower weight_in_lbs")# stacked=False is the default optionNote thatpdexploreruses Altair'stransform_foldmethod under the hood. For further customization, the Altair methods can be used explicitly e.g.,importaltairasaltfrompdexplorerimport*webuse("cars","vega")circlechart_().transform_fold(["Miles_per_Gallon","Horsepower"]# Note that circlechart_ variable labels are accessible).encode(y="value:Q",x=alt.X("weight_in_lbs",title="Weight_in_lbs"),color="key:N")()alt.Chart.transform_foldis Altair's version ofpandas.melt. So another option is to first reshape the data using
pandas and then use Altair for charting.importaltairasaltfrompdexplorerimport*webuse("cars","vega")melt("miles_per_gallon horsepower,  keep(weight_in_lbs)")alt.Chart(current.df_labeled).mark_circle().encode(y="value:Q",x='Weight_in_lbs',color="variable:N")()Note that theAltair documentationsuggests the latter approach
in most cases where a data transformation is required.AbbreviationsAs mentioned ealier, Stata supports name abbreviations for both variable names as well as command names. Inpdexplorer, all the following regression statements are equivalent:frompdexplorerimport*webuse("auto")reg('price mpg weight')regr('price mpg weight')regre('price mpg weight')regres('price mpg weight')regress('price mpg weight')reg('pr mpg wei')Similarly, for charting,frompdexplorerimport*webuse("cars","vega")circle("miles_per_gallon horsepower")circlec("miles_per_gallon horsepower")circlech("miles_per_gallon horsepower")circlecha("miles_per_gallon horsepower")circlechar("miles_per_gallon horsepower")circlechart("miles_per_gallon horsepower")circle("miles horse")Referenceshttps://aeturrell.github.io/coding-for-economists/coming-from-stata.htmlhttps://www.stata.com/manuals13/u27.pdfhttps://www.stata.com/manuals13/u11.pdf