Package: visions
And these visions of data types, they kept us up past the dawn.The Semantic Data LibraryVisionsprovides a set of tools for defining and usingsemanticdata types.Semantic typedetection &
inference on sequence data.Automated data processingCompletely customizable.Visionsmakes it easy to build and modify semantic data types for domain specific
purposesOut of the box support for
multiplebackend implementationsincluding pandas,
spark, numpy, and pythonA robust set
ofdefault types and typesetscovering the most common use cases.Check out the complete
documentationhere.InstallationSource code is available ongithuband binary installers via pip.# Pip
pip install visionsComplete installation instructions (including extras) are available in
thedocs.Quick Start GuideIf you want to play immediately check out the examples folder
on. Otherwise,
let's get some dataimportpandasaspddf=pd.read_csv("https://raw.githubusercontent.com/datasciencedojo/datasets/master/titanic.csv")df.head(2)PassengerIdSurvivedPclassNameSexAgeSibSpParchTicketFareCabinEmbarked103Braund, Mr. Owen Harrismale22.010A/5 211717.2500NaNS211Cumings, Mrs. John Bradley (Florence Briggs Thayer)female38.010PC 1759971.2833C85CThe most important abstraction invisionsare Types - these represent semantic notions about data. You have access to
a
range of well tested types likeInteger,Float, andFilescovering the most common software development use cases.
Types can be bundled together into typesets. Behind the scenes,visionsbuilds a traversable graph for any collection
of types.fromvisionsimporttypes,typesets# StandardSet is the basic builtin typesettypeset=typesets.CompleteSet()typeset.plot_graph()Note: Plots require pygraphviz to beinstalled.Because of the special relationship between types these graphs can be used to detect the type of your data orinfera
more appropriate one.# Detection looks like thistypeset.detect_type(df)# While inference looks like thistypeset.infer_type(df)# Inference works well even if we monkey with the data, say by converting everything to stringstypeset.infer_type(df.astype(str))>>{'PassengerId':Integer,'Survived':Integer,'Pclass':Integer,'Name':String,'Sex':String,'Age':Float,'SibSp':Integer,'Parch':Integer,'Ticket':String,'Fare':Float,'Cabin':String,'Embarked':String}Visionssolves many of the most common problems working with tabular data for example, sequences of Integers are still
recognized as integers whether they have trailing decimal 0's from being cast to float, missing values, or something
else altogether. Much of this cleaning is performed automatically providing nicely cleaned and processed data as well.cleaned_df=typeset.cast_to_inferred(df)This is only a small taste of everything visions can do
includingbuilding your owndomain
specific types and typesets so please check out theAPIdocumentation or theexamples/directory for more
info!Supported frameworksThanks to its dispatch based implementationVisionsis able to exploit framework specific capabilities offered by
libraries like pandas and spark. Currently it works with the following backends by default.Pandas(feature complete)Numpy(boolean, complex, date time, float, integer, string, time deltas, string,
objects)Spark(boolean, categorical, date, date time, float, integer, numeric, object,
string)Python(string, float, integer,
date time, time delta, boolean, categorical, object, complex - other datatypes are untested)If you're using pandas it will also take advantage of parallelization tools likeswifterif available.It also offers a simple annotation based API for registering new implementations as needed. For example, if you wished
to extend the categorical data type to include a Dask specific implementation you might do something likefromvisions.types.categoricalimportCategoricalfrompandas.apiimporttypesaspdtimportdask@Categorical.contains_op.registerdefcategorical_contains(series:dask.dataframe.Series,state:dict)->bool:returnpdt.is_categorical_dtype(series.dtype)Contributing and supportContributions tovisionsare welcome. For more information, please visit the community
contributionspageand join on us
onslack. The
githubissues trackeris used for reporting bugs, feature
requests and support questions.Also, please check out some of the other companies and packages usingvisionsincluding:pandas profilingCompressioBitrookIf you're currently usingvisionsor would like to be featured here please let us know.AcknowledgementsThis package is part of thedylan-profilerproject. The package is core component
ofpandas-profiling. More information can be
foundhere. This work was partially supported
bySIDN Fonds.