Package: optimuspyspark
NOTE: We are working hard in version 3.0https://github.com/ironmussa/Optimus/tree/develop-3.0. in which you could use pandas, cudf, Dask, Dask/cudf as backend using a unified API.To launch a live notebook server to test optimus using binder or Colab, click on one of the following badges:Optimus is the missing framework to profile, clean, process and do ML in a distributed fashion using Apache Spark(PySpark).Installation (pip):In your terminal just typepip install optimuspysparkRequirementsApache Spark>= 2.4.0Python>=3.6ExamplesYou can go to the 10 minutes to Optimusnotebookwhere you can find the basic to start working.Also you can go to theexamplesfolder to found specific notebooks about data cleaning, data munging, profiling, data enrichment and how to create ML and DL models.Besides check theCheat SheetDocumentationFeedbackFeedback is what drive Optimus future, so please take a couple of minutes to help shape the Optimus' Roadmap:http://bit.ly/optimus_surveyAlso if you want to a suggestion or feature request usehttps://github.com/ironmussa/optimus/issuesStart OptimusfromoptimusimportOptimusop=Optimus(verbose=True)You also can use an already created Spark session:frompyspark.sqlimportSparkSessionfromoptimusimportOptimusspark=SparkSession.builder.appName('optimus').getOrCreate()op=Optimus(spark)Loading dataNow Optimus can load data in csv, json, parquet, avro, excel from a local file or URL.#csvdf=op.load.csv("../examples/data/foo.csv")#json# Use a local filedf=op.load.json("../examples/data/foo.json")# Use a urldf=op.load.json("https://raw.githubusercontent.com/ironmussa/Optimus/master/examples/data/foo.json")# parquetdf=op.load.parquet("../examples/data/foo.parquet")# avro# df = op.load.avro("../examples/data/foo.avro").table(5)# exceldf=op.load.excel("../examples/data/titanic3.xls")Also you can load data from oracle, redshift, mysql and postgres. SeeDatabase connectionSaving Data#csvdf.save.csv("data/foo.csv")# jsondf.save.json("data/foo.json")# parquetdf.save.parquet("data/foo.parquet")# avro#df.save.avro("examples/data/foo.avro")Also you can save data to oracle, redshift, mysql and postgres. SeeDatabase connectionHandling Spark jars, packages and repositoriesWith optimus is easy to loading jars, packages and repos. You can init optimus/spark likeop=Optimus(repositories="myrepo",packages="org.apache.spark:spark-avro_2.12:2.4.3",jars="my.jar",driver_class_path="this_is_a_jar_class_path.jar",verbose=True)Create dataframesAlso you can create a dataframe from scratchfrompyspark.sql.typesimport*fromdatetimeimportdate,datetimedf=op.create.df([("names","str",True),("height(ft)","int",True),("function","str",True),("rank","int",True),("age","int",True),("weight(t)","float",True),("japanese name",ArrayType(StringType()),True),("last position seen","str",True),("date arrival","str",True),("last date seen","str",True),("attributes",ArrayType(FloatType()),True),("DateType"),("Tiemstamp"),("Cybertronian","bool",True),("NullType","null",True),],[("Optim'us",28,"Leader",10,5000000,4.3,["Inochi","Convoy"],"19.442735,-99.201111","1980/04/10","2016/09/10",[8.5344,4300.0],date(2016,9,10),datetime(2014,6,24),True,None),("bumbl#ebéé  ",17,"Espionage",7,5000000,2.0,["Bumble","Goldback"],"10.642707,-71.612534","1980/04/10","2015/08/10",[5.334,2000.0],date(2015,8,10),datetime(2014,6,24),True,None),("ironhide&",26,"Security",7,5000000,4.0,["Roadbuster"],"37.789563,-122.400356","1980/04/10","2014/07/10",[7.9248,4000.0],date(2014,6,24),datetime(2014,6,24),True,None),("Jazz",13,"First Lieutenant",8,5000000,1.80,["Meister"],"33.670666,-117.841553","1980/04/10","2013/06/10",[3.9624,1800.0],date(2013,6,24),datetime(2014,6,24),True,None),("Megatron",None,"None",10,5000000,5.70,["Megatron"],None,"1980/04/10","2012/05/10",[None,5700.0],date(2012,5,10),datetime(2014,6,24),True,None),("Metroplex_)^$",300,"Battle Station",8,5000000,None,["Metroflex"],None,"1980/04/10","2011/04/10",[91.44,None],date(2011,4,10),datetime(2014,6,24),True,None),],infer_schema=True).h_repartition(1)With .table() you have a beautifull way to show your data. You have extra information like column number, column data type and marked white spacesdf.table()Also you can create a dataframe from a panda dataframeimportpandasaspdpdf=pd.DataFrame({'A':{0:'a',1:'b',2:'c',3:'d'},'B':{0:1,1:3,2:5,3:7},'C':{0:2,1:4,2:6,3:None},'D':{0:'1980/04/10',1:'1980/04/10',2:'1980/04/10',3:'1980/04/10'},})s_pdf=op.create.df(pdf=pdf)s_pdf.table()Cleaning and ProcessingOptimus V2 was created to make data cleaning a breeze. The API was designed to be super easy to newcomers and very familiar for people that comes from Pandas.
Optimus expands the Spark DataFrame functionality adding .rows and .cols attributes.For example you can load data from a url, transform and apply some predefined cleaning functions:# This is a custom functiondeffunc(value,arg):return"this was a number"new_df=df\.rows.sort("rank","desc")\.withColumn('new_age',df.age)\.cols.lower(["names","function"])\.cols.date_transform("date arrival","yyyy/MM/dd","dd-MM-YYYY")\.cols.years_between("date arrival","dd-MM-YYYY",output_cols="from arrival")\.cols.remove_accents("names")\.cols.remove_special_chars("names")\.rows.drop(df["rank"]>8)\.cols.rename(str.lower)\.cols.trim("*")\.cols.unnest("japanese name",output_cols="other names")\.cols.unnest("last position seen",separator=",",output_cols="pos")\.cols.drop(["last position seen","japanese name","date arrival","cybertronian","nulltype"])You transform thisdf.table()Into thisnew_df.table()Note that you can use Optimus functions and Spark functions(.WithColumn()) and all the df function availables in a Spark Dataframe at the same time. To know about all the Optimus functionality please go to thisnotebooksHandling column outputWith Optimus you can handle how the output column from a transformation in going to be handled.frompyspark.sqlimportfunctionsasFdeffunc(col_name,attr):returnF.upper(F.col(col_name))If astringis passed toinput_colsandoutput_colsis not defined the result from the operation is going to be saved in the same input columnoutput_df=df.cols.apply(input_cols="names",output_cols=None,func=func)output_df.table()If astringis passed toinput_colsand astringis passed tooutput_colsthe output is going to be saved in the output columnoutput_df=df.cols.apply(input_cols="names",output_cols="names_up",func=func)output_df.table()If alistis passed toinput_colsand astringis passed toout_colsOptimus will concatenate the list with every element in the list to create a new column name with the outputoutput_df=df.cols.apply(input_cols=["names","function"],output_cols="_up",func=func)output_df.table()If alistis passed toinput_colsand alistis passed inout_colsOptimus will output every input column in the respective output columnoutput_df=df.cols.apply(input_cols=["names","function"],output_cols=["names_up","function_up"],func=func)output_df.table()Custom functionsSpark has multiple ways to transform your data like rdd, Column Expression, udf and pandas udf. In Optimus we created theapply()andapply_exprwhich handles all the implementation complexity.Here you apply a function to the "billingid" column. Sum 1 and 2 to the current column value. All powered by Pandas UDFdeffunc(value,args):returnvalue+args[0]+args[1]df.cols.apply("height(ft)",func,"int",[1,2]).table()If you want to apply a Column Expression useapply_expr()like this. In this case we pass an argument 10 to divide the actual column valuefrompyspark.sqlimportfunctionsasFdeffunc(col_name,args):returnF.col(col_name)/20df.cols.apply("height(ft)",func=func,args=20).table()You can change the table output back to ascii if you wishop.output("ascii")To return to HTML just:op.output("html")Data profilingOptimus comes with a powerful and unique data profiler. Besides basic and advance stats like min, max, kurtosis, mad etc,
it also let you know what type of data has every column. For example if a string column have string, integer, float, bool, date Optimus can give you an unique overview about your data.
Just rundf.profile("*")to profile all the columns. For more info about the profiler please go to thisnotebook.Let's load a "big" datasetdf=op.load.csv("https://raw.githubusercontent.com/ironmussa/Optimus/master/examples/data/Meteorite_Landings.csv").h_repartition()Numericop.profiler.run(df,"mass (g)",infer=False)op.profiler.run(df,"name",infer=False)Processing DatesFor dates data types Optimus can give you extra informationop.profiler.run(df,"year",infer=True)Profiler SpeedWithrelative_errorandapprox_countparams you can control how some operations are caculated so you can speedup the profiling in case is needed.relative_error: Relative Error for quantile discretizer calculation. 1 is Faster, 0 Slowerapprox_count: Useapprox_count_distinctorcountDistinct.approx_count_distinctis fasterop.profiler.run(df,"mass (g)",infer=False,relative_error=1,approx_count=True)PlotsBesides histograms and frequency plots you also have scatter plots and box plots. All powered by Apache by pysparkdf=op.load.excel("../examples/data/titanic3.xls")df=df.rows.drop_na(["age","fare"])You can output to the notebook or as an image# Output and imagedf.plot.frequency("age")df.plot.scatter(["fare","age"],buckets=30)df.plot.box("age")df.plot.correlation("*")Using other plotting librariesOptimus has a tiny API so you can use any plotting library. For example, you can usedf.cols.scatter(),df.cols.frequency(),df.cols.boxplot()ordf.cols.hist()to output a JSON that you can process to adapt the data to any plotting library.OutliersGet the ouliers using tukeydf.outliers.tukey("age").select().table()Remove the outliers using tukeydf.outliers.tukey("age").drop().table()df.outliers.tukey("age").info()You can also use z_score, modified_z_score or maddf.outliers.z_score("age",threshold=2).drop()df.outliers.modified_z_score("age",threshold=2).drop()df.outliers.mad("age",threshold=2).drop()Database connectionOptimus have handy tools to connect to databases and extract informacion. Optimus can handleredshift,postgres,oracleandmysqlfromoptimusimportOptimusop=Optimus(verbose=True)# This import is only to hide the credentialsfromcredentialsimport*# For others databases use in db_type accepts 'oracle','mysql','redshift','postgres'db=op.connect(db_type=DB_TYPE,host=HOST,database=DATABASE,user=USER,password=PASSWORD,port=PORT)# Show all tables namesdb.tables(limit="all")# # Show a summary of every tabledb.table.show("*",20)# # Get a table as dataframedf_=db.table_to_df("places_interest").table()# # Create new table in the databasedb.df_to_table(df,"new_table")Data enrichmentYou can connect to any external API to enrich your data using Optimus. Optimus uses MongoDB to download the data and then merge it with the Spark Dataframe. You need to install MongoDBLet's load a tiny dataset we can enrichdf=op.load.json("https://raw.githubusercontent.com/ironmussa/Optimus/master/examples/data/foo.json")importrequestsdeffunc_request(params):# You can use here whatever header or auth info you need to send.# For more information see the requests libraryurl="https://jsonplaceholder.typicode.com/todos/"+str(params["id"])returnrequests.get(url)deffunc_response(response):# Here you can parse de responsereturnresponse["title"]e=op.enrich(host="localhost",port=27017,db_name="jazz")df_result=e.run(df,func_request,func_response,calls=60,period=60,max_tries=8)df_result.table("all")df_result.table()Clustering StringsOptimus implements some funciton to cluster Strings. We get graet inspiration from OpenRefineHere a quote from its site:"In OpenRefine, clustering refers to the operation of "finding groups of different values that might be alternative representations of the same thing". For example, the two strings "New York" and "new york" are very likely to refer to the same concept and just have capitalization differences. Likewise, "Gödel" and "Godel" probably refer to the same person."For more informacion see this:https://github.com/OpenRefine/OpenRefine/wiki/Clustering-In-DepthKeycolisiondf=op.read.csv("../examples/data/random.csv",header=True,sep=";")fromoptimus.mlimportkeycollisionaskeyColdf_kc=keyCol.fingerprint_cluster(df,'STATE')df_kc.table()df_kc.table()keyCol.fingerprint_cluster(df,"STATE").to_json()df_kc=keyCol.n_gram_fingerprint_cluster(df,"STATE",2)df_kc.table()df_kc.table()keyCol.n_gram_fingerprint_cluster(df,"STATE",2).to_json()Nearest Neighbor Methodsfromoptimus.mlimportdistanceclusterasdcdf_dc=dc.levenshtein_matrix(df,"STATE")df_dc.table()df_dc=dc.levenshtein_filter(df,"STATE")df_dc.table()df_dc.table()df_dc=dc.levenshtein_cluster(df,"STATE")df_dc.table()df_dc.table()dc.to_json(df,"STATE")Machine LearningMachine Learning is one of the last steps, and the goal for most Data Science WorkFlows.Apache Spark created a library called MLlib where they coded great algorithms for Machine Learning. Now
with the ML library we can take advantage of the Dataframe API and its optimization to create Machine Learning Pipelines easily.Even though this task is not extremely hard, it is not easy. The way most Machine Learning models work on Spark
are not straightforward, and they need lots of feature engineering to work. That's why we created the feature engineering
section inside Optimus.One of the best "tree" models for machine learning is Random Forest. What about creating a RF model with just
one line? With Optimus is really easy.df_cancer=op.load.csv("https://raw.githubusercontent.com/ironmussa/Optimus/master/tests/data_cancer.csv")columns=['diagnosis','radius_mean','texture_mean','perimeter_mean','area_mean','smoothness_mean','compactness_mean','concavity_mean','concave points_mean','symmetry_mean','fractal_dimension_mean']df_predict,rf_model=op.ml.random_forest(df_cancer,columns,"diagnosis")This will create a DataFrame with the predictions of the Random Forest model.So lets see the prediction compared with the actual label:df_predict.cols.select(["label","prediction"]).table()The rf_model variable contains the Random Forest model for analysis.Contributing to OptimusContributions go far beyond pull requests and commits. We are very happy to receive any kind of contributionsincluding:Documentationupdates, enhancements, designs, or   bugfixes.Spelling or grammar fixes.README.md corrections or redesigns.Adding unit, or functionaltestsTriaging GitHub issues -- especially determining whether an issue still persists or is reproducible.Searching #optimusdata on twitterand helping someone else who needs help.Blogging, speaking about, or creating tutorialsabout Optimus and its many features.Helping others onDiscordBackers[Become a backer] and get your image on our README on Github with a link to your site.Sponsors[Become a sponsor] and get your image on our README on Github with a link to your site.Core TeamArgenis LeonContributors:https://github.com/ironmussa/Optimus/graphs/contributorsLicense:Apache 2.0 ©