Package: pyoptimus
OptimusOverviewOptimus is an opinionated python library to easily load, process, plot and create ML models that run over pandas, Dask, cuDF, dask-cuDF, Vaex or Spark.Some amazing things Optimus can do for you:Process using a simple API, making it easy to use for newcomers.More than 100 functions to handle strings, process dates, urls and emails.Easily plot data from any size.Out of box functions to explore and fix data quality.Use the same code to process your data in your laptop or in a remote cluster of GPUs.See DocumentationTry OptimusTo launch a live notebook server to test optimus using binder or Colab, click on one of the following badges:Installation (pip):In your terminal just type:pip install pyoptimusBy default Optimus install Pandas as the default engine, to install other engines you can use the following commands:EngineCommandDaskpip install pyoptimus[dask]cuDFpip install pyoptimus[cudf]Dask-cuDFpip install pyoptimus[dask-cudf]Vaexpip install pyoptimus[vaex]Sparkpip install pyoptimus[spark]To install from the repo:pip install git+https://github.com/hi-primus/optimus.git@develop-23.5To install other engines:pip install git+https://github.com/hi-primus/optimus.git@develop-23.5#egg=pyoptimus[dask]RequirementsPython 3.7 or 3.8ExamplesYou can go to10 minutes to Optimuswhere you can find the basics to start working in a notebook.Also you can go to theExamplessection and find specific notebooks about data cleaning, data munging, profiling, data enrichment and how to create ML and DL models.Here's a handyCheat Sheetwith the most common Optimus' operations.Start OptimusStart Optimus using"pandas","dask","cudf","dask_cudf","vaex"or"spark".fromoptimusimportOptimusop=Optimus("pandas")Loading dataNow Optimus can load data in csv, json, parquet, avro and excel formats from a local file or from a URL.#csvdf=op.load.csv("../examples/data/foo.csv")#jsondf=op.load.json("../examples/data/foo.json")# using a urldf=op.load.json("https://raw.githubusercontent.com/hi-primus/optimus/develop-23.5/examples/data/foo.json")# parquetdf=op.load.parquet("../examples/data/foo.parquet")# ...or anything elsedf=op.load.file("../examples/data/titanic3.xls")Also, you can load data from Oracle, Redshift, MySQL and Postgres databases.Saving Data#csvdf.save.csv("data/foo.csv")# jsondf.save.json("data/foo.json")# parquetdf.save.parquet("data/foo.parquet")You can also save data to oracle, redshift, mysql and postgres.Create dataframesAlso, you can create a dataframe from scratchdf=op.create.dataframe({'A':['a','b','c','d'],'B':[1,3,5,7],'C':[2,4,6,None],'D':['1980/04/10','1980/04/10','1980/04/10','1980/04/10']})Usingdisplayyou have a beautiful way to show your data with extra information like column number, column data type and marked white spaces.display(df)Cleaning and ProcessingOptimus was created to make data cleaning a breeze. The API was designed to be super easy to newcomers and very familiar for people that comes from Pandas.
Optimus expands the standard DataFrame functionality adding.rowsand.colsaccessors.For example you can load data from a url, transform and apply some predefined cleaning functions:new_df=df\.rows.sort("rank","desc")\.cols.lower(["names","function"])\.cols.date_format("date arrival","yyyy/MM/dd","dd-MM-YYYY")\.cols.years_between("date arrival","dd-MM-YYYY",output_cols="from arrival")\.cols.normalize_chars("names")\.cols.remove_special_chars("names")\.rows.drop(df["rank"]>8)\.cols.rename("*",str.lower)\.cols.trim("*")\.cols.unnest("japanese name",output_cols="other names")\.cols.unnest("last position seen",separator=",",output_cols="pos")\.cols.drop(["last position seen","japanese name","date arrival","cybertronian","nulltype"])Need help? 🛠️FeedbackFeedback is what drive Optimus future, so please take a couple of minutes to help shape the Optimus' Roadmap:http://bit.ly/optimus_surveyAlso if you want to a suggestion or feature request usehttps://github.com/hi-primus/optimus/issuesTroubleshootingIf you have issues, see ourTroubleshooting GuideContributing to Optimus 💡Contributions go far beyond pull requests and commits. We are very happy to receive any kind of contributionsincluding:Documentationupdates, enhancements, designs, or bugfixes.Spelling or grammar fixes.README.md corrections or redesigns.Adding unit, or functionaltestsTriaging GitHub issues -- especially determining whether an issue still persists or is reproducible.Blogging, speaking about, or creating tutorialsabout Optimus and its many features.Helping others on our official chatsBackers and SponsorsBecome abackeror asponsorand get your image on our README on Github with a link to your site.