Package: lineapy
Capture, analyze, and transform messy notebooks into data pipelineswith just two lines of code.Ask questions or learn about our workshops on ourSlack!👇 Try It Out! 👇https://user-images.githubusercontent.com/13392380/169427654-487d8d4b-3eda-462a-a96c-51c151f39ab9.mp4What Problems Can LineaPy Solve?Use Case 1: Cleaning Messy NotebooksUse Case 2: Revisiting Previous WorkUse Case 3: Building PipelinesGetting StartedPrerequisitesInstallationInterfacesJupyter and IPythonCLIQuick StartUsage ReportingWhat Next?What Problems Can LineaPy Solve?Use Case 1: Cleaning Messy NotebooksWhen working in a Jupyter notebook day after day, it's easy to write messy code — You might execute cells out of order, execute the same cell repeatedly, and edit or delete cells until you've acquired good results, especially when generating tables, models, and charts. This highly dynamic and interactive notebook use, however, can introduce some issues. Our colleagues may not be able to reproduce our results by rerunning our notebook, and worse still, we ourselves may forget the steps required to produce our previous results.One way to avoid this problem is to keep the notebook in sequential order by constantly re-executing
the entire notebook during development. This approach, however, interrupts our natural workflows and stream of
thoughts, decreasing our productivity. Therefore, it is much more common to clean up the notebook after development. This is a time-consuming process that is not immune from the reproducibility issues caused by deleted cells and out-of-order cell executions.To see how LineaPy can help with messy notebooks, check outthisdemo or.Use Case 2: Revisiting Previous WorkData science is often a team effort where one person's work relies on results from another's. For example, a data scientist building a model may use features engineered by other colleagues. When using results generated by other people, we may encounter data quality issues including missing values, suspicious numbers, and unintelligible variable names. When we encounter these issues, we may need to check how these results came into being in the first place. Often, this means tracing back the code that was used to generate the result in question. In practice, this can be a challenging task because we may not know who produced the result. Even if we know who to ask, that person might not remember where the exact version of the code is stored, or worse, may have overwritten the code without version control. Additionally, the person may no longer be at the organization and may not have handed over the relevant knowledge. In any of these cases, it becomes extremely difficult to identify the root any issues, rendering the result unreliable and even unusable.To see how LineaPy can help here, check outthisdemo or.Use Case 3: Building PipelinesAs our notebooks become more mature, we may use them like pipelines. For example, our notebook might process the latest data to update a dashboard, or pre-process data and dump it into the file system for downstream model development. To keep our results up-to-date, we might be expected to re-execute these processes on a regular basis. Running notebooks manually is a brittle process that's prone to errors, so we may want to set up proper pipelines for production. If relevant engineering support is not available, we need to clean up and refactor our notebook code so that it can be used in orchestration systems or job schedulers, such as cron, Apache Airflow, Argo, Kubeflow, DVC, or Ray. Of course, this assumes that we already know how these tools work and how to use them — If not, we need to spend time learning about them in the first place! All this operational work is time-consuming, and detracts from the time that we can spend on our core duties as a data scientist.To see how LineaPy can help here, check outthisdemo or.Getting StartedLineaPy is a Python package for capturing, analyzing, and automating data science workflows.
At a high level, LineaPy traces the sequence of code execution to form a comprehensive understanding
of the code and its context. This understanding allows LineaPy to provide a set of tools that help
data scientists bring their work to production more quickly and easily, with justtwo linesof code.Check thissectionfor types of problems that LineaPy can help to solve.PrerequisitesLineaPy runs onPython>=3.7,<3.11andIPython>=7.0.0. It does not come with a Jupyter installation,
so you will need toinstall onefor interactive computing.InstallationTo install LineaPy, run:pipinstalllineapyIf you want to run the latest version of LineaPy directly from the source, follow instructionshere.LineaPy offers several extras to extend its core capabilities, such as support for PostgreSQL or Amazon S3.
Learn more about these and other installation optionshere.InterfacesJupyter and IPythonTo use LineaPy in an interactive computing environment such as Jupyter Notebook/Lab or IPython, load its extension by executing the following command at the top of your session:%load_extlineapyPlease note:You must run this as the first command in a given session. Executing it
in the middle of a session will lead to erroneous behaviors by LineaPy.This command loads the extension for the current session only. It does not carry over to different sessions, so you will need to repeat it for each new session.Alternatively, you can launch the environment with thelineapycommand, like so:lineapyjupyternotebooklineapyjupyterlablineapyipythonThis will automatically load the LineaPy extension in the corresponding interactive shell application,
and you will not need to manually load it for every new session.NOTE:If your Jupyter environment has multiple kernels, choosePython 3 (ipykernel)whichlineapydefaults to.CLIYou can also use LineaPy as a CLI command or runnable Python module. To see available options, run the following commands:# LineaPy as a CLI commandlineapypython--helpor# LineaPy as a runnable Python modulepython-mlineapy--helpQuick StartOnce LineaPy is installed and loaded, you are ready to start using the package. Let's look at a simple
example using theIris datasetto demonstrate
how to use LineaPy to 1) store a variable's history, 2) get its cleaned-up code,
and 3) build an executable pipeline for the variable.The following development code fits a linear regression model to the Iris dataset:importlineapyimportpandasaspdimportmatplotlib.pyplotaspltfromsklearn.linear_modelimportLinearRegression# Load dataurl="https://raw.githubusercontent.com/LineaLabs/lineapy/main/examples/tutorials/data/iris.csv"df=pd.read_csv(url)# Map each species to a colorcolor_map={"Setosa":"green","Versicolor":"blue","Virginica":"red"}df["variety_color"]=df["variety"].map(color_map)# Plot petal vs. sepal width by speciesdf.plot.scatter("petal.width","sepal.width",c="variety_color")plt.show()# Create dummy variables encoding speciesdf["d_versicolor"]=df["variety"].apply(lambdax:1ifx=="Versicolor"else0)df["d_virginica"]=df["variety"].apply(lambdax:1ifx=="Virginica"else0)# Initiate the modelmod=LinearRegression()# Fit the modelmod.fit(X=df[["petal.width","d_versicolor","d_virginica"]],y=df["sepal.width"],)Let's say you're happy with your above code, and you've decided to save the trained model. You can store the model as a LineaPyartifactwith the following code:# Save the model as an artifactlineapy.save(mod,"iris_model")A LineaPy artifact encapsulates both the valueandcode, so you can easily retrieve
the model's code, like so:# Retrieve the model artifactartifact=lineapy.get("iris_model")# Check code for the model artifactprint(artifact.get_code())The print statement will output:import pandas as pd
from sklearn.linear_model import LinearRegression

url = "https://raw.githubusercontent.com/LineaLabs/lineapy/main/examples/tutorials/data/iris.csv"
df = pd.read_csv(url)
color_map = {"Setosa": "green", "Versicolor": "blue", "Virginica": "red"}
df["variety_color"] = df["variety"].map(color_map)
df["d_versicolor"] = df["variety"].apply(lambda x: 1 if x == "Versicolor" else 0)
df["d_virginica"] = df["variety"].apply(lambda x: 1 if x == "Virginica" else 0)
mod = LinearRegression()
mod.fit(
    X=df[["petal.width", "d_versicolor", "d_virginica"]],
    y=df["sepal.width"],
)Note that these are the minimal essential steps to produce the model. That is, LineaPy has automatically
cleaned up the original code by removing extraneous operations that do not affect the model (e.g., plotting).Let's say you're asked to retrain the model on a regular basis to account for any updates in the source data.
You need to set up a pipeline to train the model — LineaPy makes this as simple as a single function call:lineapy.to_pipeline(artifacts=["iris_model"],input_parameters=["url"],# Specify variable(s) to parametrizepipeline_name="iris_model_pipeline",output_dir="output/",framework="AIRFLOW",)This command generates several files that can be used to execute the pipeline from the UI or CLI. (Check thistutorialfor more details.)In short, LineaPy automates time-consuming, manual steps in a data science workflow, helping us get
our work to production more quickly and easily.Usage ReportingLineaPy collects anonymous usage data that helps our team to improve the product.
Only LineaPy's API calls and CLI commands are being reported.
We strip out as much potentially sensitive information as possible, and we will
never collect user code, data, variable names, or stack traces.You can opt-out of usage tracking by setting environment variable:exportLINEAPY_DO_NOT_TRACK=trueWhat Next?To learn more about LineaPy, please check out the projectdocumentationwhich contains many examples you can follow with. Some key resources include:ResourceDescriptionDocsThis is our knowledge hub — when in doubt, start here!ConceptsLearn about key concepts underlying LineaPy!TutorialsThese notebook tutorials will help you better understand core functionalities of LineaPyUse CasesThese domain examples illustrate how LineaPy can help in real-world applicationsAPI ReferenceNeed more technical details? This reference may help!ContributeWant to contribute? These instructions will help you get set up!SlackHave questions or issues unresolved? Join our community and ask away!