Package: lftakakura-mage-ai
🧙 A modern replacement for Airflow.Documentation🌪️Get a 5 min overview🌊Play with live tool🔥Get instant helpGive your data teammagicalpowersIntegrateand synchronize data from 3rd party sourcesBuild real-time and batch pipelines totransformdata using Python, SQL, and RRun, monitor, andorchestratethousands of pipelines without losing sleep1️⃣ 🏗️BuildHave you met anyone who said they loved developing in Airflow?That’s why we designed an easy developer experience that you’ll enjoy.Easy developer experienceStart developing locally with a single command or launch a dev environment in your cloud using Terraform.Language of choiceWrite code in Python, SQL, or R in the same data pipeline for ultimate flexibility.Engineering best practices built-inEach step in your pipeline is a standalone file containing modular code that’s reusable and testable with data validations. No more DAGs with spaghetti code.↓2️⃣ 🔮PreviewStop wasting time waiting around for your DAGs to finish testing.Get instant feedback from your code each time you run it.Interactive codeImmediately see results from your code’s output with an interactive notebook UI.Data is a first-class citizenEach block of code in your pipeline produces data that can be versioned, partitioned, and cataloged for future use.Collaborate on cloudDevelop collaboratively on cloud resources, version control with Git, and test pipelines without waiting for an available shared staging environment.↓3️⃣ 🚀LaunchDon’t have a large team dedicated to Airflow?Mage makes it easy for a single developer or small team to scale up and manage thousands of pipelines.Fast deployDeploy Mage to AWS, GCP, or Azure with only 2 commands using maintained Terraform templates.Scaling made simpleTransform very large datasets directly in your data warehouse or through a native integration with Spark.ObservabilityOperationalize your pipelines with built-in monitoring, alerting, and observability through an intuitive UI.🧙 IntroMage is an open-source data pipeline tool for transforming and integrating data.Quick startDemoTutorialsDocumentationFeaturesCore design principlesCore abstractionsContributing🏃‍♀️ Quick startYou can install and run Mage using Docker (recommended),pip, orconda.Install using DockerCreate a new project and launch tool (changedemo_projectto any other name if you want):dockerrun-it-p6789:6789-v$(pwd):/home/srcmageai/mageai\/app/run_app.shmagestartdemo_projectIf you want to run Mage locally on a different port, change the first port after-pin the command above. For example, to change the port to6790, run:dockerrun-it-p6790:6789-v$(pwd):/home/srcmageai/mageai\/app/run_app.shmagestartdemo_projectWant to use Spark or other integrations? Read more aboutintegrations.Openhttp://localhost:6789in your browser and build a pipeline.If you changed the Docker port for running Mage locally, go to the urlhttp://127.0.0.1:[port](e.g.http://127.0.0.1:6790) in your browser to
view the pipelines dashboard.UsingpiporcondaInstall Mage(a) To the current virtual environment:pipinstallmage-aiorcondainstall-cconda-forgemage-ai(b) To a new virtual environment (e.g.,myenv):python3-mvenvmyenvsourcemyenv/bin/activate
pipinstallmage-aiorcondacreate-nmyenv-cconda-forgemage-ai
condaactivatemyenvFor additional packages (e.g.spark,postgres, etc), please seeInstalling extra packages.If you run into errors, please seeInstall errors.Create a new project and launch tool (changedemo_projectto any other name if you want):magestartdemo_projectOpenhttp://localhost:6789in your browser and build a pipeline.🎮 DemoLive demoBuild and run a data pipeline with ourdemo app.WARNINGThe live demo is public to everyone, please don’t save anything sensitive (e.g. passwords, secrets, etc).Demo video (5 min)Click the image to play video👩‍🏫 TutorialsLoad data from API, transform it, and export it to PostgreSQLIntegrate Mage into an existing Airflow projectTrain model on Titanic datasetSet up dbt models and orchestrate dbt runs🔮Features🎶OrchestrationSchedule and manage data pipelines with observability.📓NotebookInteractive Python, SQL, & R editor for coding data pipelines.🏗️Data integrationsSynchronize data from 3rd party sources to your internal destinations.🚰Streaming pipelinesIngest and transform real-time data.❎dbtBuild, run, and manage your dbt models with Mage.A sample data pipeline defined across 3 files ➝Load data ➝@data_loaderdefload_csv_from_file():returnpd.read_csv('default_repo/titanic.csv')Transform data ➝@transformerdefselect_columns_from_df(df,*args):returndf[['Age','Fare','Survived']]Export data ➝@data_exporterdefexport_titanic_data_to_disk(df)->None:df.to_csv('default_repo/titanic_transformed.csv')What the data pipeline looks like in the UI ➝New? We recommend reading aboutblocksand
learning from ahands-on tutorial.🏔️Core design principlesEvery user experience and technical design decision adheres to these principles.💻Easy developer experienceOpen-source engine that comes with a custom notebook UI for building data pipelines.🚢Engineering best practices built-inBuild and deploy data pipelines using modular code. No more writing throwaway code or trying to turn notebooks into scripts.💳Data is a first-class citizenDesigned from the ground up specifically for running data-intensive workflows.🪐Scaling is made simpleAnalyze and process large data quickly for rapid iteration.🛸Core abstractionsThese are the fundamental concepts that Mage uses to operate.ProjectLike a repository on GitHub; this is where you write all your code.PipelineContains references to all the blocks of code you want to run, charts for visualizing data, and organizes the dependency between each block of code.BlockA file with code that can be executed independently or within a pipeline.Data productEvery block produces data after it's been executed. These are called data products in Mage.TriggerA set of instructions that determine when or how a pipeline should run.RunStores information about when it was started, its status, when it was completed, any runtime variables used in the execution of the pipeline or block, etc.🙋‍♀️ Contributing and developingAdd features and instantly improve the experience for everyone.Check out thecontributing guideto set up your development environment and start building.👨‍👩‍👧‍👦 CommunityIndividually, we’re a mage.🧙 MageMagic is indistinguishable from advanced technology.
A mage is someone who uses magic (aka advanced technology).
Together, we’re Magers!🧙‍♂️🧙 Magers (/ˈmājər/)A group of mages who help each other realize their full potential!
Let’s hang out and chat together ➝For real-time news, fun memes, data engineering topics, and more, join us on ➝TwitterLinkedInGitHubSlack🤔 Frequently Asked Questions (FAQs)Check out ourFAQ pageto find answers to some of our most asked questions.🪪 LicenseSee theLICENSEfile for licensing information.