Package: anacostia-pipeline
Welcome to the Anacostia Pipeline!Requirements:Python 3.11 or greater, if you need to update Python, here are the steps:Updating Python on Mac
Follow these steps to update your Python installation to the latest version on Mac:Check Current Python Version
Open a terminal and use the python3 --version command to check which Python version you have installed.
For example:python3 --version 
Python 3.7.4Install Homebrew
If you don't already have Homebrew installed, you can install it with:/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"Update Homebrew
Before installing Python, update Homebrew's package index with:brew updateSearch for Latest Python
Now search Brew for the latest available Python version:brew search pythonMake note of the latest 3.x release.Install Latest Python
Use brew to install the latest Python version:brew install python@3.11Remove Old Python Version
Remove the outdated Python version, replacing 3.7 with your current outdated version:sudo rm -rf /Library/Frameworks/Python.framework/Versions/3.7Set New Python as Default
Set the new Python as the default by adding it to your PATH in bash profile:source ~/.zshrcConfirm New Version
Check that the new version is now set as default:python --versionHow to install the pipeline:Create the python virtual env (Linux/Mac)python3 -m venv venv
source venv/bin/activateCreate the python virtual env (Windows)python -m venv venv
venv\Scripts\activatepip install package locallypip install -e .Update the requirements.txt if neccessaryOnce you have all requirements installed, let's try to run one of the tests:
Go into the tests folder and run test_phase1.py:python3 test_phase1.pyCreate a .env file inside the tests folder:
Insert these credentials and set them:.envUSERNAME=
PASSWORD=Other notes:
If you already have a Python virtual environment set up, follow these steps:Check Python version inside virtual environment:(venv) python --versionFor example:(venv) Python 3.8.2Deactivate the virtual environment:(venv) deactivateCheck system Python version:python --versionThis may be older than your virtual environment version.
4. Check Python 3 version:python3 --version5. Create a new virtual environment with Python 3:python3 -m venv venv6. Activate the new virtual environment:source venv/bin/activate7. Confirm Python version is updated:(venv) python --versionUnderstanding the repo:Engine folder - has two important files 1. node.py - which contains all teh base classes for the action node and the resource nodes, other nodes inherit the base node logic. This is where all the synctionaization primitives are defined. 2. pipeline.py is our way of intereacting with the dag, that will have commands for staring, terminating, shutting down, pausing the node and pipelines, The ability to export the graph.json file is here as well.
constants.py defines the different statuses we have in the pipeline for the nodes.Resource Folder: Defines the basic anacostia resources that users have right off the bat. It has 3 files:
2.1 data_store.py refines where the user would input their data files in order to be consumed by the pipeline. Ex: if user has folder of images, it would take that folder and start keeping track of the state for that folder via data_store.json
2.2 feature_store.py - default anocostia datastore, this file converts features into a numpy array and saves the numpy array as a .npy file and keeps track of the state.
2.3 model_registry - where models are going to be kept. Keeps state of models i.e current, old, and new(this is applicatable to anacostia resources)Tests folder - where all the tests for pipeline and nodes reside
The most uptodate is tests_phase1.py which includes all teh tests for the phase 1 configuration which is the ETL pipeline.
tests_data_store.py includes all the tests for the data store
tests_feature_store includes all the tests for the feature store
tests_model_registry include all tests for model registry - this is currently not up to date
tests_utils. includes all functionaliteis for creating the tests
tests_node and test_basic includes all tests for nodes which might not be used anymore. We might get rid of these.setup.py - the main folder that is used to turn ana into a python package that we can put on PyPI - python package index. Allows you topip install anacostiaFuture work:Take the tests from phase1 and create the suite of tests for phase 2, which is a full retraining pipeline without model evaluation.We will create a metadata store node using a database (Most likely Mongo)We will upgrade phase 2 tests to include model evaluation (now you can have full retraining pipeline)Example Usagen1=TrueNode(name='n1')n2=FalseNode(name='n2')n3=TrueNode(name='n3',listen_to=n1|n2)n4=TrueNode(name="n4",listen_to=n3)p1=Pipeline(nodes=[n1,n2,n3,n4])