Package: apache-liminal-test-spark
Apache LiminalApache Liminal is an end-to-end platform for data engineers & scientists, allowing them to build,
train and deploy machine learning models in a robust and agile way.The platform provides the abstractions and declarative capabilities for
data extraction & feature engineering followed by model training and serving.
Liminal's goal is to operationalize the machine learning process, allowing data scientists to
quickly transition from a successful experiment to an automated pipeline of model training,
validation, deployment and inference in production, freeing them from engineering and
non-functional tasks, and allowing them to focus on machine learning code and artifacts.BasicsUsing simple YAML configuration, create your own schedule data pipelines (a sequence of tasks to
perform), application servers,  and more.Getting StartedA simple getting stated guide for Liminal can be foundhereApache Liminal DocumentationFull documentation of Apache Liminal can be foundhereHigh Level ArchitectureHigh level architecture documentation can be foundhereExample YAML config file---name:MyLiminalStackowner:Bosco Albert Baracusvolumes:-volume:myvol1local:path:/Users/me/myvol1pipelines:-pipeline:my_pipelinestart_date:1970-01-01timeout_minutes:45schedule:0 * 1 * *metrics:namespace:TestNamespacebackends:['cloudwatch']tasks:-task:my_python_tasktype:pythondescription:static input taskimage:my_python_task_imgsource:write_inputsenv_vars:NUM_FILES:10NUM_SPLITS:3mounts:-mount:mymountvolume:myvol1path:/mnt/vol1cmd:python -u write_inputs.py-task:my_parallelized_python_tasktype:pythondescription:parallelized python taskimage:my_parallelized_python_task_imgsource:write_outputsenv_vars:FOO:BARexecutors:3mounts:-mount:mymountvolume:myvol1path:/mnt/vol1cmd:python -u write_inputs.pyservices:-service:name:my_python_servertype:python_serverdescription:my python serverimage:my_server_imagesource:myserverendpoints:-endpoint:/myendpoint1module:my_serverfunction:myendpoint1funcInstallationInstall this repository (HEAD)pipinstallgit+https://github.com/apache/incubator-liminal.gitOptional: set LIMINAL_HOME to path of your choice (if not set, will default to ~/liminal_home)echo'export LIMINAL_HOME=</path/to/some/folder>'>>~/.bash_profile&&source~/.bash_profileAuthoring pipelinesThis involves at minimum creating a single file called liminal.yml as in the example above.If your pipeline requires custom python code to implement tasks, they should be organizedlike thisIf your pipeline  introduces imports of external packages which are not already a part
of the liminal framework (i.e. you had to pip install them yourself), you need to also provide
a requirements.txt in the root of your project.Testing the pipeline locallyWhen your pipeline code is ready, you can test it by running it locally on your machine.Ensure you have The Docker engine running locally, and enable a local Kubernetes cluster:And allocate it at least 3 CPUs (under "Resources" in the Docker preference UI).If you want to execute your pipeline on a remote kubernetes cluster, make sure the cluster is configured
using :kubectlconfigset-context<yourremotekubernetescluster>Build the docker images used by your pipeline.In the example pipeline above, you can see that tasks and services have an "image" field - such as
"my_static_input_task_image". This means that the task is executed inside a docker container, and the docker container
is created from a docker image where various code and libraries are installed.You can take a look at what the build process looks like, e.g.hereIn order for the images to be available for your pipeline, you'll need to build them locally:cd</path/to/your/liminal/code>
liminalbuildYou'll see that a number of outputs indicating various docker images built.Create a kubernetes local volumeIn case your Yaml includes working withvolumesplease first run the following command:cd</path/to/your/liminal/code>liminalcreateDeploy the pipeline:cd</path/to/your/liminal/code>liminaldeployNote: after upgrading liminal, it's recommended to issue the commandliminaldeploy--cleanThis will rebuild the airlfow docker containers from scratch with a fresh version of liminal, ensuring consistency.Start the serverliminalstartStop the serverliminalstopDisplay the server logsliminallogs--follow/--tail

Numberoflinestoshowfromtheendofthelog:
liminallogs--tail=10Followlogoutput:
liminallogs--followNavigate tohttp://localhost:8080/adminYou should see yourThe pipeline is scheduled to run according to thejson schedule: 0 * 1 * *field in the .yml file you provided.To manually activate your pipeline:
Click your pipeline and then click "trigger DAG"
Click "Graph view"
You should see the steps in your pipeline getting executed in "real time" by clicking "Refresh" periodically.ContributingMore information on contributing can be foundhereRunning Tests (for contributors)When doing local development and running Liminal unit-tests, make sure to set LIMINAL_STAND_ALONE_MODE=True