Package: tx-parallex
parallexSystem RequirementsPython >= 3.8installDefault object storepip install tx-parallexPlasma storehttps://arrow.apache.org/pip install tx-parallex[arrow]Install from sourceClone the repoEasy install instructions:# Create a virtual environment called 'px'
conda create -n px python=3.8
# start-up the environment you just created
conda activate px
# install the rest of the tx-parallex pre-requirements
pip install -r requirements.txtTest# run the tests, a number of test 'specs'
PYTHONPATH=src pytest -x -vv --full-trace -s --timeout 60
# deactivate the environment (if desired)
conda deactivateset log levelset environment variable LOG_LEVEL to one of Python's logging library setLevel.IntroductionA queue with dependenciesUsagefrom tx.parallex import run_python

ret = run_python(number_of_workers = 4, pyf = "spec.py", dataf = "data.yml")Spectx-parallexspecs can be written in YAML or a Python-like DSL. The Python-like DSL is translated to YAML bytx-parallex. Each object in a spec specifies a task. When the task is executed, it is given a dict calleddata. The pipeline will return a dictionary.YAMLAssuming you have a functionsqrdefined in modulemathwhich returns the square of its argument.def sqr(x):
  return x * xletThelettask setsdatafor its subtask. It adds a new var value pair intodatawithin the scope of its subtask, and executes that task.Syntax:type: let
var: <var>
obj: <value>
sub: <subtask>Example:type: let
var: a
obj:
  data: 1
sub:
  type: python
  name: y
  mod: math
  func: sqr
  params: 
    x:
      name: amapThemaptask reads a listcollfromdataand applies a subtask to each member of the list. The members will be assigned tovarindatapassed to those tasksSyntax:type: map
coll: <value>
var: <variable name>
sub: <subtask><value>is an object of the form:Reference an entry indataor the name of a task"name": <variable name>Constant"data": <constant>Example:type: map
coll: 
  data:
  - 1
  - 2
  - 3
var: a
sub:
  type: python
  name: y
  mod: math
  func: sqr
  params: 
    x:
      name: acondThecondtask reads a boolean value and if it is true then it executes thethentask otherwise it executes theelsetask.Syntax:type: cond
on: <value>
then: <subtask>
else: <subtask>Example:type: cond
on: 
  data:
    true
then:
  type: ret
  obj:
    data: 1
else:
  type: ret
  obj:
    data: 0pythonYou can use any Python module.Thepythontask runs a Python function. It reads parameters fromdata. The return value must be pickleable.Syntax:type: python
name: <name>
mod: <module>
func: <function>
params: <parameters><parameters>is an object of the form:<param> : <value>
...
<param> : <value>where<param>can be either name or position.Example:type: python
  name: y
  mod: math
  func: sqr
  params: 
    x:
      data: 1topThetoptask toplogically sorts subtasks based on their dependencies and ensure the tasks are executed in parallel in the order compatible with those dependencies.Syntax:type: top
sub: <subtasks>It reads thenameproperties of subtasks that are not in data.Example:type: top
sub:
- type: python
  name: y
  mod: math
  func: sqr
  params: 
    x:
      data: 1
- type: python
  name: z
  mod: math
  func: sqr
  params: 
    x:
      name: yseqTheseqtask forces all subtasks to be run sequentially.Syntax:type: top
sub: <subtasks>It reads thenameproperties of subtasks that are not in data.Example:type: seq
sub:
- type: python
  name: y
  mod: math
  func: sqr
  params: 
    x:
      data: 1
- type: python
  name: z
  mod: math
  func: sqr
  params: 
    x:
      name: yretretspecify a value. The pipeline will return a dictionary. When a task appears under amaptask, it is prefix with the index of the element in that collection as following<index>For nested maps, the indices will be chained together as followings<index>. ... .<index>Syntax:type: ret
obj: <value>Example:type: ret
obj: 
    name: zPythonA dsl block contains a subset of Python.There is a semantic difference from python. Any assignment in block is not visiable outside of the block.Assignment within a block are unorderedreturn statementAvailable syntax:importfrom <module> import *
from <module> import <func>, ..., <func>import names from module<module>absolute module namesassignment<var> = <const>where<const> = <integer> | <number> | <boolean> | <string> | <list> | <dict>This translates tolet.Example:a = 1
y = sqr(x=a)
yield yfunction application<var> = [<module>.]<func>(<param>=<expr>, ...) | <expr>This translate topython.
where<var>isname<expr>is<expr> = <expr> if <expr> else <expr> | <expr> <binop> <expr> | <expr> <boolop> <expr> | <expr> <compare> <expr> | <unaryop> <expr> | <var> | <const><binop>,<boolop>and<compare>and<unaryop>are python BinOp, BoolOp, Compare, and UnaryOp.<expr>is translated to a set of assignments,name, ordatadepending on its content.Example:y = math.sqr(1)
z = math.sqr(y)
return zparallel forfor <var> in <expr>:
    ...This translates tomap.Example:for a in [1, 2, 3]:
  y = math.sqr(a)
  yield yifif <expr>:
    ...
else:
    ...This translates tocond.Example:if z:
    yield 1
else:
    yield 0The semantics of if is different from python, variables inside if is not visible outsidewithwith Seq:
    ...This translates toseq.Example:with Seq:
    y = math.sqr(1)
    return yyieldyield <expr>This translates toret.Example:y = math.sqr(1)
return yDatadata can be arbitrary yaml