Package: hark-lang
The Hark Programming LanguageFormerly, Teal.Change your remotes:git remote set-url origin git@github.com:condense9/hark-lang.gitHark hides the complexity of AWS Lambda + SQS, so you can build serverless data
workflows without managing infrastructure.Describe your workflows in areal programming languagewith first-class
functions, concurrency, and native Python inter-op. Test end-to-end locally,
then deploy to serverless AWS infrastructure in under 60s and start workflows
from anything that can invoke Lambda.Like AWS Step Functions but cheaper and much nicer to use (overheads: a little
Lambda runtime, and a DynamoDB for Hark state).Like Serverless Framework, but handles runtime glue logic in addition to
deployment.Status: Hark works well for small workflows: 5-10 Lambda invocations. Larger
workflows may cause problems, and there is a known issue caused by DynamoDB
restrictions (#12).Get started in 2 minutes.Read the documentation.PyCon Africa 2020 Demos!.ContributingHark is growing rapidly, and contributions arewarmly welcomed.Is Hark for me?Harkisfor you if:You use Python for processing data, or writing business process workflows.You want an alternative to AWS Step Functions.You don't want to to deploy and manage a task platform (Airflow, Celery, etc).Data in: You can invoke Hark like any Lambda function (AWS cli, S3 trigger,
API gateway, etc).Data out: Use the Python libraries you already have for database access.
Hark just connects them together.Development: Hark runs locally, so you can thoroughly test Hark programs
before deployment (using minio and localstack for any additional infrastructure
that your code uses.Operating: Hark enables contextual cross-thread logging and stacktraces out
of the box, since the entire application is described in one place.Hark is like...But...AWS Step FunctionsHark programs aren't bound to AWS and don't use Step Functions under the hood (just plain Lambda + DynamoDB).Orchestrators (Apache Airflow, etc)You don't have to manage infrastructure, or think in terms of DAGs, and you can test everything locally.Task runners (Celery, etc)You don't have to manage infrastructure.Azure Durable FunctionsWhile powerful, Durable Functions (subjectively) feel complex - their behaviour isn't always obvious.Read more...Up and running in 2 minutesAll you need:An AWS account, andAWS CLIconfigured.A Python 3.8 virtual environmentHark is built with Python, and distributed as a Python package. To install it,
run:$pipinstallhark-langThis gives you theharkexecutable. Tryhark -h.Copy the following snippet into a file calledservice.hk:// service.hk

fn main() {
  print("Hello World!");
}Run it (-f mainis optional, andmainis the default):~/new_project$>harkservice.hk-fmainInitialise the project (required for deployment):~/new_project$>harkinitAnd deploy the service to your AWS account (requires AWS credentials andAWS_DEFAULT_REGIONto be defined):~/new_project$>harkdeployFinally, invoke it in AWS (-f mainis optional, as before):~/new_project$>harkinvoke-fmainThat's it! You now have a Hark instance configured in your AWS account, built on
the AWS serverless platform (S3 + Lambda + DynamoDB).More info...Explore a more complex example:Fractals.Create an issueif none of this
makes sense, or you'd like help getting started.Read more...about the languageabout the development processabout configuring HarkWhy should I learn a new language?It's a big ask! There'sso muchthat's missing from a brand new language. For
now, think about it like learning a new library or API -- you can do most of the
hard work in regular Python, using existing packages and code, while Hark lets
you express things you can't easily do in Python.They key concept is this: when running in AWS, Hark threads run in separate
lambda invocations, and the language comes with primitives to manage these
threads.Concurrency & SynchronisationThis is useful when a set computations are related, and must be kept together./*** Return f(x) + g(x), computing f(x) and g(x) in parallel in two separate* threads (Lambda invocations in AWS).*/fncompute(x){a=asyncf(x);// Start computing f(x) in a new threadb=asyncg(x);// Likewise with g(x)awaita+awaitb;// Stop this thread, and resume when {a, b} are ready}Traditional approach: Manually store intermediate results in an external
database, and build the synchronisation logic into the cloud functionsfandg, or use an orchestrator service.Read more...Trivial PipelinesUse this approach when each individual function may take several minutes (and
hence, together would break the 5 minute AWS Lambda limit)./*** Compute f(g(h(x))), using a separate lambda invocation for each* function call.*/fnpipeline(x){a=asynch(x);b=asyncg(awaita);f(awaitb);}Traditional approach:This is functionally similar to a "chain" of AWS Lambda
functions and SQS queues.Mapping / reducingHark functions are first-class, and can be passed around (closures and anonymous
functions are planned, giving Hark object-oriented capabilities)./*** Compute [f(element) for element in x], using a separate lambda invocation for* each application of f.*/fnmap(f,x,accumulator){ifnullp(x){accumulator}else{// The Hark compiler has tail-recursion optimisationmap(func,rest(x),append(accumulator,asyncf(first(x))))}}This could be used like:fnadd2(x){x+2}fnmain(){futures=map(add2,[1,2,3,4],[]);// ...}Read more...Notes about syntaxThe syntax should look familiar, but there are a couple of things to point out.No 'return' statementEvery expression must return a value, so there is noreturnstatement. The
last expression in a 'block' (expressions between{and}) is returned
implicitly.fnfoo(){"something"}fnmain(){print(foo())// -> prints "something"}Semi-colons are required...... when there is more than one expression in a block.This is ok:fnmain(){print("done")}So is this:fnmain(){print("one");print("two")}And this:fnmain(){print("one");print("two");}But this is not ok:fnmain(){print("one")// <- missing semicolon!print("two")}'print' returns the value printedIn this snippet, "Hello Worlds!" is actually printed twice. First inbar, then
inmain.fnbar(){print("Hello Worlds!")}fnmain(){print(bar())}$>hark-qservice.hk
HelloWorlds!
HelloWorlds!'if' is an expression, and returns a valueThink about it like this: Anifexpression represents a choice betweenvalues.v=ifsomething{true_value}else{false_value};// if 'something' is not true, v is set to nullv=ifsomething{value};FAQWhy is this not a library/DSL in Python?When Hark threads wait on a Future, they stop completely. The Lambda function
saves the machine state and then terminates. When the Future resolves, the
resolving thread restarts any waiting threads by invoking new Lambdas to pick up
execution.To achieve the same thing in Python, the framework would need to dump the entire
Python VM state to disk, and then reload it at a later point -- this may be
possible, but would certainly be non-trivial. An alternative approach would be
to build a langauge on top of Python that looked similar to Python, but harkwrongbecause it was really faking things under the hood.How is Hark like Go?Goroutines are very lightweight, while Harkasyncfunctions are pretty heavy --
they involve creating a new Lambda (or process, when running locally).Hark's concurrency model is similar to Go's, but channels are not fully
implemented so data can only be sent to/from a thread at call/return points.Is this an infrastructure-as-code tool?No, Hark does not do general-purpose infrastructure management. There are
already great tools to do that (Terraform,Pulumi,Serverless
Framework, etc).Instead, Hark reduces the amount of infrastructure you need. Instead of a
distinct Lambda function for every piece of application logic, you only need the
core Hark interpreter (purely serverless) infrastructure.Hark will happily manage that infrastructure for you (throughhark deployandhark destroy), or you can set it up with your in-house custom system.Current Limitations and RoadmapHark is beta quality, which means that it's not thoroughly tested or feature
complete. This is a non-exhaustive list.LibrariesOnly one Hark program file is supported, but a module/package system isplanned.Error HandlingThere's no error handling - if your function fails, you'll have to restart the
whole process manually. An exception handling system isplanned.TypingFunction inputs and outputs aren't typed. This is a limitation, and will be
fixed soon, probably usingProtoBufsas the interface
definition language.Calling Arbitrary ServicesCurrently you can only call Hark or Python functions -- arbitrary microservices
can't be called. Before Hark v1.0 is released, this will be possible. You will
be able to call a long-running third party service (e.g. an AWS ML service) as a
normal Hark function andawaiton the result.AboutHark is maintained byCondense9 Ltd.. Get in touch
withric@condense9.comfor help getting running, or if you
need enterprise deployment.Hark started because we couldn't find any data engineering tools that were
productive andharklike software engineering. As an industry, we've spent
decades growing a wealth of computer science knowledge, but building data
pipelines in $IaC, or manually crafting workflow DAGs with $AutomationTool,just isn't software.LicenseApache License (Version 2.0). SeeLICENSEfor details.