Package: opal-common-fork
⚡OPAL⚡Open Policy Administration LayerOPAL is an administration layer for Open Policy Agent (OPA), detecting changes to both policy and policy data in realtime and pushing live updates to your agents.OPAL brings open-policy up to the speed needed by live applications.
As your application state changes (whether it's via your APIs, DBs, git, S3 or 3rd-party SaaS services), OPAL will make sure your services are always in sync with the authorization data and policy they need (and only those they need).Check out our main site atOPAL.acand thisMicrosoft video briefly explaining OPAL and how it works with OPA.Table of contentsGetting StartedIntro to OPALWhy use OPALArchitectureKey ConceptsHOW-TOsDesign ChoicesJoin the Community🛠️ Getting startedOPAL is available both aspython packageswith a built-in CLI as well as pre-builtdocker imagesready-to-go.Getting started with the pre-built docker containersPlay with a live playground environment in docker-composeTry the getting started guide for containersCheck out the Helm Chart for KubernetesGetting started with the python packages and CLIInstallpip install opal-clientpip install opal-serverRun server (example):# Run server#  in secure mode -verifying client JWTs (Replace secrets with actual secrets ;-) )exportOPAL_AUTH_PRIVATE_KEY=~/opalexportOPAL_AUTH_PUBLIC_KEY=~/opal.pubexportOPAL_AUTH_MASTER_TOKEN="RANDOM-SECRET-STRING"#  Watching a GIT repository from a webhookexportOPAL_POLICY_REPO_URL=https://github.com/authorizon/opal-example-policy-repo.gitexportOPAL_POLICY_REPO_WEBHOOK_SECRET="RANDOM-SECRET-STRING-SHARED-WITH-GITHUB"opal-serverrunRun client (example):# Run client#  authenticating with a JWT (replace 'JWT-CRYPTOGRAPHIC-CONTENT' with actual token )exportOPAL_CLIENT_TOKEN="JWT-CRYPTOGRAPHIC-CONTENT"# connect to serverexportOPAL_SERVER_URL=https://opal.mydomain.com:7002# Subscribe to specific data-topicsexportOPAL_DATA_TOPICS=tenants/my-org,stripe_billing,tickets
opal-clientrunTry it yourself -Read the getting started guide📖 Introduction to OPAL - data and policy realtime deliveryModern applications are complex, distributed, multi-tenant and run at scale - often creating overwhelming authorization challenges. OPA (Open-Policy-Agent) brings the power of decoupled policy to the infrastructure layer (especially K8s), and light applications. OPAL supercharges OPA to meet the pace of live applications, where the state relevant to authorization decisions may change with every user click and api call.OPAL builds on top of OPA adding realtime updates (via Websocket Pub/Sub) for both policy and data.OPAL embraces decoupling of policy and code, and doubles down on decoupling policy (git driven) and data (distributed data-source fetching engines).Why use OPALOPAL is the easiest way to keep your solution's authorization layer up-to-date in realtime.OPAL aggregates policy and data from across the field and integrates them seamlessly into the authorization layer.OPAL is microservices and cloud-native (seekey conceptsbelow)Why OPA + OPAL == 💪 💜OPA (Open Policy Agent) is great! It decouples policy from code in a highly-performant and elegant way. But the challege of keeping policy agents up-to-date is hard - especially in applications - where each user interaction or API call may affect access-control decisions.
OPAL runs in the background, supercharging policy-agents, keeping them in sync with events in realtime.What OPALis notA Policy Engine:OPAL uses a policy-engine, but isn't one itselfCheck outOpen-Policy-Agent, andOSOLarge scale Global FGA:Currently OPAL is not meant for managing ridiculous (>100GB) amounts of data  within one layer. Though it can complement a CDN to achieve a similar result -see below.Check outGoogle-ZanzibarFullstack authorization:OPAL and OPA essentially provide microservices for authorizationDevelopers still need to add control interfaces on top (e.g. user-management, api-key-management, audit, impersonation, invites) both as APIs and UIsCheck outauthorizon📡  ArchitectureSee amore detailed diagramOPAL consists of two key components that work together:OPAL ServerCreates a Pub/Sub channel clients subscribe toTracks a git repository (via webhook / polling) for updates to policy (or static data)Additional versioned repositories can be supported (e.g. S3, SVN)Accepts data update notifications via Rest APIpushes updates to clients (as diffs)scales with other server instances via a configurable backbone pub/sub (Currently supported: Postgres, Redis, Kafka; more options will be added in the future)OPAL ClientDeployed alongside a policy-agent, and keeping it up to dateSubscribes to Pub/Sub updates, based on topics for data and policyDownloads data-source configurations from serverFetches data from multiple sources (e.g. DBs, APIs, 3rd party services)Downloads policy from serverKeeps policy agents up to dateFurther readingArchitecturedeep diveCode modulesreview💡 Key ConceptsOPAL is realtime (with Pub/Sub updates)OPAL is all about easily managing your authorization layer in realtime.This is achieved by aWebsocket Pub/Subchannel between OPAL clients and servers.Each OPAL-client (and through it each policy agent) subscribes to and receives updates instantly.OPAL is statelessOPAL is designed for scale, mainly via scaling out both client and server instances; as such, neither are stateful.State is retained in the end components (i.e: the OPA agent, as an edge cache) and data-sources (e.g. git, databases, API servers)OPAL is extensibleOPAL's Pythonic nature makes extending and embedding new components extremely easy.Built with typed Python3,pydantic, andFastAPI- OPAL is balanced just right for stability and fast development.A key example is OPAL's FetchingEngine and FetchProviders.
Want to use authorization data from a new source (a SaaS service, a new DB, your own proprietary solution)? Simplyimplement a new fetch-provider.👩‍🏫HOW-TOsHow to get started with OPAL (Packages and CLI)How to get started with OPAL (Container Images)How to trigger Data Updates via OPALHow to extend OPAL to fetch data from your sources with FetchProviders🗿 FoundationsOPAL is built on the shoulders of open-source giants, including:Open Policy Agent- the default policy agent managed by OPAL.FastAPI- the ASGI API framework used by OPAL-servers and OPAL-clients.FastAPI Websocket PubSub- powering the live realtime update channelsBroadcasterallowing syncing server instances through a backend backbone (e.g. Redis, Kafka)🎨 Design choicesNetworkingOPAL creates a highly efficient communications channel usingwebsocket Pub/Sub connectionsto subscribe to both data and policy updates. This allows OPAL clients (and the services they support) to be deployed anywhere - in your VPC, at the edge, on-premises, etc.By usingoutgoingwebsocket connections to establish the Pub/Sub channel most routing/firewall concerns are circumnavigated.Using Websocket connections allows network connections to stay idle most of the time, saving CPU cycles for both clients and servers (especially when comparing to polling-based methods).Implementation with PythonOPAL is written completely in Python3 using asyncio, FastAPI and Pydantic.
OPAL was initially created as a component ofauthorizon.com, and we've chosen Python for development speed, ease of use and extensibility (e.g. fetcher providers).Python3 with coroutines (Asyncio) and FastAPI has presentedsignificant improvements for Python server performance. While still not on par with Go or Rust - the results match and in some cases even surpass Node.js.PerformanceIt's important to note that OPALdoesn't replacethe direct channel to the policy-engine - so for example with OPA all authorization queries are processed directly by OPA's Go based engine.Pub/Sub benchmarks - While we haven't run thorough benchmarksyet, we are using OPAL in production - seeing its Pub/Sub channel handle 100s of events per second per server instance with no issue.Decouple Data from PolicyOpen Policy Agent sets the stage forpolicy codeandpolicy datadecoupling by providing separate APIs to manage each.OPAL takes this approach a step forward by enabling independent update channels for policy code and policy data, mutating the policy agent cache separately.Policy(Policy as code): is code, and as such is naturally maintained best within version control (e.g. git). OPAL allows OPA agents to subscribe to the subset of policy that they need directly from source repositories (as part of CI/CD or independently).Data: OPAL takes a more distributed approach to authorization data - recognizing that there aremanypotential data sources we'd like to include in the authorization conversation (e.g. billing data, compliance data, usage data, etc etc). OPAL-clients can be configured and extended to aggregate data from any data-source into whichever service needs it.Decouple data/policy management from policy agentsOPAL was built initially with OPA in mind, and OPA is mostly a first-class citizen in OPAL. That said OPAL can support various and multiple policy agents, even in parallel - allowing developers to choose the best policy agent for their needs.FGA, large scale / global authorization (e.g. Google Zanzibar)OPAL is built for fine grained authorizon (FGA), allowing developers to aggregate all and any data they need and restructure it for the authorization layer.OPAL achieves this by making sure each policy-agent is loaded with only the data it needs via topic subscriptions (i.e: data focus and separation).Examples of data separation: the back-office service doesn't need to know about customer users, a tenant specific service doesn't need the user list of other tenants, ...That said OPAL is still limited by OPA'sresource utilization capacity.If the size of the dataset you need to load into OPA cache is huge (i.e: > 5GB), you may opt to pass this specific dataset byoverloading inputto your policy.OPAL can still help you if you decide toshard your datasetacross multiple OPA agents. Each agent's OPAL-client can subscribe only to the relevant shard.For these larger scale cases, OPAL can potentially become a link between a solution like Google Zanzibar (or equivalent CDN) and local policy-agents, allowing both Google-like scales, low latency, and high performance.If you're developing such a service, or considering such high-scale scenarios; you're welcome to contact us, and we'd be happy to share our plans for OPAL in that area.Using OPAL for other live update needsWhile OPAL was created and primarily designed for open-policy and authorization needs; it can be generically applied for other live updates and data/code propagation needsIf you'd like to use OPAL or some of its underlying modules for other update cases - please contact us (See below), we'd love to help you do that.Administration capabilities and UIWe've already built policy editors, back-office, frontend-embeddable interfaces, and more as part ofauthorizon.com.We have plans to migrate more parts ofauthorizon.comto be open-source; please let us know what you'd like to see next.Joining the communityWe are eager to hear from you 😃Raise questions and ask for features to be added to the road-map in ourGithub discussionsReport issues inGithub issuesChat with us in ourSlack communityContacting us (the authors)We love talking about authorization, open-source, realtime communication, and tech in general.Feel free to reach out to us on ourGitHub discussionsor directly overemail.Contributing to OPALPull requests are welcome! (please make sure to includepassingtests and docs)Prior to submitting a PR - open an issue on GitHub, or make sure your PR addresses an existing issue well.