Project: Reactive Interaction Gateway (Orchestration & Management / API Gateway)
File: CHANGELOG.md

All notable changes to this project will be documented in this file. The format is based on , and this project adheres to . Support publishing events consumed from topics. See the for how to get started. Added validation for reverse proxy configuration. Now it crashes RIG on start when configuration is not valid or returns `400` when using REST API to update configuration. Added basic distributed tracing support in with Jaeger and Openzipkin exporters. RIG opens a span at the API Gateway and emits trace context in Cloud Events following the . Added possibility to set response code for `responsefrom` messages in reverse proxy (`kafka` and `httpasync`). Added new version - `v3` - for internal endpoints to support response code in the `/responses` endpoint Added Helm v3 template to the `deployment` folder Added detailed features summary on the website with architecture diagrams. Added for the JWT Blacklist feature. Added longpolling examples to the `/examples` folder Support JSON and Google Cloud Logger (GCL) log formats. Set the new environment variable `LOG_FMT` to `JSON` or `GCL` to see this in action. Added CORS headers to unauthenticated Proxy requests Added rate limiting for maximum number of WS + SSE + Longpolling connections per minute. Configurable via `MAXCONNECTIONSPER_MINUTE` env var, by default it's 5000. Added basic setup for the Phoenix LiveDashboard. Now it's accessible at `:4010/dashboard`. Added Prometheus metrics for events, subscriptions and blacklist. Check the for more info. Added custom Grafana dashboard. Check the for more info. Incorporated to handle binary and structured modes for in a proper way. This introduces some breaking changes: Binary mode is now using `ce_` prefix for CloudEvents context attribute headers, before it was `ce-` - done according to the Change above affects also `"response_from": "kafka"` proxy functionality. RIG will forward to clients only Kafka body, no headers. This means, when using binary mode, clients receive only the data part, no CloudEvents context attributes. Changed `responsefrom` handler to expect a message in binary format, **NOT** a cloud event (`kafka` and `httpasync`). Updated Helm v2 template, kubectl yaml file and instructions in the `deployment` folder Publish Helm Chart to Github pages. With this change, we can simply install the chart using ```shell helm repo add accenture https://accenture.github.io/reactive-interaction-gateway helm install rig accenture/reactive-interaction-gateway ``` More information, follow the . make README smaller, easier to read and highlight features. Updated Phoenix LiveDashboard setup to show also metrics based on the Prometheus metrics (for now only proxy and events metrics). Updated to use and NodeJS 14. Updated to use NodeJS 14. Updated Kafka SSL configuration to be able to set (skip) only specific SSL certificates. This is needed to allow connection to Azure Event Hubs. See more in the . Fixed a bug where distributed set processes would crash when one of their peers has died but hasn't been removed yet from the pg2 group. Fixed wrong endpoint validation for reverse proxy. Now it should correctly check for `path` or `pathregex`. Before it would require `path` even with `pathregex` in place.

### Deprecated --> Removed deprecated or unused code/functionality, these are breaking changes : Removed deprecated internal API `/v1`. Removed deprecated environment variables: `PROXYKAFKAREQUESTAVRO`, `PROXYKAFKAREQUESTTOPIC`, `PROXYKINESISREQUEST_STREAM`. This means that you can set topic and schema for publishing to event streams only in the proxy config as described in the . Removed experimental feature Firehose (forwarding events to an HTTP endpoint). Removed `path` field in proxy configuration. Reason is that the `path_regex` field is already covering `path` functionality and thus it doesn't make sense to have both of them. Should cause less confusion and improve maintainability. Migration: `"path": "/foo"` -> `"path_regex": "/foo"` `"path": "/foo/{id}"` -> `"path_regex": "/foo/(.+)"` - or pretty much whatever regex you need (e.g. UUID pattern) Update to Erlang/OTP 23.2.2, which fixes a critical TLS certificate verification issue. Updated dependencies to support OTP 23. We've also replaced the versions file with `.tool-versions`, which makes it easier for those using the - just run `asdf install` to obtain the correct versions of Erlang and Elixir. Added possibility to define Kafka/Kinesis topic and schema per reverse proxy endpoint. The current solution using environment variables is deprecated, but still used as a fallback -- will be removed in the version 3.0. Added Kinesis + Localstack example. Upgrade the Elixir version to 1.10 for source code and Docker images. Upgrade version for multiple dependencies. Added integration for easier Slack access - check the main page badge! In addition to SSE and WebSocket, RIG now also supports HTTP long-polling for listening to events. Frontends should only use this as a fallback in situations where neither SSE nor WebSocket is supported by the network. When terminating an SSE connection after its associated session has been blacklisted, RIG now sends out a `rig.session_killed` event before closing the socket. For WebSocket connections, the closing frame contains "Session killed." as its payload. New API for querying and updating the session blacklist: `/v2/session-blacklist`, which introduces the following breaking changes (`/v1/session-blacklist` is unaffected) : When a session has been added to the session blacklist successfully, the endpoint now uses the correct HTTP status code "201 Created" instead of "200 Ok". When using the API to blacklist a session, the `validityInSeconds` should now be passed as an integer value (using a string still works though). Fixed usage of external check for `SUBMISSIONCHECK` and `SUBSCRIPTIONCHECK`. Logging incoming HTTP request to Kafka works again and now also supports Apache Avro. Fixed HTTP response for `DELETE 4010/v1/apis/apiid` and `DELETE 4010/v2/apis/apiid` to correctly return `204` and no content. Removed the `JWTBLACKLISTDEFAULTEXPIRYHOURS` environment variable (). A connection is now associated to its session right after the connection is established, given the request carries a JWT in its authorization header. Previously, this was only done by the subscriptions endpoint, which could cause a connection to remain active even after blacklisting its authorization token. Upgrade the Elixir and Erlang versions for source code and Docker images. Automated UI-tests using Cypress make sure that all examples work and that code changes do not introduce any unintended API changes. Refactor JWT related code in favor of `RIG.JWT`. Fix flaky cypress tests; this shouldn't be an issue anymore when running Travis

Increased maximum number of Erlang ports from 4096 to 65536 to allow more HTTP connections. New Prometheus metric: `rigproxyrequests_total`. For details see . The respond-via-Kafka feature uses a correlation ID for associating the response with the original request. This correlation ID is now cryptographically verified, which prevents an attacker on the internal network to reroute responses to other users connected to RIG. Apache Avro is now supported for consuming from, and producing to, Kafka. The implementation uses the Confluent Schema Registry to fetch Avro schemas. Added new set of topics in documentation about the API Gateway, even streams and scaling. Added examples section to the documentation website. Added new `responsefrom` option -- `httpasync` together with new internal `POST` endpoint `/v1/responses`. You can send correlated response to `/v1/responses` and complete initial Proxy request. Implement . A special fallback to "structured mode" in case the content type is "application/json" and the "ce-specversion" header is not set ensures this change is backward compatible with existing setups. New request body format for endpoints with `kafka` and `kinesis` targets; see below. The environment variable `KAFKAGROUPID` has been replaced with the following environment variables, where each of them has a distinct default value: `KAFKATOFILTERKAFKAGROUPID`, `KAFKATOHTTPKAFKAGROUPID`, `PROXYKAFKARESPONSEKAFKAGROUP_ID`. The default Kafka source topic for the Kafka-to-HTTP event stream has been changed to `rig`. The feature was introduced to forward all incoming events to an (external) HTTP endpoint, so it makes sense to use the default topic for incoming events here too. Changed `:refresh_subscriptions` GenServer handler from `call` to `cast` to improve performance. Fixed a bug that caused the subscriptions endpoint to return an internal server error when running RIG in a clustered setup. Support for forwarding HTTP/1.1 responses over a HTTP/2 connection by dropping connection-related HTTP headers. Added missing `id` field to swagger spec for `message` API. Fixed random generation of group IDs for Kafka consumer groups. This led to wrong partition distribution when using multiple RIG nodes. Now consumers will have the same ID which can be changed via environment variable - defaults to `rig`. When forwarding an HTTP request, the request header is now set to the `target_url` defined by the proxy configuration. Fixed missing `swagger.json` file in production Docker image. Added missing CORS headers for Kafka/Kinesis target type when not using `response_from`. Fixed schema registry validation when using binary messages in Kafka consumer. Forwarding events to HTTP did not contain (all) Kafka messages, as the Kafka consumer group ID was shared with the consumer for forwarding events to frontends. Endpoints configured with target `kafka` or `kinesis` now expect a different body format (that is, the previous format is deprecated). This aligns the request body format with the other endpoints that accept CloudEvents. For example, instead of using this: ```json { "partition": "the-partition-key", "event": { "specversion": "0.2", "type": "whathashappened", "source": "ui", "id": "123" } } ``` you should put the partition key in the CloudEvent's "rig" extension instead: ```json { "specversion": "0.2", "rig": { "target_partition": "the-partition-key" }, "type": "whathashappened", "source": "ui", "id": "123" } ``` When using the proxy, RIG will now add an additional . Increased length of header value in HTTP requests to 16384 to support long tokens for

HTTPS certificates may now be passed using absolute paths. (Previously, the locations of the HTTPS certificates were limited to the OTP-applications' `priv` directories `rigapi/priv/cert` and `riginbound_gateway/priv/cert`.) Additionally, for security reasons we no longer include the self-signed certificate with the docker image. Please adapt your environment configuration accordingly. Validation errors for SSE & WS connections and the subscriptions endpoint should now be a lot more helpful. Invalid JWTs, as well as invalid subscriptions, cause the endpoints to respond with an error immediately. Parsing of JSON files in proxy module - `api.id` was expected to be an atom, but when using files it's a string. Kinesis: Support for CloudEvents versions 0.1 and 0.2. Fixed channels example with latest RIG API changes. Fixed sse/ws examples to use JWT inferred subscriptions correctly. Prometheus monitoring endpoint. The proxy configuration can now also be passed as a JSON string. This allows to run the Docker image in environments where mounting a file in a container is not possible. Rate limiting. Upgraded a dependency to fix the Docker build. A library upgrade caused idle SSE connections to time out after 60 seconds. This timeout is now disabled. HTTP/2 and HTTPS support. The SSE and WebSocket endpoints now take a "subscriptions" parameter that allows to create (manual) subscriptions (JSON encoded list). This has the same effect as establishing a connection and calling the subscriptions endpoint afterwards. OpenAPI (Swagger) documentation for RIG's internal API. Support for the CloudEvents v0.2 format. In API definitions regular expressions can now be used to define matching request paths. Also, request paths can be rewritten (see for an example). The SSE and WebSocket endpoints' "token" parameter is renamed to "jwt" (to not confuse it with the connection token). When forwarding requests, RIG related meta data (e.g. correlation ID) in CloudEvents is now put into an object under the top-level key "rig". Note that in terms of the current specification this makes "rig" an . Also, all RIG related keys have been renamed from snake_case to camelCase. Previously API definitions for proxy were turning on security check for endpoints by `not_secured: false` which is a bit confusing -- changed to more readable form `secured: true`. No longer assumes the "Bearer" token type when no access token type is prepended in the Authorization header. Consequently, a client is expected to explicitly use "Bearer" for sending its JWT authorization token. More more details, see . All events that RIG creates are now in CloudEvents v0.2 format (before: CloudEvents v0.1). When using Kafka or Kinesis as the target, connection related data is added to the event before publishing it to the respective topic/partition. With the introduction of CloudEvents v0.2, RIG now follows the CloudEvent extension syntax with all fields put into a common top-level object called "rig". Additionally, the object's field names have been changed slightly to prevent simple mistakes like case-sensitivity issues. Also, the expected request body fields have been renamed to be more descriptive. To that end, usage information returned as plaintext should help the API user in case of a Bad

Extractor configuration reload Fixed response to CORS related preflight request. JWT now supports RS256 algorithm in addition to HS256. Support Kafka SSL and SASL/Plain authentication. Add new endpoints at `/_rig/v1/` for subscribing to CloudEvents using SSE/WS, for creating subscriptions to specific event types, and for publishing CloudEvents. Expose setting for proxy response timeout. Subscriptions inference using JWT on SSE/WS connection and subscription creation. Allow publishing events to Kafka and Kinesis via reverse-proxy HTTP calls. Optionally, a response can be waited for (using a correlation ID). Simple event subscription examples for SSE and WS. Kafka/Kinesis firehose - set topic/stream to consume and invoke HTTP request when event is consumed. SSE heartbeats are now sent as comments rather than events, and events without data carry an empty data line to improve cross-browser compatibility. General documentation and outdated info. Previous SSE/WS communication via Phoenix channels. Events that don't follow the CloudEvents spec are no longer supported (easy migration: put your event in a CloudEvent's `data` field). Flaky tests in `router_test.exs` -- switching from `Bypass` to `Fakeserver`. Channels example. Amazon Kinesis integration. Use lazy logger calls for debug logs. Format (most files) using Elixir 1.6 formatter. Add new endpoint `POST /messages` for sending messages (=> Kafka is no longer a hard dependency). Add a dedicated developer guide. Release configuration in `rel/config.exs` and custom `vm.args` (based on what distillery is using). Production configuration for peerage to use DNS discovery. Module for auto-discovery, using `Peerage` library. Kubernetes deployment configuration file. Smoke tests setup and test cases for API Proxy and Kafka + Phoenix messaging. Kafka consumer ready check utility function. List of all environment variables possible to set in `guides/operator-guide.md`. Possibility to set logging level with env var `LOG_LEVEL`. Variations of Dockerfiles - basic version and AWS version. Helm deployment chart. Proxy is now able to do request header transformations. Endpoint for terminating a session no longer contains user id in path. Convert to umbrella project layout. Move documentation from `doc/` to `guides/` as the former is the default for ex_doc output. Revised request logging (currently Kafka and console as backends). Disable WebSocket timeout. Dockerfile to use custom `vm.args` file & removed `mix release.init` step. Make presence channel respect `JWTUSERFIELD` setting (currently hardcoded to "username"). Set proper environment variable for Phoenix server `INBOUND_PORT`. Set proper environment variable for Phoenix server `API_PORT`. Channels example fixed to be compatible with version 2.0.0. User defined query auth values are no longer overridden by `JWT` auth type. Handle content-type correctly. More strict regex match for routes in proxy. Downcased response headers to avoid duplicates in proxy. Basic Travis configuration. Configuration ADR document. Websocket and SSE channels example. Maintain changelog file. Increase default rate limits. Make producing of Kafka messages in proxy optional (and turned off by default). Fix Travis by disabling credo rule `Design.AliasUsage`. Add `mix docs` script to generate documentation of code base. Add ethics documentation such as code of conduct and contribution guidelines. Update configuration to be able to modify almost anything by environment variables on RIG start. Rework Dockerfile to use multistage approach for building RIG Docker image. Update entire code base to use `rig` keyword. Disable Origin checking.