Project: deepstream (App Definition and Development / Streaming & Messaging)
File: CHANGELOG.md

Separate docker ubuntu amd and arm releases due to intermitent build failures in arm. See https://github.com/nodejs/docker-node/issues/1335 http auth issue #1138 multi arch docker image builds. Thanks @daanh432 update deps. Thanks @hugojosefson Cork uWebsocketjs http responses update uWebsocketjs use cli/config file options for cluster deepstream cluster CLI enabled in order to run a cluster of deepstream servers on each available processor core combine monitoring: now deepstream accepts an array of monitoring plugins in order to have separate plugins when it comes to monitoring messages, server activity and other custom functionality that might be required. For example you can run the included log monitoring, and an audit plugin that could save to another system/server which users are writing to which records by listening to incoming messages. This allows to create inmediate database replication strategies and so forth. set pkg entrypoint for daemon log monitoring daemon issues due to pkg bug pkg issues Update nodejs to version 18.x Update uWebsockets Update deps Enforce http origins options log a warning when a websocket message is not sent uws server add maxBackpressure option, defaults to 1024*1024 update deps eliminate externalUrl from config and cluster messages since it was not used set `provideRequestorName` and `provideRequestorData` as false by default to avoid overhead since from now on that data can actually be sent on rpc messages and handled on @deepstream/client > 6.0.2. update submodules and deps Revert to lockfileVersion 1 Check maxMessageSize limit on POST requests when using uws http server update @deepstream/client dev dependency and update e2e tests Send write ack error messages when cache, storage or permission fails. For this to work it requires client version >= 6 Do not remove cluster nodes if cluster size is 1 Enable vertical cluster node option: Now we can use all available cores in a vertical cluster! By setting the cluster option to `clusterNode: { name: 'vertical' }` you can run as many deepstream instances you want in each of the server cores and it will be one syncronized cluster. Check that executable works in pre-push hook Use pkg instead of nexe Update nodejs to version 14.x Update uWebsockets Update dependencies Update dependencies Disable telemetry enable combine authentication. Now when the auth config has more than one authentication strategy the server will query them in order untill one passes or all fail. enable querying for specific user presence on http endpoint uws idleTimeout is in seconds! That's why it didn't closed the connection on time. buffer ack messages remove noDelay from default-options and use it on subscription registry as a param to enable/disable buffering Github Actions CI/CD Updating uws to support apple silicon Increasing CI heartbeat timeouts Calling destroy on socketWrapper instead of close, since close is a reaction and destroy an action Fix #1091 heartbeat not working with node-http/ws This fix now adds a timestamps to every message frame recieved and sets up an interval per socket to check heartbeat exists. Didn't do any performance tests but I'm assuming having a single interval is cheaper than setting a timeout and canceling on each message. This fix also exposes that we don't serialize STATETOPICREGISTRY/NOT_SUBSCRIBED message correctly which throws an error and crashes the server. That fix will require more elbow grease and hasn't been reported so will just keep an eye on issues. Fix #1089 install service gives error Updating dependencies Fixed bug causing the server to crash with older sdks. Use upgrade property for uWebSockets http

Do not run e2e:v3 test on the pre-push hook Updating all dependencies Disabling telemetry for tests Fixing critical bug with client sdk version tracking Adding telemetry. The server code is also in the rep (under telemetry-server). This uses a random uuid as your deploymentId, which is pretty much the only way I can avoid having thousands of records in the database from one machine restarting / ci process. If possible please use a different ID for production environments! ```yaml enabledFeatures: record: true event: true rpc: true presence: true telemetry: type: deepstreamIO options: enabled: true debug: false deploymentId: <uuid goes here> ``` Fixes by the awesome @jaime-ez around heartbeats and ping messages! Add two new plugins: heap-snapshot This allows deepstream to save its heap space for analysis by v8 tools ```yaml plugins: heap-snapshot: name: 'heap-snapshot' options: interval: 60000 outputDir: file(../heap-snapshots) ``` aws This is a general plugin for all AWS services, currently allows us to sync the heap-snapshot directory to S3 which is useful when running via docker. The functionality however is generic, so you could have a plugin that outputs other useful data. You can also very simply add more services (or more of the same ones). This just makes it easier for us to maintain the plugin. ```yaml plugins: aws: name: aws options: accessKeyId: ${AWSACCESSKEY} secretAccessKey: ${AWSSECRETACCESS_KEY} services: type: s3-sync options: syncInterval: 60000 syncDir: file(../heap-snapshots) bucketName: ${SYNCBUCKETNAME} bucketRegion: ${AWSDEFAULTREGION} ``` SocketData is passed to monitoring service in order to allow for fined grain monitoring. uws service accepts options requests and enforces CORS params Allowing valve file to be passed in via nodeJS config (@jaime-ez) Updating uws dependency Adding a log line for MQTT incoming connections for clarity Fixing issue where record updates via clusters didn't always get sent correctly to subscribers Updating dependencies Fixing issue where sending messages between multiple protocols can break. Verbose logging will be removed in the next release. adding debug logs for listening, and allowing to subscribers to listen to self Fixing log level output as debug logs are being ignored Attempt to fix npm publishing issue due to travis bug Call onClientDisconnect with userId instead of socket Fixes in storage auth handler by @abird Updating all dependencies Call onClientDisconnect from combined auth handler Adding winston to docker and package image Fixed critical issue where invalid websocket frames kill server Provide all HTTP headers to auth endpoint when using ws Adding a log monitoring plugin, useful for kibana log forwarding Adding header authentication for http monitoring Production only dependency did not include ds-types Close down MQTT server on stop Replaces ENV variables in config loaded using fileLoader Fixes odd types scripting issue breaking plugin interfaces when using `deepstream.getServices()` Fixes hash generation via CLI New License Singular HTTP Service SSL Support reintroduced Better Config file validation JSON Logger NGINX Helper Combined authentication handler Embedded dependencies Builtin HTTP Monitoring Storage authentication endpoint Guess whats back, official clustering support! Custom authentication plugins now have to use the async/await API Custom permission handlers need to slightly tweak the function arguments Deployment configuration has to be (simplified) to respect the single HTTP/Websocket port You can see the upgrade guide for backwards compatibility Removing dom lib from typescript, which exposed variables that throw exceptions #1008 Cannot read property 'N' of undefined #920 Hardening the config validator Allow empty password for mqtt endpoint when authentication is not enabled (@Aapkostka) #1003 ResolvePluginClass should look for lowercase

#1002 RPC not routed to different deepstream node depending on startup order #1001 Updating dependencies Adding health-checks for all ws based endpoints. Remove conflicting port for those starting using node with an empty config object. Limit the dead socket log to reduce insane spam. Two new connection endpoints have been added. They are currently experimental and will be properly announced with associated documentation. One endpoint is mqtt! This allows us to support mqtt auth (using username and password), retain using records and QoS 1 using write acks. The only issue is since mqtt only supports one sort of concept (with flags distinguishing them) we bridge both events and records together. That means if you subscribe to 'temperature/london', you'll get the update from both a client doing `event.emit('temperature/london')` and `record.setData('temperature/london')`. The second endpoint is `ws-json` which allows users to interact with deepstream by just passing through json serialized text blobs instead of protobuf. This is mainly to help a few people trying to write SDKs without the hassle of a protobuf layer. Value also injects a `name` variable which allows you to reference the name your currently in. Useful for cross referencing. Subscription registry seemed to have a massive leak when it came to dead sockets! This has now been fixed. The sockets seemed to have gotten the remove event deleted earlier in their lifecycle which prohibited it from doing a proper clean up later. Backwards compatibility with V3 clients / text protocol using a ws-text connection endpoint This has a couple of small differences, like `has` is no longer supported and `snapshot` errors are exposed using the global `error` callback instead of via the response. Otherwise all the e2e tests work, and best of all you can run both at the same time if you want to run JS 4.0 and Java 3.0 simultaneously! It is worth keeping in mind there is a small CPU overhead between switching from V3 custom deepstream encoding to JSON (V4), so it is advised to monitor your CPU when possible! ``` type: ws-text options: port: 6021 host: 0.0.0.0 ``` Allow SUBSCRIBE and READ without CREATE actions, for clients that are in read only mode Adding declaration types (thank you @Vortex375!) Adding meta objects to logs and monitoring for easier tagging to monitoring solutions Don't buffer error messages in relation to connections, otherwise the client will get the close event first Ignore ping messages during the connecting and authenticating stages Notify monitoring plugin of all messages sent out individually Alpine docker image Override the http port and host correctly Exit immediately if HTTP server port is occupied When using debug, log exact error to why a plugin could not be loaded New protobuf protocol support (under the hood) Bulk actions instead of individual subscribes (under the hood) Official Plugin Support Monitoring Support Clustering Support (with small caveats) Listening Discovery Simplification V2 storage API V2 cache API Notify API Lazy data parsing Improved deepstream lifecycle Upgraded development tools New deepstream.io website All V3 SDKs no longer compatible due to protobuf binary protocol You can see the upgrade guide for backwards compatibility You can see the in depth side explanation of the changes a new standardised logging API with `debug`, `info`, `warn` and `error` methods the presence feature can now be used on a per user basis. The online status of individual users can be queried for as well as subscribed

Check out the tutorial on our website `perMessageDeflate` option can now be passed to uws, courtesy of various fixes and performance improvements to the subscription registry , courtesy of . allow updating and writing to Lists via the HTTP API courtesy of no data when sending HTTP requests is now considered undefined, rather than null . internal refactor to pull e2e client operations into framework and abstract from Cucumber steps. Added a `restart` option to deepstream service CLI deepstream will now write to a pid file at `/var/run/deepstream/deepstream.pid` while running as a service Authentication and permission plugins can now be configured via config and will be resolved as normal plugins. Either a path or name will need to be provided at the top level, and any options specified will also be passed in. Enabling clients to create, read, update and delete records, emit events, request RPCS and read presence using a JSON bulk request/response format via HTTP. The HTTP API is enabled by default on PORT 8080 and can be configured in the connectionEndpoints -> http section of deepstream's `config.yml` To disable the HTTP API set the above config to null The above HTTP API makes deepstream.io compatible with the deepstream PHP client The deepstream 3.0 release lays the groundwork for multiple combinable endpoints/protocols, e.g. GraphQL or Binary to be used together. It also introduces a new endpoint type enabling developers to write their own. Please note - at the moment it is not possible to run multiple subscription based endpoints (e.g. websocket) simultaneously. To address the scalability issues associated with the message connector interface's coarse topics deepstream will move to a build-in, high performance p2p/small world network based clustering approach, available as an enterprise plugin. The current message connector support is discontinued. Moved end-to-end tests into this repository from `deepstream.io-client-js`. Replaced `javascript-state-machine` dependency with custom state machine. Improved handling of invalid record names. Added new CLI command, including: deepstream daemon This command forks deepstream and monitors it for crashes, allowing it to restart automatically to avoid downtime deepstream service add This command allows you to create an init.d or systemd script automatically and add it to your system. ```bash sudo deepstream service --help Usage: service [options] [add|remove|start|stop|status] Add, remove, start or stop deepstream as a service to your operating system ``` Added brew cask support You can now install easily install deepstream on your mac using `brew cask install deepstream` driven by config files within `/user/local/etc/deepstream/conf` Fix issue where certain invalid paths would return 'Invalid Type' on the server. Fix issue in request/response where selecting a remote server as not done uniformly. Fix issue where using both `.0.` and `[0]` within a json path resulted in inserting into an array. However, when using other SDKs such as Java they would be treated as an Object key or array index. Fix issue where nested array access/manipulation didn't work via json paths. Due to the nature of this fix, it may result in compatability issues with applications that used json paths incorrectly ( using `.0.` intead of `[0]` ). Please ensure you change those before upgrading. Fix for issue where record deletions were not being propogated correctly within a cluster. Fixes config-loading issue present in the binary release of 2.3.5. Hardcode v3.0.0-rc1 dependency on javascript-state-machine, as v3.0.1 causes

startup to fail Hot path needs to store values in the correct format Binary config files have the correct latest structure Fix an issue where heavy concurrent writes on the same record fail Fixing a connection data regression where it wasn't formatted the same as pre 2.3.0 Correctly merging config options from `config.yml` file with the default options Adds "storageHotPathPatterns" config option. Adds support for `setData()`: upsert-style record updates without requiring that a client is subscribed to the record. This uses a new 'CU' (Create and Update) message. The `setData()` API is up to 10x faster than subscribing, setting, then discarding a record. Support for connection endpoint plugins. Significant performance improvements stemming from message batching. Moved uws into a connection endpoint plugin. Explicit state-machine that initializes and closes dependencies in a well-defined order. Adds support for custom authentication and permissioning plugins. Adds support for generic plugins. Added check to ensure subscriptions are not removed from distributed state registry prematurely. Unsolicited RPCs now get a `INVALIDRPCCORRELATION_ID` message RPC lifecycles have been improved and don't throw exceptions on response after a timeout by Correct options now being passed into the `RuleCache`, courtesy of Records now can be set with a version -1, which ignores version conflicts by Delete events are now propagated in the correct order by You can now request the HEAD of a record to retrieve just its version number by Providers for listeners are now by default selected randomly instead of in order of subscription Ensure record updates are not scalar values before trying to save them in cache by Long lived RPC requests now use dynamic lookups for providers rather than building the Set upfront by [ronag]{ronag} Huge optimization to subscription registry, where the time for registering a subscriber has been reduced from n^2 to O(n log n) Deleting grunt since everything is script based Due to uws releases being pulled from NPM, we're now using uws from a git repo Created a separate repo with binaries. Due to problems with build resulting from uws unpublishing, these two npm packages have been unpublished (noop) Unsolicited message in Listening when all clients unsubscribe [#531] Handle Non text based websocket frame [#538] Aligning binary config with node [#488] Event subscription data mishandled in Valve [#510] Logging after logger is destroyed [#527] Deepstream crash on empty users file [#512] Logging error object instead of name in connection error [#420] maxRuleIterations must be 1 or higher [#498] Ignore sender in subscriptionRegistry if messagebus [#473] Removing dead config options [#599] getAlternativeProvider in RPC Handler deals with more edge cases [#566] Update UWS build version to 0.12 Packages built against node 6.10 Fixing write error where only initial value is written to storage Valve cross referencing in both a create and read results in a ack timeout Record write acknowledgement. Records are now able to be set with an optional callback which will be called with any errors from storing the record in cache/storage Applying an ESLint rule set to the repo Stricter valve permissioning language checks by Update uWS version to Better handling/parsing of authentication messages Properly returning handshake data (headers) from SocketWrapper Fix case where CLIENT_DISCONNECTED is not sent from SocketWrapper Fixed issue where listen does not recover from server restart Handling presence events properly. Now when a user logs in, subscribed clients are only notified the first time the user logs in, and the last time they log out Fixed issue where connectionData was not available in auth requests Changelog of

mistakenly said that heartbeats were on port 80 instead of 6020 User presence has been added, enabling querying and subscription to who is online within a cluster Introduces the configuration option `broadcastTimeout` to `config.yml` to allow coalescing of broadcasts. This option can be used to improve broadcast message latency such as events, data-sync and presence For example, the performance of broadcasting 100 events to 1000 subscribers was improved by a factor of 20 Adds client heartbeats, along with configuration option`heartbeatInterval` in `config.yml`. If a connected client fails to send a heartbeat within this timeout, it will be considered to have disconnected Adds healthchecks deepstream now responds to http GET requests to path `/health-check` on port 6020 with code 200. This path can be configured with the `healthCheckPath` option in `config.yml` E2E tests refactored uWS is now compiled into the deepstream binary, eliminating reliability issues caused by dynamic linking Clients prior to v2.0.0 are no longer compatible Changed format of RPC request ACK messages to be more consistent with the rest of the specs We removed support for TCP and engine.io, providing huge performance gains by integrating tightly with native uWS Support for webRTC has been removed You can no longer set custom data transforms directly on deepstream Sending an invalid connection message is not caught by server Storage connector now logs errors with the correct namepspace RPC now uses distributed state and no longer depends on custom rpc discovery logic Deepstream now uses connection challenges by default rather than automatically replying with an ack Upgraded to uWS 0.9.0 Fix wrong validation of Valve Permissions when `data` is used as a property Outgoing connections now have throttle options that allow you to configure maximum package sizes to find your personal sweet spot between latency and speed ```yaml timeBetweenSendingQueuedPackages: 1 maxMessagesPerPacket: 1000 ``` Listening: Listeners have been drastically improved [https://github.com/deepstreamIO/deepstream.io/issues/211], and now: works correctly across a cluster can inform the user whenever the last subscription has been removed even if the listener itself is subscribed only allows a single listener to provide a record has a concept of provided, allowing records on the client side to be aware if the data is being actively updated by a backend component As part of this story, we now have multiple significant improvements in the server itself, such as: a `distributed state registry` which allows all clusters to keep their state in sync a `unique state provider` allowing cluster wide locks a `cluster-registry` that provides shares server presence and state across the cluster Because of these we can now start working on some really cool features such as advanced failover, user presence and others! Auth: File authentication sends server data to client on cleartext passwords Auth: HTTP authentication missing logger during when attempting to log any errors occured on http server CLI: installer for connectors sometimes fail to download (and extract) the archive Auth: File authentication doesn't contain `serverData` and `clientData` ```yaml userA: password: tsA+yfWGoEk9uEU/GX1JokkzteayLj6YFTwmraQrO7k=75KQ2Mzm serverData: role: admin clientData: nickname: Dave ``` Users can now provide a `unauthenticatedClientTimeout` config option that forces connections to close if they don't authenticate in time. This helps reduce load on server by terminating idle connections. `null`: Disable timeout `number`: Time in milliseconds before connection is terminated Fixed issue regarding last subscription to a deleted record not being cleared up Fix issue when try to pass options to the default logger ( as well) Fix issue deleting a record does not unsubscribe it and all other connections, not allowing resubscriptions to

#293 You can start deepstream via a command line interface. You find it in the bin directory. It provides these subcommands: `start` `stop` `status` `install` `info` `hash` append a `--help` to see the usage. You can now use a file based configuration instead of setting options via `ds.set(key, value)`. deepstream is shipped with a conf directory which contains three files: config.yml this is the main config file, you can specify most of the deepstream options in that file permissions.yml this file can be consumed by the PermissionHandler. It's not used by default, but you can enable it in the config.yml users.yml this file can be consumed by the AuthenticationHandler. It's not used by default, but you can enable it in the config.yml For all config types support these file types: .yml, .json and .js There are different options what you can pass: not passing any arguments ( consistent with 0.x ) passing `null` will result in loading the default configuration file in the directory conf/config.yml passing a string which is a path to a configuration file, supported formats: .yml, .json and .js passing an object which defines several options, all other options will be merged from deepstream's default values You can write your permission into a structured file. This file supports a special syntax, which allows you to do advanced permission checks. This syntax is called Valve. deepstream now uses , a native C++ websocket server deepstream will not longer stops your process via `process.exit()`. This happened before when a connector failed to initialize correctly instead it will throw an error now. Currently the API provides no event or callback to handle this error other than subscribing to the global `uncaughtException` event. ```javascript process.once('uncaughtException', err => { // err.code will equal to of these constant values: // C.EVENT.PLUGININITIALIZATIONTIMEOUT // or C.EVENT.PLUGININITIALIZATIONERROR }) ``` Keep in mind that deepstream will be in an unpredictable state and you should consider to create a new instance. In 0.x you can set a `permissionHandler` which needs to implement two functions: `isValidUser(connectionData, authData, callback)` `canPerformAction(username, message, callback)` In deepstream 1.0 the `isValidUser` and `onClientDisconnect` methods are no longer part of the `permissionHandler` and are instead within the new `authenticationHandler`. You can reuse the same 0.x permission handler except you will have to set it on both explicitly. ```javascript const permissionHandler = new CustomPermissionHandler() ds.set( 'permissionHandler', permissionHandler ) ds.set( 'authenticationHandler', permissionHandler ) ``` All connectors including, the `permissionHandler`, `authenticationHandler` and `logger` all need to implement the plugin interface which means exporting an object that: has a constructor has an `isReady` property which is true once the connector has been initialized. For example in the case a database connector this would only be `true` once the connection has been established. If the connector is synchronous you can set this to true within the constructor. extends the EventEmitter, and emits a `ready` event once initialized and `error` on error. The color flag can't be set in the root level of the configuration anymore. The default logger will print logs to the StdOut/StdErr in colors. You can use the which can be configured in the config.yml file with several options. deepstream clients now have a handshake protocol which allows them to be redirected to the most efficient node and expect an initial connection ack before logging in. As such In order to connect a client to deepstream server you need also to have a client with version 1.0 or higher. More details in the .