Detailed Introduction: Redis Modules, from Graphs to Machine Learning (Part 1)
By: Ashish Rana
Tags: Data, Graph, Redis, Database, Data Engineering

I know its late( modules are already out for quite a time) but better late than never 😅. Also, this part contains explanation and an example about RedisGraph module only. Others, will be covered in next part in coming weekend.

Machine learning models, artificial neural networks are definitely the new in thing not only for research but for major enterprise industries also. Already enough data is available with various organizations operating in multiple domains especially the big ones. Now, the task is to find suitable mechanism to make the most out this resource available in abundant quantity. To carry out of the box POCs (Proof of Concepts) on all the possible options available in market to maximize the results as per your needs.

But, in times of such great advancements in the trendy fields of ML, AI and Deep Learning it would be quite unjustified to not discuss about advancements in databases which are founding stones for everything from research projects to industries in terms of storing data. Write choice of a database can save you millions of dollar on enterprise level applications over the years. Databases and their architectures are evolving to serve this modern industrial need of rising ML and AI tech. One of the best examples that I have seen is Redis(a key-value in-memory store with persistence) which becomes more powerful with its incredible modules which provides it large number of functionalities.

A time-saver tip: You can skip to any section mentioned below that interests you and start reading. They are written in modular manner 😅.

Let’s carry on our discussion in following manner stated below. So, that we can learn all about Redis from its basics to different functionalities of below mentioned modules.

1. Redis, its basics and use-cases

2. Introduction to its modules and Examples

3. Basics of RedisGraph

4. Basics of Neural-Redis

5. Basics of Redis-ML

If you are here for learning how to use Redis for ML & AI use-cases directly jump onto 3,4,5 section specified above. Only, section 3 present in this article 😉.

Redis, its basics and use-cases

Redis is an in-memory key-value NoSQL data structure store that can be used either as a database or for cache. Its specialty is it supports same data-structures that are used in programming languages like strings, hashes, lists, sets etc. and some other special ones too, few of them will be discussed later in this article. It uses simple commands like SET, DEL for setting and deleting keys. RPUSH, LPUSH, LLEN, LRANGE, LPOP, and RPOP for working with list data structure. And many more commands to interact with the database store. Structure to datastore is provided by these commands only which also encodes typing mechanisms to data. Just follow along documentation it’ll be an excellent guide for you for getting familiar with commands and their functionalities. Also, you can opt for Redis Introductory course RU101.

The main question is: When to use Redis ? And for What purpose ?

The finest way to answer that question will be to mention all the important architectural features of Redis and then present suitable use-case scenarios for it.

Fast and Addition of new Modules: It is very fast as written in C (plus bench-marked as fastest database) and supports addition of multiple modules that can be loaded as binary object file via foreign function interface mechanism. Modules are dynamic libraries that extend functionality of Redis by implementing new commands with features that acts as if they are implemented in Redis core itself. Here, is a sample command to load any module onto your redis-server.

# redis-server --loadmodule path/to/module/src/module-name.so redis-server --loadmodule path/to/module/src/redisgraph.so

CAP Theorem, is stated for distributed systems. That a distributed system can have can only have 2 of the 3: Consistency, Availability and Partition Tolerance. It is very important to decide the trade-offs while designing systems.

CAP theorem, Where exactly it falls ? If you are using only single Redis instance or a master/slave configuration then it’ll be quite wrong to evaluate to evaluate it with CAP theorem as these will not be called as distributed systems in true sense. Here, ‘P’ will be zero as it has no native partition tolerances, the way to scale it horizontally is by partitioning data across multiple Redis instances. It can be done via client side partitioning where clients determine on which instance to write or ‘Proxy Assisted’ partitioning where a proxy looking like a single Redis instance partitions the data.

We can discuss CAP theorem in context of Redis Sentinal which is a high availability solution with multiple master Redis instances and a list of replicating slaves. It monitors masters and promote slaves via a consensus when master node dies. But, Sentinal is designed keeping performance and strong data model values in mind. It is a general solution high availability and horizontal scalability.

True consistency is given up in favor of performance with replication being performed asynchronously. Similarly, strong availability(all writes be successful, even when the nodes cannot reach each other across the network.) is given up in favor of strong data model of Redis. Let’s say both masters should have same data and if a master joins later data must be merged onto it. But, Redis never actually merges data because it is difficult to correctly merge lists, sets, and some of the other advanced data structures Redis supports. Instead, it will always accept the most recently written state of the data.

Along with three parameters of CAP theorem Redis clusters are designed to keeping in mind to retain performance and strong data models. Salability being the ideal gem for it to focus on. It won’t be good match for heavy industrial grade application demanding very high consistency and availability(HTAP or OLTP use-cases) but for applications demanding high performance or analytics (OLAP use-cases) as prime criteria it can do wonders for them.

I have seen big organization skip away from Redis and its module when it comes to utilizing as primary datastore because many organizations demand consistency and high availability. But, it is used majorly as caching layer in the existing data pipeline for providing high throughput.

So, when to use Redis ? It can be used for cache sessions of users. It isn’t a mission critical consistency issue and with persistence it gains advantage over Memcached systems. Real time analytics like leader-boards, order by user-votes and time, deletion and filtering etc. are some other prvelant use-cases. Also, it can broadcast messages to every server that listens to a channel using Publisher/Subscriber technologies. Redis used Pub/Sub feature to trigger scripts, create a list of most common tasks on system, social network connections and many more boundless use-cases for it. Redis implements queues for jobs where master posts the jobs and worker picks up these jobs.

Key Takeaway: Do consider all Cloud Databases available before considering Redis as your primary data-store. Also, Redis increases performance for many other use-cases. Leverage its potential to the fullest !!

Now, exploring its potential with the modules it provides to extend its functionalities and carry on multiple new operations. Each module deserves its special article but we’ll explore modules related to AI and ML here. Also, keeping out RediSearch module of our discussion as it will be out of scope for now and requires a complete separate discussion on its features.

This page contains list of all the modules supported by Redis. Here, I’ ll be listing discussing about major ones that are often used with Redis and list some examples use-cases related to that module. In following sections I’ll be explaining in detail modules for working with graphs & its queryable relationships, ML models and neural networks. Let’s get started with introduction to these modules in this section.

RediSearch: Again, bench-marked with top results !!

RediSearch: It implements search engine like functionalities on top of Redis, it is a full-text and secondary index engine. Secondary indices are often implemented when you don’t want to alter your database structure but still want some feature specific queries to be executed with high efficiency. For example, querying for customers with shoe size 7 or all customer in a given city from a NoSQL storing information customer information indexed with their unique-id. It also have advanced features numeric filtering for text queries and exact phrase matching. Its popular use-cases are auto-complete and other fuzzy suggestions even with a typo on prefix. When index is big enough, it can be sharded across multiple machines. But, instead of normal approach RediSearch uses index partitioning method which queries all the shards concurrently and returns the result after merging. This module clearly requires more discussion, hopefully a dedicated future article and some coding examples for this module.

RediSQL: RediSQL is a module that embed a SQLite database, which makes it possible to have several smaller decentralized databases instead of a single giant one. It is created to add more structure on the given in-memory database instead of redis commands we can use SQL queries for our datastore. It utilizes velocity, portability, simplicity, and capability to work in memory of SQLite DB. It is developed by RedBeardLabs which for enterprise version provide 14 days free trial. But, I personally feel its 990€/year cost is quite expensive. Hence, you must analyze your use-case requirements carefully before selecting this one.

Well, the remaining popular modules add different data structure capabilities for Redis datastore for example: ReJSON provides JSON as native datatype and allows storing, updating and fetching JSON values from Redis keys. Also, RedisTimeSeries module which provides times series as linked list of memory chunks where each chunk has a predefined size of samples, each sample is a tuple of the time and the value.

Just a Quick Fact to end discussion about redis. Redis is incredibly fast. If performance is your prime criteria for your datastore. It has been bench-marked with 50 million operations per sec with 0.99 seconds of latency to be precise. It is a incredibly high throughput for a database to have. Also, it can be used for session cache which is one of its prime usage. But, there are also other parameters that are needed to be kept in mind while selecting a datastore like replication mechanism, conflict handling etc. Do analyze it before using it for your organization.