Repository: mindsdb/mindsdb

Issue:
Caching historical data for streams
I'll be using an example here and generalize whenever needed, let's say we have the following dataset we train a timeseries predictor on:

```
time,gb,target,aux
1     ,A, 7,        foo
2     ,A, 10,      foo
3     ,A,  12,     bar
4     ,A,  14,     bar
2     ,B,  5,       foo
4     ,B,  9,       foo
```

In this case `target` is what we are predicting, `gb` is the column we are grouping on and we are ordering by `time`. `aux` is an unrelated column that's not timeseries in nature and just used "normally".

We train a predictor with a window of `n`

Then let's say we have an input stream that looks something like this:

```
time, gb, target, aux
6,      A ,   33,     foo
7,      A,    54,     foo
```

### Caching

First, we will need to store, for each value of the column `gb` `n` recrods.

So, for example, if `n==1` we would save the last row in the data above, if `n==2` we would save both, whem new rows come in, we un-cache the older rows`.

### Infering

Second, when a new datapoint comes into the input stream we'll need to "infer" that the prediction we have to make is actually for the "next" datapoint. Which is to say that when: `7,      A,    54,     foo` comes in we need to infer that we need to actually make predictions for:

`8,   A,  <this is what we are predicting>, foo`

The challenge here is how do we infer that the next timestamp is `8`, one simple way to do this is to just subtract from the previous record, but that's an issue for the first observation (since we don't have a previous record to substract from, unless we cache part of the training data) *or* we could add a feature to native, to either:

a) Provide a `delta` argument for each group by (representing by how much we increment the order column[s])
b) Have an argument when doing timeseries prediction that tells it to predict for the "next" row and then do the inferences under the cover.

@paxcema let me know which of these features would be easy to implement in native, since you're now the resident timeseries expert.


Discussion:
@George3d6 Could you please clarify, what does `We train a predictor with a window of n` mean? Is it a part of timeseries predictor settings?
@ilia-tsyplenkov yes, there is a timeseries setting called `window` which is the number of rows needed to make a prediction for each group-by (so if `window=4` you need to store 4 rows in the past for each group by value)
@George3d6 @paxcema Could you please provide small example here. How to make a prediction for the next time stamp, in case with enough records in cache?
Yes, let's consider this data frame: 

```
time, gb, target, aux
_____________________
   1,  A,      7, foo
   2,  A,     10, foo
   3,  A,     12, bar
   4,  A,     14, bar
   5,  A,     18, bar
   1,  B,      1, bar
   2,  B,      5, foo
   3,  B,      7, bar
   4,  B,      9, foo
   5,  B,     11, foo
```

Internally, `mindsdb_native` shapes these values into a format appropriate for the time series models. Here, `window` and `order_by` parameters are required (in this case, `group_by` is also used). Assuming `order_by='time', group_by='gb', window=4`, the final data frame looks like this:

```
                    time, gb, target, __mdb_ts_previous_target, aux
___________________________________________________________________
   [None, None, None, 1],  A,      7, [None, None, None, None], foo
      [None, None, 1, 2],  A,     10,    [None, None, None, 7], foo
         [None, 1, 2, 3],  A,     12,      [None, None, 7, 10], bar
            [1, 2, 3, 4],  A,     14,        [None, 7, 10, 12], bar
            [2, 3, 4, 5],  A,     18,          [7, 10, 12, 14], bar
   [None, None, None, 1],  B,      1, [None, None, None, None], bar
      [None, None, 1, 2],  B,      5,    [None, None, None, 1], foo
         [None, 1, 2, 3],  B,      7,       [None, None, 1, 5], bar
            [1, 2, 3, 4],  B,      9,          [None, 1, 5, 7], foo
            [2, 3, 4, 5],  B,     11,             [1, 5, 7, 9], foo
```

The model is trained to predict the `target` column with this data format. Which means at inference time, we need at least `window` previous rows per each group if we want to maximize model performance, as the first `window` data points could be seen as a warm-up period where the model does not have complete information.

This is why we need a cache with the last `window` values per each group. Else, prediction would always be sub-par.

Hopefully this makes sense @ilia-tsyplenkov, let me know!
@George3d6 regarding MindsDB Native changes, I think the best way is probably a) with a twist. We introduce a parameter:

```
predictor.learn(advanced_args={
    ...
    'timeseries_settings': {
        ...
        'delta_infer': Int,
        ...
    }
})
```

And check: if at inference time the last row of the query has a missing target value (i.e. `None`), we are not in the streaming case and predict as normal. On the other hand, if the target value is present, we activate streaming logic and use `delta_infer` if passed by the user or an automatically determined value.

Let me know your thoughts on this approach.
@paxcema It is still not clear what I need to provide into `Predictor.predict` method in this case. I suspect that I need to provide a cache and probably a `time` value for predicted record. But I don't know how to do it.
Yes, we need to store last `window` rows for each group in this cache, including the `time` column. That way, any new stream data point will have a good prediction because the stream is used as historical context.

I am also unsure what's the best way of implementing this, i.e. is this something MindsDB or Native should handle, @George3d6 any thoughts? 
Ok, I think this may not be clear enough, let me try and ellucidate a few things by proposing an implementation:

When making a prediction with historical context, say for the row:

```
time, gb, target, aux
_____________________
   4,  A,     14, bar
```

Assuming you have the cached history:

```
time, gb, target, aux
_____________________
   3,  A,     12, foo
   2,  A,     10, bar
   1,  A,      7, foo
```

We need to go through the following steps:

1. *IF* the "target" value is present in the row, assume that the user is asking for a prediction for the *NEXT* row.

As a result of  this, add a column `make_predictions` == `False` to every row (in mindsdb)

```
time, gb, target, aux, make_predictions
_____________________
   4,  A,     14, bar, False
   3,  A,     12, foo, False
   2,  A,     10, bar, False
   1,  A,      7, foo, False
```

As a result of this, @paxcema , we need to impl a feature in native that "infers" the value of the next row when `make_predictions` is set to false for everything, in this case it should infer it as ` 5,  A,     ?, bar`.

2. If the "target" value is not in the row, assume the user is asking for a prediction for this row, and send:

```
time, gb, target, aux, make_predictions
_____________________
   4,  A,     None, bar, True
   3,  A,     12, foo, False
   2,  A,     10, bar, False
   1,  A,      7, foo, False
```

3. The dataframes being sent by mindsdb are composed of the latest row from the stream + the history based on previous rows in the stream. So e.g. row `2,  A,     10, bar, False` came from somewhere in the stream and was predicted for at some point in the past

This row should be stored in.... ? 

I'd say create a new collection in redis called `mindsdb_historical_data_collection` and store the history there, using a hash map containing lists, where the keys of the hash map  are keyd of the value in every group.

@paxcema @George3d6 I don't think that `adding make_predictions` feature is required:
 - there is no needs to store records with/without target value in cache. cache is only for records without **_target_** value.
@George3d6 another question: what we need to do if new record without target has come (`4,  A,     None, bar`) but there are not enough records in the cache for making prediction?

Resolving patch:
diff --git a/mindsdb/integrations/redis/redisdb.py b/mindsdb/integrations/redis/redisdb.py
--- a/mindsdb/integrations/redis/redisdb.py
+++ b/mindsdb/integrations/redis/redisdb.py
@@ -1,12 +1,8 @@
-import os
-import json
 from threading import Thread
 import walrus
-from mindsdb.utilities.config import STOP_THREADS_EVENT
-from mindsdb.utilities.log import log
-from mindsdb.integrations.base import Integration
+from mindsdb.integrations.base import StreamIntegration
 from mindsdb.streams.redis.redis_stream import RedisStream
-from mindsdb.interfaces.storage.db import session, Stream, Configuration
+from mindsdb.interfaces.storage.db import session, Stream
 
 
 class RedisConnectionChecker:
@@ -32,11 +28,11 @@ def check_connection(self):
             return False
 
 
-class Redis(Integration, RedisConnectionChecker):
+class Redis(StreamIntegration, RedisConnectionChecker):
     """Redis Integration which is more a Streams factory
     than classical Integration."""
     def __init__(self, config, name):
-        Integration.__init__(self, config, name)
+        StreamIntegration.__init__(self, config, name)
         intergration_info = self.config['integrations'][self.name]
         self.host = intergration_info.get('host')
         self.port = intergration_info.get('port', 6379)
@@ -46,27 +42,6 @@ def __init__(self, config, name):
         self.password = intergration_info.get('password', None)
         self.client = self._get_connection()
         self.control_stream = self.client.Stream(self.control_stream_name)
-        self.company_id = os.environ.get('MINDSDB_COMPANY_ID', None)
-        self.streams = {}
-        self.stop_event = STOP_THREADS_EVENT
-
-    def should_i_exist(self):
-        config_record = session.query(Configuration).filter_by(company_id=self.company_id).first()
-        if config_record is None:
-            return False
-        integrations = json.loads(config_record.data)["integrations"]
-        if self.name not in integrations:
-            return False
-        return True
-
-    def setup(self):
-        """Launches streams stored in db and
-        Launches a worker in separate thread which waits
-        data from control stream and creates a particular Streams."""
-
-        self.start_stored_streams()
-        # launch worker which reads control steam and spawn streams
-        self.start()
 
     def start(self):
         Thread(target=Redis.work, args=(self, )).start()
@@ -83,31 +58,22 @@ def start_stored_streams(self):
                           "stream_out": stream.stream_out,
                           "type": stream._type}
 
-                log.error(f"Integration {self.name} - launching from db : {params}")
+                self.log.error(f"Integration {self.name} - launching from db : {params}")
                 to_launch.start()
                 self.streams[stream.name] = to_launch.stop_event
 
-    def delete_stream(self, predictor):
-        """Deletes stream from database and stops it work by
-        setting up a special threading.Event flag."""
-        stream_name = f"{self.name}_{predictor}"
-        log.error(f"Integration {self.name}: deleting {stream_name}")
-        session.query(Stream).filter_by(company_id=self.company_id, integration=self.name, name=stream_name).delete()
-        session.commit()
-        if stream_name in self.streams:
-            self.streams[stream_name].set()
-            del self.streams[stream_name]
-
     def work(self):
         """Creates a Streams by receiving initial information from control stream."""
-        log.error(f"Integration {self.name}: start listening {self.control_stream_name} redis stream")
+        self.log.error(f"Integration {self.name}: start listening {self.control_stream_name} redis stream")
         while not self.stop_event.wait(0.5):
             try:
                 # break if no record about this integration has found in db
-                if not self.should_i_exist():
+                if not self.exist_in_db():
                     break
                 # First, lets check that there are no new records in db, created via HTTP API for e.g.
                 self.start_stored_streams()
+                # stop streams which have been deleted from db.
+                self.stop_deleted_streams()
                 # Checking new incoming records(requests) for new stream.
                 # Could't use blocking reading here, because this loop must
                 # also check new records in db (created via HTTP api for e.g)
@@ -121,51 +87,50 @@ def work(self):
                         self.control_stream.delete(r_id)
                         continue
 
+                    stream_params['name'] = f"{self.name}_{stream.predictor}"
                     stream = self.get_stream_from_kwargs(**stream_params)
-                    log.error(f"Integration {self.name}: creating stream: {stream_params}")
+                    self.log.error(f"Integration {self.name}: creating stream: {stream_params}")
                     stream.start()
                     # store created stream in database
                     self.store_stream(stream)
                     # need to delete record which steam is already created
                     self.control_stream.delete(r_id)
             except Exception as e:
-                log.error(f"Integration {self.name}: {e}")
+                self.log.error(f"Integration {self.name}: {e}")
+                raise e
 
         # received exit event
-        log.error(f"Integration {self.name}: exiting...")
+        self.log.error(f"Integration {self.name}: exiting...")
         self.client.close()
         self.stop_streams()
         session.close()
 
-    def stop_streams(self):
-        for stream in self.streams:
-            self.streams[stream].set()
-
     def store_stream(self, stream):
         """Stories a created stream."""
-        stream_name = f"{self.name}_{stream.predictor}"
-        stream_rec = Stream(name=stream_name, host=stream.host,
+        stream_rec = Stream(name=stream.name, host=stream.host,
                             port=stream.port, db=stream.db,
                             _type=stream._type, predictor=stream.predictor,
                             integration=self.name, company_id=self.company_id,
                             stream_in=stream.stream_in_name, stream_out=stream.stream_out_name)
         session.add(stream_rec)
         session.commit()
-        self.streams[stream_name] = stream.stop_event
+        self.streams[stream.name] = stream.stop_event
 
     def get_stream_from_db(self, db_record):
         kwargs = {"type": db_record._type,
+                  "name": db_record.name,
                   "predictor": db_record.predictor,
                   "input_stream": db_record.stream_in,
                   "output_stream": db_record.stream_out}
         return self.get_stream_from_kwargs(**kwargs)
 
     def get_stream_from_kwargs(self, **kwargs):
+        name = kwargs.get('name')
         stream_in = kwargs.get('input_stream')
         stream_out = kwargs.get('output_stream')
         predictor_name = kwargs.get('predictor')
         stream_type = kwargs.get('type', 'forecast')
-        return RedisStream(self.host, self.port, self.db,
+        return RedisStream(name, self.host, self.port, self.db,
                            stream_in, stream_out, predictor_name,
                            stream_type)
 
@@ -173,17 +138,8 @@ def _decode(self, b_dict):
         """convert binary key/value into strings"""
         decoded = {}
         if not isinstance(b_dict, dict):
-            log.error(f"Integration {self.name}: got unexpected data format from redis control stream {self.control_stream_name}: {b_dict}")
+            self.log.error(f"Integration {self.name}: got unexpected data format from redis control stream {self.control_stream_name}: {b_dict}")
             return {}
         for k in b_dict:
             decoded[k.decode('utf8')] = b_dict[k].decode('utf8')
         return decoded
-
-    def _query(self):
-        pass
-
-    def register_predictors(self, model_data_arr):
-        pass
-
-    def unregister_predictor(self, name):
-        pass
diff --git a/mindsdb/streams/kafka/kafka_stream.py b/mindsdb/streams/kafka/kafka_stream.py
--- a/mindsdb/streams/kafka/kafka_stream.py
+++ b/mindsdb/streams/kafka/kafka_stream.py
@@ -32,7 +32,98 @@ def __init__(self, host, port, topic_in, topic_out, predictor, _type):
 
         self.stop_event = Event()
         self.company_id = os.environ.get('MINDSDB_COMPANY_ID', None)
-        super().__init__(target=KafkaStream.make_prediction, args=(self,))
+        self.caches = {}
+        if self._type == 'timeseries':
+            super().__init__(target=KafkaStream.make_timeseries_predictions, args=(self,))
+        else:
+            super().__init__(target=KafkaStream.make_prediction, args=(self,))
+
+    def _get_target(self):
+        return "pnew_case"
+        # pass
+
+    def _get_window_size(self):
+        return 10
+        # pass
+
+    def _get_gb(self):
+        return "state"
+        # pass
+
+    def _get_dt(self):
+        return "time"
+        # pass
+
+    def predict_ts(self, cache_name):
+        when_list = [x for x  in self.caches[cache_name]]
+        for x in when_list:
+            if self.target not in x:
+                x['make_predictions'] = False
+            else:
+                x['make_predictions'] = True
+
+        result = self.native_interface.predict(self.predictor, self.format_flag, when_data=when_list)
+        log.error(f"TIMESERIES STREAM: got {result}")
+        for res in result:
+            in_json = json.dumps(res)
+            to_send = in_json.encode('utf-8')
+            log.error(f"sending {to_send}")
+            self.producer.send(self.stream_out_name, to_send)
+        self.caches[cache_name] = self.caches[cache_name][1:]
+
+    def make_prediction_from_cache(self, cache_name):
+        cache = self.caches[cache_name]
+        log.error("STREAM: in make_prediction_from_cache")
+        if len(cache) >= self.window:
+            log.error(f"STREAM: make_prediction_from_cache - len(cache) = {len(cache)}")
+            self.predict_ts(cache_name)
+
+    def to_cache(self, record):
+        gb_val = record[self.gb]
+        cache_name = f"cache.{gb_val}"
+        if cache_name not in self.caches:
+            cache = []
+            self.caches[cache_name] = cache
+
+        log.error(f"STREAM: cache {cache_name} has been created")
+        self.make_prediction_from_cache(cache_name)
+        self.handle_record(cache_name, record)
+        self.make_prediction_from_cache(cache_name)
+        log.error("STREAM in cache: current iteration has done.")
+
+    def handle_record(self, cache_name, record):
+        log.error(f"STREAM: handling cache {cache_name} and {record} record.")
+        cache = self.caches[cache_name]
+        cache.append(record)
+        cache = self.sort_cache(cache)
+        self.caches[cache_name] = cache
+
+    def sort_cache(self, cache):
+        return sorted(cache, key=lambda x: x[self.dt])
+
+    def make_timeseries_predictions(self):
+        log.error("STREAM: running 'make_timeseries_predictions'")
+        predict_record = session.query(DBPredictor).filter_by(company_id=self.company_id, name=self.predictor).first()
+        if predict_record is None:
+            log.error(f"Error creating stream: requested predictor {self.predictor} is not exist")
+            return
+        self.target = self._get_target()
+        self.window = self._get_window_size()
+        self.gb = self._get_gb()
+        self.dt = self._get_dt()
+
+        while not self.stop_event.wait(0.5):
+            try:
+                msg_str = next(self.consumer)
+                when_data = json.loads(msg_str.value)
+                self.to_cache(when_data)
+            except StopIteration:
+                pass
+
+        log.error("Stopping stream..")
+        self.producer.close()
+        self.consumer.close()
+        session.close()
 
     def make_prediction(self):
         predict_record = session.query(DBPredictor).filter_by(company_id=self.company_id, name=self.predictor).first()
@@ -55,3 +146,4 @@ def make_prediction(self):
         log.error("Stopping stream..")
         self.producer.close()
         self.consumer.close()
+        session.close()
diff --git a/mindsdb/streams/redis/redis_stream.py b/mindsdb/streams/redis/redis_stream.py
--- a/mindsdb/streams/redis/redis_stream.py
+++ b/mindsdb/streams/redis/redis_stream.py
@@ -2,6 +2,7 @@
 import json
 from threading import Thread, Event
 import walrus
+
 from mindsdb.utilities.log import log
 from mindsdb.interfaces.storage.db import session
 from mindsdb.interfaces.storage.db import Predictor as DBPredictor
@@ -9,7 +10,8 @@
 
 
 class RedisStream(Thread):
-    def __init__(self, host, port, database, stream_in, stream_out, predictor, _type):
+    def __init__(self, name, host, port, database, stream_in, stream_out, predictor, _type):
+        self.stream_name = name
         self.host = host
         self.port = port
         self.db = database
@@ -22,15 +24,118 @@ def __init__(self, host, port, database, stream_in, stream_out, predictor, _type
         self._type = _type
         self.native_interface = NativeInterface()
         self.format_flag = 'explain'
-
         self.stop_event = Event()
         self.company_id = os.environ.get('MINDSDB_COMPANY_ID', None)
-        super().__init__(target=RedisStream.make_prediction, args=(self,))
+        if self._type == 'timeseries':
+            super().__init__(target=RedisStream.make_timeseries_predictions, args=(self,))
+        else:
+            super().__init__(target=RedisStream.make_predictions, args=(self,))
 
     def _get_client(self):
         return walrus.Database(host=self.host, port=self.port, db=self.db)
 
-    def make_prediction(self):
+    def _get_target(self):
+        return "pnew_case"
+        # pass
+
+    def _get_window_size(self):
+        return 10
+        # pass
+
+    def _get_gb(self):
+        return "state"
+        # pass
+
+    def _get_dt(self):
+        return "time"
+        # pass
+
+    def predict(self, stream_in, stream_out, timeseries_mode=False):
+        predict_info = stream_in.read(block=0)
+        when_list = []
+        for record in predict_info:
+            record_id = record[0]
+            raw_when_data = record[1]
+            when_data = self.decode(raw_when_data)
+            if timeseries_mode:
+                # if self.target not in when_data:
+                #     when_data['make_predictions'] = False
+                # else:
+                #     when_data['make_predictions'] = True
+                when_list.append(when_data)
+            else:
+                result = self.native_interface.predict(self.predictor, self.format_flag, when_data=when_data)
+                log.error(f"STREAM: got {result}")
+                for res in result:
+                    in_json = json.dumps(res)
+                    stream_out.add({"prediction": in_json})
+                stream_in.delete(record_id)
+
+        if timeseries_mode:
+            result = self.native_interface.predict(self.predictor, self.format_flag, when_data=when_list)
+            log.error(f"TIMESERIES STREAM: got {result}")
+            for res in result:
+                in_json = json.dumps(res)
+                stream_out.add({"prediction": in_json})
+            stream_in.trim(len(stream_in) - 1, approximate=False)
+
+
+    def make_prediction_from_cache(self, cache):
+        log.error("STREAM: in make_prediction_from_cache")
+        if len(cache) >= self.window:
+            log.error(f"STREAM: make_prediction_from_cache - len(cache) = {len(cache)}")
+            self.predict(cache, self.stream_out, timeseries_mode=True)
+
+    def make_timeseries_predictions(self):
+        log.error("STREAM: running 'make_timeseries_predictions'")
+        predict_record = session.query(DBPredictor).filter_by(company_id=self.company_id, name=self.predictor).first()
+        if predict_record is None:
+            log.error(f"Error creating stream: requested predictor {self.predictor} is not exist")
+            return
+        self.target = self._get_target()
+        self.window = self._get_window_size()
+        self.gb = self._get_gb()
+        self.dt = self._get_dt()
+
+        while not self.stop_event.wait(0.5):
+            # block==0 is a blocking mode
+            predict_info = self.stream_in.read(block=0)
+            for record in predict_info:
+                record_id = record[0]
+                raw_when_data = record[1]
+                when_data = self.decode(raw_when_data)
+                log.error(f"STREAM: next record have read from {self.stream_in.key}: {when_data}")
+                self.to_cache(when_data)
+                self.stream_in.delete(record_id)
+        session.close()
+
+    def to_cache(self, record):
+        gb_val = record[self.gb]
+        cache = self.client.Stream(f"{self.stream_name}.cache.{gb_val}")
+        log.error(f"STREAM: cache {cache.key} has been created")
+        self.make_prediction_from_cache(cache)
+        self.handle_record(cache, record)
+        self.make_prediction_from_cache(cache)
+        log.error("STREAM in cache: current iteration has done.")
+
+    def handle_record(self, cache, record):
+        log.error(f"STREAM: handling cache {cache.key} and {record} record.")
+        records = cache.read()
+        # log.error(f"STREAM: current {cache.key} state: {records}")
+        records = [self.decode(x[1]) for x in records]
+        log.error(f"STREAM: read {records} from cache.")
+        records.append(record)
+        records = self.sort_cache(records)
+        log.error(f"STREAM: after updating and sorting - {records}.")
+        cache.trim(0, approximate=False)
+        for rec in records:
+            cache.add(rec)
+        log.error(f"STREAM: finish updating {cache.key}")
+
+    def sort_cache(self, cache):
+        return sorted(cache, key=lambda x: x[self.dt])
+
+    def make_predictions(self):
         predict_record = session.query(DBPredictor).filter_by(company_id=self.company_id, name=self.predictor).first()
         if predict_record is None:
             log.error(f"Error creating stream: requested predictor {self.predictor} is not exist")


Test patch: