Repository: mindsdb/mindsdb

Issue:
TS prediction through MySQL api returns additional rows
If use 'between' or some particular date in query, then in result will be additional rows out of range borders
```
mysql> SELECT * from MYSQL_LOCAL.test_data.covid AS ta JOIN mindsdb.covid_2_5 AS tb WHERE ta.state = "CA" AND ta.time BETWEEN "2020-10-19" AND "2020-10-20";
+-------+------------+---------------------------+----------------------------+----------------------------------+---------------------------------+----------+-----------+
| state | time       | adult_icu_bed_utilization | inpatient_beds_utilization | percent_of_inpatients_with_covid | inpatient_bed_covid_utilization | new_case | pnew_case |
+-------+------------+---------------------------+----------------------------+----------------------------------+---------------------------------+----------+-----------+
| CA    | 2020-10-18 | 0.678434                  | 0.710012                   | 0.0637691                        | 0.0454588                       | 2862     | 0.0       |
| CA    | 2020-10-17 | 0.674238                  | 0.701717                   | 0.0668514                        | 0.0471342                       | 2979     | 0.0       |
| CA    | 2020-10-16 | 0.668853                  | 0.685736                   | 0.0765255                        | 0.0530074                       | 3075     | 0.0       |
| CA    | 2020-10-15 | 0.673766                  | 0.690326                   | 0.0731741                        | 0.0505856                       | 3329     | 0.0       |
| CA    | 2020-10-14 | 0.6831                    | 0.715068                   | 0.0684178                        | 0.0490201                       | 2666     | 0.0       |
| CA    | 2020-10-20 | 0.684998                  | 0.718231                   | 0.0637367                        | 0.0459381                       | 3286     | 0.0       |
| CA    | 2020-10-19 | 0.68512                   | 0.70747                    | 0.063462                         | 0.0450346                       | 3474     | 0.0       |
| CA    | 2020-10-19 | 0.68512                   | 0.70747                    | 0.063462                         | 0.0450346                       | 3474     | 0.0       |
+-------+------------+---------------------------+----------------------------+----------------------------------+---------------------------------+----------+-----------+
8 rows in set (0.61 sec)
```


Discussion:
Additional rows are WINDOW_SIZE number of rows on which TS prediction based on.
for prediction with WINDOW SIZE = 5:
```
mysql> SELECT * from MYSQL_LOCAL.test_data.covid AS ta JOIN mindsdb.covid_2_5 AS tb WHERE ta.state = "CA" AND ta.time BETWEEN "2020-10-19" AND "2020-10-20";
+-------+------------+---------------------------+----------------------------+----------------------------------+---------------------------------+----------+-----------+
| state | time       | adult_icu_bed_utilization | inpatient_beds_utilization | percent_of_inpatients_with_covid | inpatient_bed_covid_utilization | new_case | pnew_case |
+-------+------------+---------------------------+----------------------------+----------------------------------+---------------------------------+----------+-----------+
| CA    | 2020-10-18 | 0.678434                  | 0.710012                   | 0.0637691                        | 0.0454588                       | 2862     | 0.0       |
| CA    | 2020-10-17 | 0.674238                  | 0.701717                   | 0.0668514                        | 0.0471342                       | 2979     | 0.0       |
| CA    | 2020-10-16 | 0.668853                  | 0.685736                   | 0.0765255                        | 0.0530074                       | 3075     | 0.0       |
| CA    | 2020-10-15 | 0.673766                  | 0.690326                   | 0.0731741                        | 0.0505856                       | 3329     | 0.0       |
| CA    | 2020-10-14 | 0.6831                    | 0.715068                   | 0.0684178                        | 0.0490201                       | 2666     | 0.0       |
| CA    | 2020-10-20 | 0.684998                  | 0.718231                   | 0.0637367                        | 0.0459381                       | 3286     | 0.0       |
| CA    | 2020-10-19 | 0.68512                   | 0.70747                    | 0.063462                         | 0.0450346                       | 3474     | 0.0       |
| CA    | 2020-10-19 | 0.68512                   | 0.70747                    | 0.063462                         | 0.0450346                       | 3474     | 0.0       |
+-------+------------+---------------------------+----------------------------+----------------------------------+---------------------------------+----------+-----------+
8 rows in set (0.61 sec)
```

for WINDOWS SIZE = 10

```
mysql> SELECT * from MYSQL_LOCAL.test_data.covid AS ta JOIN mindsdb.covid_2 AS tb WHERE ta.state = "CA" AND ta.time BETWEEN "2020-10-19" AND "2020-10-20";
+-------+------------+---------------------------+----------------------------+----------------------------------+---------------------------------+----------+-----------+
| state | time       | adult_icu_bed_utilization | inpatient_beds_utilization | percent_of_inpatients_with_covid | inpatient_bed_covid_utilization | new_case | pnew_case |
+-------+------------+---------------------------+----------------------------+----------------------------------+---------------------------------+----------+-----------+
| CA    | 2020-10-18 | 0.678434                  | 0.710012                   | 0.0637691                        | 0.0454588                       | 2862     | 0.0       |
| CA    | 2020-10-17 | 0.674238                  | 0.701717                   | 0.0668514                        | 0.0471342                       | 2979     | 0.0       |
| CA    | 2020-10-16 | 0.668853                  | 0.685736                   | 0.0765255                        | 0.0530074                       | 3075     | 0.0       |
| CA    | 2020-10-15 | 0.673766                  | 0.690326                   | 0.0731741                        | 0.0505856                       | 3329     | 0.0       |
| CA    | 2020-10-14 | 0.6831                    | 0.715068                   | 0.0684178                        | 0.0490201                       | 2666     | 0.0       |
| CA    | 2020-10-13 | 0.691514                  | 0.72308                    | 0.0684464                        | 0.0495877                       | 2378     | 0.0       |
| CA    | 2020-10-12 | 0.680877                  | 0.715825                   | 0.0692987                        | 0.0497533                       | 3449     | 0.0       |
| CA    | 2020-10-11 | 0.674748                  | 0.713638                   | 0.0698877                        | 0.0500497                       | 3803     | 0.0       |
| CA    | 2020-10-10 | 0.676162                  | 0.705967                   | 0.0725691                        | 0.0514656                       | 4170     | 0.0       |
| CA    | 2020-10-09 | 0.659624                  | 0.682471                   | 0.0749988                        | 0.0513919                       | 3806     | 0.0       |
| CA    | 2020-10-20 | 0.684998                  | 0.718231                   | 0.0637367                        | 0.0459381                       | 3286     | 0.0       |
| CA    | 2020-10-19 | 0.68512                   | 0.70747                    | 0.063462                         | 0.0450346                       | 3474     | 0.0       |
| CA    | 2020-10-19 | 0.68512                   | 0.70747                    | 0.063462                         | 0.0450346                       | 3474     | 0.0       |
+-------+------------+---------------------------+----------------------------+----------------------------------+---------------------------------+----------+-----------+
13 rows in set (0.31 sec)
```

Resolving patch:
diff --git a/mindsdb/api/mysql/mysql_proxy/classes/sql_query.py b/mindsdb/api/mysql/mysql_proxy/classes/sql_query.py
--- a/mindsdb/api/mysql/mysql_proxy/classes/sql_query.py
+++ b/mindsdb/api/mysql/mysql_proxy/classes/sql_query.py
@@ -141,6 +141,14 @@ def join_query_data(target, source):
             target['columns'][table_name] = list(set(target['columns'][table_name]))
 
 
+def is_empty_prediction_row(predictor_value):
+    "Define empty rows in predictor after JOIN"
+    for key in predictor_value:
+        if predictor_value[key] is not None and pd.notna(predictor_value[key]):
+            return False
+    return True
+
+
 class SQLQuery():
     def __init__(self, sql, session):
         self.session = session
@@ -343,6 +351,7 @@ def _parse_query(self, sql):
                             predictor_metadata[model_name] = {
                                 'timeseries': True,
                                 'window': window,
+                                'nr_predictions': ts_settings.get('nr_predictions'),
                                 'order_by_column': order_by,
                                 'group_by_column': group_by
                             }
@@ -488,6 +497,43 @@ def _parse_query(self, sql):
                     integration_type=self.session.integration_type
                 )
 
+                if is_timeseries:
+                    if 'LATEST' not in self.raw:
+                        # remove additional records from predictor results:
+                        # first 'window_size' and last 'nr_prediction' records
+                        # otherwise there are many unxpected rows in prediciton result:
+                        # ----------------------------------------------------------------------------------------
+                        # mysql> SELECT tb.time, tb.state, tb.pnew_case, tb.new_case from
+                        # MYSQL_LOCAL.test_data.covid AS
+                        # ta JOIN mindsdb.covid_hor3 AS tb
+                        # WHERE ta.state = "CA" AND ta.time BETWEEN "2020-10-19" AND "2020-10-20";
+                        # ----------------------------------------------------------------------------------------
+                        # +------------+-------+-----------+----------+
+                        # | time       | state | pnew_case | new_case |
+                        # +------------+-------+-----------+----------+
+                        # | 2020-10-09 | CA    | 0         | 2862     |
+                        # | 2020-10-10 | CA    | 0         | 2979     |
+                        # | 2020-10-11 | CA    | 0         | 3075     |
+                        # | 2020-10-12 | CA    | 0         | 3329     |
+                        # | 2020-10-13 | CA    | 0         | 2666     |
+                        # | 2020-10-14 | CA    | 0         | 2378     |
+                        # | 2020-10-15 | CA    | 0         | 3449     |
+                        # | 2020-10-16 | CA    | 0         | 3803     |
+                        # | 2020-10-17 | CA    | 0         | 4170     |
+                        # | 2020-10-18 | CA    | 0         | 3806     |
+                        # | 2020-10-19 | CA    | 0         | 3286     |
+                        # | 2020-10-20 | CA    | 0         | 3474     |
+                        # | 2020-10-21 | CA    | 0         | 3474     |
+                        # | 2020-10-22 | CA    | 0         | 3474     |
+                        # +------------+-------+-----------+----------+
+                        # 14 rows in set (2.52 sec)
+
+                        window_size = predictor_metadata[predictor]['window']
+                        nr_predictions = predictor_metadata[predictor]['nr_predictions']
+                        if len(data) > (window_size + nr_predictions):
+                            data = data[window_size:]
+                            if len(data) > nr_predictions:
+                                data = data[:-nr_predictions+1]
                 data = [{(key, key): value for key, value in row.items()} for row in data]
 
                 table_name = get_preditor_alias(step, self.database)
@@ -508,9 +554,11 @@ def _parse_query(self, sql):
                 right_data = steps_data[step.right.step_num]
 
                 # FIXME https://github.com/mindsdb/mindsdb_sql/issues/136
+                is_timeseries = False
                 if True in [type(step) == ApplyTimeseriesPredictorStep for step in plan.steps]:
                     right_data = steps_data[step.left.step_num]
                     left_data = steps_data[step.right.step_num]
+                    is_timeseries = True
 
                 if step.query.condition is not None:
                     raise Exception('At this moment supported only JOIN without condition')
@@ -589,6 +637,34 @@ def _parse_query(self, sql):
                         else:
                             new_row[right_key][right_columns_map[key]] = value
                     data['values'].append(new_row)
+
+                # remove all records with empty data from predictor from join result
+                # otherwise there are emtpy records in the final result:
+                # +------------+------------+-------+-----------+----------+
+                # | time       | time       | state | pnew_case | new_case |
+                # +------------+------------+-------+-----------+----------+
+                # | 2020-10-21 | 2020-10-24 | CA    | 0.0       | 5945.0   |
+                # | 2020-10-22 | 2020-10-23 | CA    | 0.0       | 6141.0   |
+                # | 2020-10-23 | 2020-10-22 | CA    | 0.0       | 2940.0   |
+                # | 2020-10-24 | 2020-10-21 | CA    | 0.0       | 3707.0   |
+                # | NULL       | 2020-10-20 | NULL  | nan       | nan      |
+                # | NULL       | 2020-10-19 | NULL  | nan       | nan      |
+                # | NULL       | 2020-10-18 | NULL  | nan       | nan      |
+                # | NULL       | 2020-10-17 | NULL  | nan       | nan      |
+                # | NULL       | 2020-10-16 | NULL  | nan       | nan      |
+                # +------------+------------+-------+-----------+----------+
+                # 9 rows in set (2.07 sec)
+
+                if is_timeseries:
+                    data_values = []
+                    for row in data['values']:
+                        for key in row:
+                            if 'mindsdb' in key:
+                                if not is_empty_prediction_row(row[key]):
+                                    data_values.append(row)
+                                    break
+                    data['values'] = data_values
+
             elif type(step) == FilterStep:
                 raise Exception('FilterStep is not implemented')
             # elif type(step) == ApplyTimeseriesPredictorStep:


Test patch: