Repository: mindsdb/lightwood

Issue:
TS float cleaner can't deal with malformed input
As reported in the community slack:

If a time series dataset has invalid entries in the target column:

```
| date | target_value | 
_______________________
|  1   |   11         |
|  2   |   .          |
|  3   |   13         |
```

An exception is triggered `'float' object is not subscriptable`. We should probably deal with this at the cleaner phase, most likely simply dropping the entire row.

EDIT:

To reproduce:

- [Dataset](https://fred.stlouisfed.org/series/T5YIE)
- SQL command: ```select a.DATE, b.T5YIE, a.T5YIE from files.inflation as a join mindsdb.inflation_rate as b where a.DATE > LATEST;```

Taking a closer look, this is actually a type inference failure to detect the target column as a quantity, opting instead for the `categorical` dtype.


Discussion:

Resolving patch:
diff --git a/lightwood/analysis/nc/util.py b/lightwood/analysis/nc/util.py
--- a/lightwood/analysis/nc/util.py
+++ b/lightwood/analysis/nc/util.py
@@ -163,8 +163,12 @@ def get_categorical_conf(raw_confs: np.ndarray):
     """
     if len(raw_confs.shape) == 1:
         raw_confs = np.expand_dims(raw_confs, axis=0)
-    second_p = np.sort(raw_confs, axis=1)[:, -2]
-    confs = np.clip(np.subtract(1, second_p), 0.0001, 0.9999)
+    if raw_confs.shape[-1] == 1:
+        # single-class edge case (only happens if predictor sees just one known label at calibration)
+        confs = np.clip(raw_confs[:, 0], 0.0001, 0.9999)
+    else:
+        second_p = np.sort(raw_confs, axis=1)[:, -2]
+        confs = np.clip(np.subtract(1, second_p), 0.0001, 0.9999)
     return confs
 
 
diff --git a/lightwood/data/infer_types.py b/lightwood/data/infer_types.py
--- a/lightwood/data/infer_types.py
+++ b/lightwood/data/infer_types.py
@@ -204,7 +204,10 @@ def get_column_data_type(arg_tup):
     )
 
     actual_pct_invalid = 100 * (len(data) - max_known_dtype_count) / len(data)
-    if max_known_dtype is None or max_known_dtype == dtype.invalid or actual_pct_invalid > pct_invalid:
+    if actual_pct_invalid > pct_invalid:
+        raise Exception(f"The column '{col_name}' has a higher amount of invalid data ({actual_pct_invalid:.2f}%) than is tolerated ({pct_invalid:.2f}%), the predictor will not continue training. We recommend verifying that your input data looks okay. Otherwise, you can avoid this warning by setting a higher tolerance (`pct_invalid` should be higher than the actual amount of invalid data).")  # noqa
+
+    if max_known_dtype is None or max_known_dtype == dtype.invalid:
         curr_dtype = None
     else:
         curr_dtype = max_known_dtype


Test patch:
diff --git a/tests/integration/basic/test_weird_target_dist.py b/tests/integration/basic/test_weird_target_dist.py
--- a/tests/integration/basic/test_weird_target_dist.py
+++ b/tests/integration/basic/test_weird_target_dist.py
@@ -5,29 +5,31 @@
 
 
 class TestBasic(unittest.TestCase):
-    def test_0_unkown_cateogires_in_test(self):
-        """This test is meant to check if the expaliner can handle a  validation set\
-            with values that it has never seen as part of the training set.
-            In order to achieve this, we generate a dataset with a target that contains 600\
-            values, out of which 100 are distinct. It's basically certain at least 1 of these\
-            distinct value will be in train and at least one in validate (more likely 90/10 split)
-            So if training and predicting goes well, this validates that explainer works with\
-            new values in the validation set *and* with values in train that never occur in validation.
-            Note: This doesn't check how good the confidence is, it'll likely be very bad, we just care\
-            that it doesn't crash.
+    def test_0_unknown_categories_in_test(self):
         """
+        This test is meant to check if the explainer can handle a validation set
+        with values that it has never seen as part of the training set.
+        In order to achieve this, we generate a dataset with a target that contains 600
+        values, out of which 100 are distinct. It's basically certain at least 1 of these
+        distinct value will be in train and at least one in validate (more likely 90/10 split)
+        So if training and predicting goes well, this validates that explainer works with
+        new values in the validation set *and* with values in train that never occur in validation.
+        Note: This doesn't check how good the confidence is, it'll likely be very bad, we just care
+        that it doesn't crash.
+        """  # noqa
         from lightwood.api.high_level import predictor_from_problem
 
-        # The target will be cateogircal and there will be a bunch of values
+        # The target will be categorical and there will be a bunch of values
         # in all datasets (train/dev/validation) that were not present in the others
         df = pd.DataFrame({
-            'target': [1 for _ in range(500)] + [f'{i}cat' for i in range(100)],
+            'target': ['_cat' for _ in range(500)] + [f'{i}cat' for i in range(100)],
             'y': [i for i in range(600)]
         })
         target = 'target'
 
         predictor = predictor_from_problem(df, ProblemDefinition.from_dict(
-                                           {'target': target, 'time_aim': 60, 'unbias_target': True}))
+                                           {'target': target, 'time_aim': 60, 'unbias_target': True,
+                                            "pct_invalid": 20}))
         predictor.learn(df)
         assert predictor.model_analysis.dtypes['target'] == dtype.categorical
         predictor.predict(df)