internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: mindsdb/lightwood

Issue:
Improve runtime of `LightGBMArray` for long-horizon forecasting
Two main approaches:

- Disable optuna hyperparam search past some threshold.
- Opt for a recursive strategy instead of direct (i.e. same regressor trained for all timesteps v/s one for each step).


Discussion:

Resolving patch:
diff --git a/lightwood/mixer/lightgbm_array.py b/lightwood/mixer/lightgbm_array.py
--- a/lightwood/mixer/lightgbm_array.py
+++ b/lightwood/mixer/lightgbm_array.py
@@ -49,7 +49,7 @@ def __init__(
                                 dtype_dict,
                                 input_cols,
                                 False,  # fit_on_dev,
-                                True,  # use_optuna
+                                True if tss.horizon < 10 else False,  # use_optuna
                                 target_encoder)
                        for _, target_col in zip(range(self.horizon), [target] + self.offset_pred_cols)]
         self.ts_analysis = ts_analysis


Test patch:
Repository: mindsdb/lightwood

Issue:
Store original amount of columns when generating JsonAI instance
This will be useful for reference because internally columns can be dropped or added.


Discussion:

Resolving patch:
diff --git a/lightwood/api/types.py b/lightwood/api/types.py
--- a/lightwood/api/types.py
+++ b/lightwood/api/types.py
@@ -62,6 +62,7 @@ class StatisticalAnalysis:
              in the information.
 
     :param nr_rows: Number of rows (samples) in the dataset
+    :param nr_columns: Number of columns (features) in the dataset
     :param df_target_stddev: The standard deviation of the target of the dataset
     :param train_observed_classes:
     :param target_class_distribution:
@@ -77,6 +78,7 @@ class StatisticalAnalysis:
     """ # noqa
 
     nr_rows: int
+    nr_columns: int
     df_target_stddev: Optional[float]
     train_observed_classes: object  # Union[None, List[str]]
     target_class_distribution: object  # Dict[str, float]
diff --git a/lightwood/data/statistical_analysis.py b/lightwood/data/statistical_analysis.py
--- a/lightwood/data/statistical_analysis.py
+++ b/lightwood/data/statistical_analysis.py
@@ -99,6 +99,7 @@ def statistical_analysis(data: pd.DataFrame,
         except Exception:
             order_format = None
 
+    nr_columns = len(data.columns)
     df = cleaner(data, dtypes, problem_definition.pct_invalid,
                  identifiers, problem_definition.target, 'train', tss,
                  problem_definition.anomaly_detection)
@@ -204,6 +205,7 @@ def statistical_analysis(data: pd.DataFrame,
     log.info('Finished statistical analysis')
     return StatisticalAnalysis(
         nr_rows=nr_rows,
+        nr_columns=nr_columns,
         df_target_stddev=df_std,
         train_observed_classes=train_observed_classes,
         target_class_distribution=target_class_distribution,


Test patch:
Repository: mindsdb/lightwood

Issue:
🔧 Add library version to predictors
## What?

This issue is pretty simple: we should document what was the exact Lightwood version that was used to create every predictor object.

## How?

You can get the `lightwood.__version__` and add it to the predictor generation code in the `code_from_json_ai()` method inside `lightwood/api/json_ai.py`.


Discussion:
Hi @paxcema, can you have a look at my pr? I am not sure that my solution is correct and I am not able to test it.
Can i work on this if no response from @adripo ?
Yes, but let's give @adripo a full week before calling it (so, if he doesn't update in 4 days from now, I will reassign it to you).
@paxcema sorry I was was busy for the past few days. I will work this issue today.

Resolving patch:
diff --git a/lightwood/api/json_ai.py b/lightwood/api/json_ai.py
--- a/lightwood/api/json_ai.py
+++ b/lightwood/api/json_ai.py
@@ -11,6 +11,7 @@
 )
 import inspect
 from lightwood.helpers.log import log
+from lightwood.__about__ import __version__ as lightwood_version
 
 
 # For custom modules, we create a module loader with necessary imports below
@@ -1186,6 +1187,7 @@ def __init__(self):
         self.accuracy_functions = {json_ai.accuracy_functions}
         self.identifiers = {json_ai.identifiers}
         self.dtype_dict = {inline_dict(dtype_dict)}
+        self.lightwood_version = '{lightwood_version}'
 
         # Any feature-column dependencies
         self.dependencies = {inline_dict(json_ai.dependency_dict)}


Test patch:
diff --git a/tests/integration/basic/test_regression.py b/tests/integration/basic/test_regression.py
--- a/tests/integration/basic/test_regression.py
+++ b/tests/integration/basic/test_regression.py
@@ -4,6 +4,7 @@
 from sklearn.metrics import r2_score
 from lightwood.api.types import ProblemDefinition
 from lightwood.api.high_level import json_ai_from_problem, predictor_from_json_ai
+from lightwood import __version__ as lightwood_version
 
 
 class TestBasic(unittest.TestCase):
@@ -39,6 +40,8 @@ def test_0_predict_file_flow(self):
 
         assert predictor.model_analysis.dtypes[target] == dtype.quantity
 
+        assert predictor.lightwood_version == str(lightwood_version)
+
         predictions = predictor.predict(df)
 
         # sanity checks
Repository: mindsdb/lightwood

Issue:
Bug in anomaly detection with infer mode
From MindsDB. If a user inputs this query:

```sql
SELECT m.saledate as date,
m.ma as forecast,
m.ma_explain
FROM mindsdb.house_sales_model as m 
JOIN example_db.demo_data.house_sales as t
WHERE t.saledate > LATEST AND t.type = 'house'
AND t.bedrooms=2
LIMIT 4;
```

They get back an anomaly detection value for the first prediction, even though it shouldn't be possible because by definition there are no available actuals. This implies a bug, potentially in the `ICP.explain()` method.


Discussion:

Resolving patch:
diff --git a/lightwood/analysis/nc/calibrate.py b/lightwood/analysis/nc/calibrate.py
--- a/lightwood/analysis/nc/calibrate.py
+++ b/lightwood/analysis/nc/calibrate.py
@@ -446,7 +446,10 @@ def explain(self, row_insights: pd.DataFrame, global_insights: Dict[str, object]
                         anomalies = get_anomalies(row_insights,
                                                   ns.data[ns.target_name],
                                                   cooldown=ns.pred_args.anomaly_cooldown)
-                        row_insights['anomaly'] = anomalies
+                        if '__mdb_ts_inferred' in ns.data.columns:
+                            ignore_idxs = np.where(ns.data['__mdb_ts_inferred'].values)
+                            anomalies[ignore_idxs] = None
+                        row_insights['anomaly'] = list(anomalies)
 
             if ns.tss.is_timeseries and ns.tss.horizon > 1:
                 if is_numerical and ns.pred_args.simple_ts_bounds:
diff --git a/lightwood/analysis/nc/util.py b/lightwood/analysis/nc/util.py
--- a/lightwood/analysis/nc/util.py
+++ b/lightwood/analysis/nc/util.py
@@ -204,7 +204,7 @@ def get_categorical_conf(raw_confs: np.ndarray):
     return confs
 
 
-def get_anomalies(insights: pd.DataFrame, observed_series: Union[pd.Series, list], cooldown: int = 1):
+def get_anomalies(insights: pd.DataFrame, observed_series: Union[pd.Series, list], cooldown: int = 1) -> np.ndarray:
     """
     Simple procedure for unsupervised anomaly detection in time series forecasting. 
     Uses ICP analysis block output so that any true value falling outside of the lower and upper bounds is tagged as anomalous.
@@ -213,7 +213,7 @@ def get_anomalies(insights: pd.DataFrame, observed_series: Union[pd.Series, list
     :param observed_series: true values from the predicted time series. If empty, no anomalies are flagged.
     :param cooldown: minimum amount of observations (assuming regular sampling frequency) that need to pass between two consecutive anomalies.
     
-    :return: list of boolean flags, indicating anomalous behavior for each predicted value.
+    :return: np.ndarray of boolean flags, indicating anomalous behavior for each predicted value.
     """  # noqa
     anomalies = []
     counter = 0
@@ -247,4 +247,4 @@ def get_anomalies(insights: pd.DataFrame, observed_series: Union[pd.Series, list
             anomalies.append(None)
             counter += 1
 
-    return anomalies
+    return np.array(anomalies)


Test patch:
Repository: mindsdb/lightwood

Issue:
Lightwood pip packages creates tests module
Installing lightwood creates 'tests' module in python site-packages

Steps to reproduce:
- `pip install lightwood`
- in python
 - `import tests`
 - `print(tests.__file__) `
It will show that 'tests' is in site-packages


Discussion:

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -53,7 +53,7 @@ def remove_requirements(requirements, name, replace=''):
     description=about['__description__'],
     long_description=long_description,
     long_description_content_type="text/markdown",
-    packages=setuptools.find_packages(),
+    packages=setuptools.find_packages(exclude=["tests", "tests.*"]),
     package_data={'project': ['requirements.txt']},
     install_requires=requirements,
     extras_require=extra_requirements,


Test patch:
diff --git a/tests/requirements.txt b/tests/requirements.txt
deleted file mode 100644
Repository: mindsdb/lightwood

Issue:
Expose global insights generated in the analysis phase
Lightwood can generate "global" insights that are not in a row-by-row fashion (e.g. boolean flags of some sort), but we're not exposing this in the predictor interface. ~We should fix it, somehow.~

Current thoughts:

- store global insights at the predictor level, i.e. `predictor.insights`
- simple method to get these and pretty print them
- from mindsdb: call `DESCRIBE model.insights` to get the object


Discussion:

Resolving patch:
diff --git a/lightwood/api/json_ai.py b/lightwood/api/json_ai.py
--- a/lightwood/api/json_ai.py
+++ b/lightwood/api/json_ai.py
@@ -1208,6 +1208,7 @@ def code_from_json_ai(json_ai: JsonAI) -> str:
 else:
     log.info(f'[Predict phase 4/{{n_phases}}] - Analyzing output')
     insights, global_insights = {call(json_ai.explainer)}
+    self.global_insights = {{**self.global_insights, **global_insights}}
     return insights
 """
 
@@ -1243,6 +1244,7 @@ def __init__(self):
         self.statistical_analysis = None
         self.ts_analysis = None
         self.runtime_log = dict()
+        self.global_insights = dict()
 
     @timed
     def analyze_data(self, data: pd.DataFrame) -> None:


Test patch:
Repository: mindsdb/lightwood

Issue:
More user-friendly message when not able to download NLTK data
### Is there an existing issue for this?

- [X] I have searched the existing issues

### Is your feature request related to a problem? Please describe.

Ref: https://github.com/mindsdb/mindsdb/issues/3975

### Describe the solution you'd like.

_No response_

### Describe an alternate solution.

_No response_

### Anything else? (Additional Context)

_No response_


Discussion:

Resolving patch:
diff --git a/lightwood/api/json_ai.py b/lightwood/api/json_ai.py
--- a/lightwood/api/json_ai.py
+++ b/lightwood/api/json_ai.py
@@ -1024,6 +1024,8 @@ def code_from_json_ai(json_ai: JsonAI) -> str:
             raise e
 
 # Update mixers to trained versions
+if not trained_mixers:
+    raise Exception('No mixers could be trained! Please verify your problem definition or JsonAI model representation.')
 self.mixers = trained_mixers
 
 # --------------- #
diff --git a/lightwood/helpers/text.py b/lightwood/helpers/text.py
--- a/lightwood/helpers/text.py
+++ b/lightwood/helpers/text.py
@@ -8,16 +8,29 @@
  * permission of MindsDB Inc
  *******************************************************
 """
+import os
+
 import nltk
+from contextlib import redirect_stdout
+
+from lightwood.helpers.log import log
 
 
 try:
-    nltk.data.find('tokenizers/punkt')
+    with redirect_stdout(open(os.devnull, "w")):
+        nltk.data.find('tokenizers/punkt')
 except LookupError:
-    nltk.download('punkt')
+    try:
+        nltk.download('punkt', quiet=True)
+    except Exception:
+        log.error("NLTK was unable to download the 'punkt' package. Please check your connection and try again!")
 
 try:
-    from nltk.corpus import stopwords
-    stopwords.words('english')
+    with redirect_stdout(open(os.devnull, "w")):
+        from nltk.corpus import stopwords
+        stopwords.words('english')
 except LookupError:
-    nltk.download('stopwords', quiet=True)
+    try:
+        nltk.download('stopwords', quiet=True)
+    except Exception:
+        log.error("NLTK was unable to download the 'stopwords' package. Please check your connection and try again!")


Test patch:
Repository: mindsdb/lightwood

Issue:
Handle no input features gracefully.
If a dataset happens to get all its features removed, training should not be possible and a helpful exception should be displayed to the user.


Discussion:

Resolving patch:
diff --git a/lightwood/api/json_ai.py b/lightwood/api/json_ai.py
--- a/lightwood/api/json_ai.py
+++ b/lightwood/api/json_ai.py
@@ -835,6 +835,8 @@ def code_from_json_ai(json_ai: JsonAI) -> str:
 
     input_cols = [x.replace("'", "\\'").replace('"', '\\"') for x in json_ai.encoders
                   if x != json_ai.problem_definition.target]
+    if len(input_cols) < 1:
+        raise Exception('There are no valid input features. Please check your data before trying again.')
     input_cols = ",".join([f"""'{name}'""" for name in input_cols])
 
     # ----------------- #


Test patch:
Repository: pretix/pretix

Issue:
Substring search when matching voucher tags
I might use a tagging structure such as "DE-Berlin", "DE-Stuttgart" for vouchers, and it'd be neat to match all "DE-" tagged vouchers at once.



Discussion:
Hey,
I'm willing to contribute to pretix, and @raphaelm pointed me to this. I'm currently kind of getting the hang of pretix and just thought I'd let you know I'm working on this.


Resolving patch:
diff --git a/src/pretix/control/views/vouchers.py b/src/pretix/control/views/vouchers.py
--- a/src/pretix/control/views/vouchers.py
+++ b/src/pretix/control/views/vouchers.py
@@ -37,7 +37,7 @@ def get_queryset(self):
             qs = qs.filter(Q(code__icontains=s) | Q(tag__icontains=s) | Q(comment__icontains=s))
         if self.request.GET.get("tag", "") != "":
             s = self.request.GET.get("tag", "")
-            qs = qs.filter(tag=s)
+            qs = qs.filter(tag__icontains=s)
         if self.request.GET.get("status", "") != "":
             s = self.request.GET.get("status", "")
             if s == 'v':


Test patch:
Repository: pretix/pretix

Issue:
Voucher-counting returning True/False
The tag-counting of vouchers seems two have two issues at the moment:
- No voucher has been used yet:
  `False/<Number of Vouchers>` instead of `0/<Number of Vouchers>`
- Some other bug, if some vouchers have already been redeemed (cannot put my finger on, when this happens/why this happens):
  `True/<Number of Vouchers>` instead of `<Number of used Vouchers>/<Number of Vouchers>`

Second item can be observed at the moment for example for budapest.



Discussion:
`True` seems to be `1`. As I can't reproduce this locally, this _might_ be MySQL-specific.


Resolving patch:
diff --git a/src/pretix/control/views/vouchers.py b/src/pretix/control/views/vouchers.py
--- a/src/pretix/control/views/vouchers.py
+++ b/src/pretix/control/views/vouchers.py
@@ -5,7 +5,7 @@
 from django.contrib import messages
 from django.core.urlresolvers import resolve, reverse
 from django.db import transaction
-from django.db.models import Count, Q, Sum
+from django.db.models import Case, Count, IntegerField, Q, Sum, When
 from django.http import (
     Http404, HttpResponse, HttpResponseBadRequest, HttpResponseRedirect,
     JsonResponse,
@@ -97,7 +97,14 @@ def get_context_data(self, **kwargs):
 
         tags = self.request.event.vouchers.order_by('tag').filter(tag__isnull=False).values('tag').annotate(
             total=Count('id'),
-            redeemed=Sum('redeemed')
+            # This is a fix for this MySQL issue: https://code.djangoproject.com/ticket/24662
+            redeemed=Sum(
+                Case(
+                    When(redeemed=True, then=1),
+                    When(redeemed=False, then=0),
+                    output_field=IntegerField()
+                )
+            )
         )
         for t in tags:
             t['percentage'] = int((t['redeemed'] / t['total']) * 100)


Test patch:
Repository: pretix/pretix

Issue:
Check if refunds properly set the user on their log entry
especially with stripe



Discussion:

Resolving patch:
diff --git a/src/pretix/plugins/paypal/payment.py b/src/pretix/plugins/paypal/payment.py
--- a/src/pretix/plugins/paypal/payment.py
+++ b/src/pretix/plugins/paypal/payment.py
@@ -221,7 +221,7 @@ def order_control_refund_perform(self, request, order) -> "bool|str":
             payment_info = None
 
         if not payment_info:
-            mark_order_refunded(order)
+            mark_order_refunded(order, user=request.user)
             messages.warning(request, _('We were unable to transfer the money back automatically. '
                                         'Please get in touch with the customer and transfer it back manually.'))
             return
@@ -239,7 +239,7 @@ def order_control_refund_perform(self, request, order) -> "bool|str":
                                         'Please get in touch with the customer and transfer it back manually.'))
         else:
             sale = paypalrestsdk.Payment.find(payment_info['id'])
-            order = mark_order_refunded(order)
+            order = mark_order_refunded(order, user=request.user)
             order.payment_info = json.dumps(sale.to_dict())
             order.save()
 
diff --git a/src/pretix/plugins/stripe/payment.py b/src/pretix/plugins/stripe/payment.py
--- a/src/pretix/plugins/stripe/payment.py
+++ b/src/pretix/plugins/stripe/payment.py
@@ -202,10 +202,10 @@ def order_control_refund_perform(self, request, order) -> "bool|str":
                                       'support if the problem persists.'))
             logger.error('Stripe error: %s' % str(err))
         except stripe.error.StripeError:
-            mark_order_refunded(order)
+            mark_order_refunded(order, user=request.user)
             messages.warning(request, _('We were unable to transfer the money back automatically. '
                                         'Please get in touch with the customer and transfer it back manually.'))
         else:
-            order = mark_order_refunded(order)
+            order = mark_order_refunded(order, user=request.user)
             order.payment_info = str(ch)
             order.save()
diff --git a/src/pretix/plugins/stripe/views.py b/src/pretix/plugins/stripe/views.py
--- a/src/pretix/plugins/stripe/views.py
+++ b/src/pretix/plugins/stripe/views.py
@@ -55,6 +55,6 @@ def webhook(request, *args, **kwargs):
     order.log_action('pretix.plugins.stripe.event', data=event_json)
 
     if order.status == Order.STATUS_PAID and (charge['refunds']['total_count'] or charge['dispute']):
-        mark_order_refunded(order)
+        mark_order_refunded(order, user=None)
 
     return HttpResponse(status=200)


Test patch:
Repository: pretix/pretix

Issue:
Log old email when changing an order's email address
… because otherwise it's completely lost.


Discussion:
Oh, it's already logged, just not displayed. Whelp. I only saw that after implementing the log display with `old_email` and `new_email` as log attributes, so yay for consistent naming!

Resolving patch:
diff --git a/src/pretix/control/logdisplay.py b/src/pretix/control/logdisplay.py
--- a/src/pretix/control/logdisplay.py
+++ b/src/pretix/control/logdisplay.py
@@ -59,7 +59,6 @@ def pretixcontrol_logentry_display(sender: Event, logentry: LogEntry, **kwargs):
         'pretix.event.order.invoice.regenerated': _('The invoice has been regenerated.'),
         'pretix.event.order.invoice.reissued': _('The invoice has been reissued.'),
         'pretix.event.order.comment': _('The order\'s internal comment has been updated.'),
-        'pretix.event.order.contact.changed': _('The email address has been changed.'),
         'pretix.event.order.payment.changed': _('The payment method has been changed.'),
         'pretix.event.order.expire_warning_sent': _('An email has been sent with a warning that the order is about to expire.'),
         'pretix.user.settings.2fa.enabled': _('Two-factor authentication has been enabled.'),
@@ -75,6 +74,13 @@ def pretixcontrol_logentry_display(sender: Event, logentry: LogEntry, **kwargs):
     if logentry.action_type.startswith('pretix.event.order.changed'):
         return _display_order_changed(sender, logentry)
 
+    if logentry.action_type.startswith('pretix.event.order.contact.changed'):
+        data = json.loads(logentry.data)
+        return _('The email address has been changed from "{old}" to "{new}".').format(
+            old=data['old_email'],
+            new=data['new_email'],
+        )
+
     if logentry.action_type == 'pretix.user.settings.2fa.device.added':
         data = json.loads(logentry.data)
         return _('A new two-factor authentication device "{name}" has been added to your account.').format(


Test patch:
Repository: pretix/pretix

Issue:
Ticket PDF layout editor
- [x] Configurable positions and sizes
- [x] Own background PDFs
- [ ] Configurable fonts
- [ ] Drag&drop editor



Discussion:

Resolving patch:
diff --git a/src/pretix/control/views/user.py b/src/pretix/control/views/user.py
--- a/src/pretix/control/views/user.py
+++ b/src/pretix/control/views/user.py
@@ -262,6 +262,7 @@ def get_context_data(self, **kwargs):
         ctx = super().get_context_data()
 
         ctx['secret'] = base64.b32encode(self.device.bin_key).decode('utf-8')
+        ctx['secretGrouped'] = "  ".join([ctx['secret'].lower()[(i * 4): (i + 1) * 4] for i in range(len(ctx['secret']) // 4)])
         ctx['qrdata'] = 'otpauth://totp/{label}%3A%20{user}?issuer={label}&secret={secret}&digits={digits}'.format(
             label=quote(settings.PRETIX_INSTANCE_NAME), user=quote(self.request.user.email),
             secret=ctx['secret'],


Test patch:
Repository: pretix/pretix

Issue:
Voucher not being counted correctly
In the voucher by tag overview, the number sometimes just don't add up ;-)

![image](https://user-images.githubusercontent.com/654548/28206928-1c7ff030-6888-11e7-8891-14e5c794e730.png)

So the progressbar is showing nothing, because it's class is set to 200%. Okay, that's kinda expected to not work ;-)

Explanation for the numbers:
- 2: Two vouchers are created with this tag
- 4: Those two vouchers have been redeem twice each.

So I guess we have to decide if the overview should show the amount of vouchers or the amount of possible voucher redemptions.


Discussion:
We should probably show redemptions, not vouchers.
(This view is older than multi-use vouchers, so …)

Resolving patch:
diff --git a/src/pretix/control/views/vouchers.py b/src/pretix/control/views/vouchers.py
--- a/src/pretix/control/views/vouchers.py
+++ b/src/pretix/control/views/vouchers.py
@@ -101,7 +101,7 @@ def get_context_data(self, **kwargs):
         ctx = super().get_context_data(**kwargs)
 
         tags = self.request.event.vouchers.order_by('tag').filter(tag__isnull=False).values('tag').annotate(
-            total=Count('id'),
+            total=Sum('max_usages'),
             redeemed=Sum('redeemed')
         )
         for t in tags:


Test patch: