internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: mozilla/bugbug

Issue:
Component check failing because of a component that is no longer active
The "component" check is failing with the following message:
> 2023-07-02 10:05:49,740:WARNING:bugbug.models.component:Component 'Graphics: Layers' of product 'Core' doesn't exists, failure

This is because, when training, we use all available components and when checking the model, we verify if there was at least one bug filed against that component in the past 12 months (https://github.com/mozilla/bugbug/blob/ed7c4460b41c6c0b6383e5532e6786dbaae27c27/bugbug/models/component.py#L237).


Discussion:

Resolving patch:
diff --git a/bugbug/bugzilla.py b/bugbug/bugzilla.py
--- a/bugbug/bugzilla.py
+++ b/bugbug/bugzilla.py
@@ -344,6 +344,32 @@ def get_product_component_count(months: int = 12) -> dict[str, int]:
     return bugs_number
 
 
+def get_active_product_components(products=[]) -> set[tuple[str, str]]:
+    r = utils.get_session("bugzilla").get(
+        "https://bugzilla.mozilla.org/rest/product",
+        params={
+            "type": "accessible",
+            "include_fields": [
+                "name",
+                "is_active",
+                "components.name",
+                "components.is_active",
+            ],
+            "names": products,
+        },
+        headers={"X-Bugzilla-API-Key": Bugzilla.TOKEN, "User-Agent": "bugbug"},
+    )
+    r.raise_for_status()
+
+    return set(
+        (product["name"], component["name"])
+        for product in r.json()["products"]
+        if product["is_active"]
+        for component in product["components"]
+        if component["is_active"]
+    )
+
+
 def get_component_team_mapping() -> dict[str, dict[str, str]]:
     r = utils.get_session("bugzilla").get(
         "https://bugzilla.mozilla.org/rest/product",
diff --git a/bugbug/models/component.py b/bugbug/models/component.py
--- a/bugbug/models/component.py
+++ b/bugbug/models/component.py
@@ -221,10 +221,14 @@ def get_meaningful_product_components(self, full_comp_tuples, threshold_ratio=10
         max_count = product_component_counts[0][1]
         threshold = max_count / threshold_ratio
 
+        active_product_components = bugzilla.get_active_product_components(
+            list(self.PRODUCTS) + list(self.PRODUCT_COMPONENTS)
+        )
+
         return set(
             product_component
             for product_component, count in product_component_counts
-            if count > threshold
+            if count > threshold and product_component in active_product_components
         )
 
     def get_feature_names(self):


Test patch:
Repository: mozilla/bugbug

Issue:
Drop the `bug` model
The model has been renamed to `defect` in https://github.com/mozilla/bugbug/pull/335/files#diff-74f32ab12cc9be314b1fead3a281590c768699c2e9883b8346066f6c4d6daa90 

However, the definition of `bug` model was not dropped.


We need to drop the following line: https://github.com/mozilla/bugbug/blob/c228c2a686b271f7ca24a0716cda4f9059229d9f/bugbug/models/__init__.py#L16



Discussion:

Resolving patch:
diff --git a/bugbug/models/__init__.py b/bugbug/models/__init__.py
--- a/bugbug/models/__init__.py
+++ b/bugbug/models/__init__.py
@@ -13,7 +13,6 @@
     "assignee": "bugbug.models.assignee.AssigneeModel",
     "backout": "bugbug.models.backout.BackoutModel",
     "browsername": "bugbug.models.browsername.BrowserNameModel",
-    "bug": "bugbug.model.BugModel",
     "bugtype": "bugbug.models.bugtype.BugTypeModel",
     "component": "bugbug.models.component.ComponentModel",
     "component_nn": "bugbug.models.component_nn.ComponentNNModel",


Test patch:
Repository: mozilla/bugbug

Issue:
Use all threads when decompressing with zstd
We should use the `-T0` option (or use the `zstdmt` binary instead of `zstd`).


Discussion:
@lutfiaomarr you do not need to be assigned to work on the issue. Please read https://github.com/mozilla/bugbug/issues/1092.

Resolving patch:
diff --git a/bugbug/utils.py b/bugbug/utils.py
--- a/bugbug/utils.py
+++ b/bugbug/utils.py
@@ -244,7 +244,7 @@ def zstd_decompress(path: str) -> None:
     if not os.path.exists(f"{path}.zst"):
         raise FileNotFoundError(errno.ENOENT, os.strerror(errno.ENOENT), path)
 
-    subprocess.run(["zstd", "-df", f"{path}.zst"], check=True)
+    subprocess.run(["zstdmt", "-df", f"{path}.zst"], check=True)
 
 
 @contextmanager
@@ -279,7 +279,7 @@ def extract_tar_zst(path: str) -> None:
     if not os.path.exists(path):
         raise FileNotFoundError(errno.ENOENT, os.strerror(errno.ENOENT), path)
 
-    subprocess.run(["tar", "-I", "zstd", "-xf", path], check=True)
+    subprocess.run(["tar", "-I", "zstdmt", "-xf", path], check=True)
 
 
 def extract_file(path: str) -> None:


Test patch:
diff --git a/tests/test_utils.py b/tests/test_utils.py
--- a/tests/test_utils.py
+++ b/tests/test_utils.py
@@ -302,6 +302,29 @@ def test_zstd_compress_not_existing(tmp_path, mock_zst):
     assert not os.path.exists(compressed_path)
 
 
+def test_create_extract_tar_zst(tmp_path):
+    path = tmp_path / "prova"
+    tar_zst_path = "prova.tar.zst"
+
+    with open(path, "w") as f:
+        json.dump({"Hello": "World"}, f)
+
+    utils.create_tar_zst(tar_zst_path)
+
+    assert os.path.exists(tar_zst_path)
+
+    os.remove(path)
+
+    utils.extract_tar_zst(tar_zst_path)
+
+    assert os.path.exists(path)
+
+    with open(path, "r") as f:
+        file_decomp = json.load(f)
+
+    assert file_decomp == {"Hello": "World"}
+
+
 def test_extract_db_zst(tmp_path, mock_zst):
     path = tmp_path / "prova.zst"
Repository: mozilla/bugbug

Issue:
MEI: Shift indicator from ratio to % and remove the artificial +1 from the division
Based on [this doc](https://docs.google.com/document/d/1nWP3DtSWneYT8ietXuBa6uPszMKMgICXf3b2dwMVXYk/edit#heading=h.9joq5dbpnw41).

```
If WeighedOpenedDefects(TimeInterval) > 0:
    ME% := (WeighedClosedDefects(TimeInterval) * 100%) / WeighedOpenedDefects(TimeInterval)
Else
    ME% := (WeighedClosedDefects(TimeInterval) + 1) * 100%
```

Note: The +1 in the else case just ensures that for the case of 0 closed defects we get 100% and anything above 0 is (very) positive. But generally speaking the result is of low value if there are no incoming defects at all.


Discussion:

Resolving patch:
diff --git a/bugbug/bugzilla.py b/bugbug/bugzilla.py
--- a/bugbug/bugzilla.py
+++ b/bugbug/bugzilla.py
@@ -514,4 +514,9 @@ def calculate_maintenance_effectiveness_indicator(
     print("After applying weights:")
     print(data)
 
-    return (1 + sum(data["closed"].values())) / (1 + sum(data["opened"].values()))
+    weighed_opened_defects = sum(data["opened"].values())
+    weighed_closed_defects = sum(data["closed"].values())
+    if weighed_opened_defects > 0:
+        return 100 * weighed_closed_defects / weighed_opened_defects
+    else:
+        return 100 * (weighed_closed_defects + 1)


Test patch:
Repository: mozilla/bugbug

Issue:
MEI: Have a measure of incoming/closed defects vs. existing open
Based on [this doc](https://docs.google.com/document/d/1nWP3DtSWneYT8ietXuBa6uPszMKMgICXf3b2dwMVXYk/edit#heading=h.9joq5dbpnw41).

```
Incoming(TimeInterval) := NumOpenedDefects(TimeInterval) / NumOpenDefects(Start(TimeInterval))
Closed(TimeInterval)  := NumClosedDefects(TimeInterval) / NumOpenDefects(Start(TimeInterval))
```

Given that `NumOpenDefects(Start(TimeInterval))` is hard to query live, we estimate it by:

```
NumOpenDefects(Start(TimeInterval)) := NumOpenDefects(End(TimeInterval)) + NumClosedDefects(TimeInterval) - NumOpenedDefects(TimeInterval)
```

such that if the calculation is run with an end date of the time interval that is close to today we can use the up-to-date values for components and severity without bothering too much for deviations.



Discussion:
@jensstutte two questions:
- should this be made a percentage as well?
- do we want both weighed and non-weighed or only non-weighed? In the doc you mentioned the weighed, while here you mentioned the non-weighed
(Edit: misconception on my side)
Percentage is fine. I switched to non-weighed as this is probably more relatable for who knows their backlogs.
(it might be nice to have links in the mail that open a query showing those bugs?)

Resolving patch:
diff --git a/bugbug/bugzilla.py b/bugbug/bugzilla.py
--- a/bugbug/bugzilla.py
+++ b/bugbug/bugzilla.py
@@ -557,8 +557,14 @@ def calculate_maintenance_effectiveness_indicator(
     else:
         wbdtime = math.inf
 
+    estimated_start_open_defects = open_defects + closed_defects - opened_defects
+    incoming = 100 * opened_defects / estimated_start_open_defects
+    closed = 100 * closed_defects / estimated_start_open_defects
+
     return {
         "ME%": mei,
         "BDTime": bdtime,
         "WBDTime": wbdtime,
+        "Incoming%": incoming,
+        "Closed%": closed,
     }


Test patch:
Repository: mozilla/bugbug

Issue:
Train models on TaskCluster whenever their corresponding files are modified
Whenever a file in the `bugbug/models` directory is modified, we could have a task to train the corresponding model.

For example, if a new Pull Request is modifying the `bugbug/models/defect.py` file, the task should train the `defect` model:
```
python -m scripts.trainer defect
```


Discussion:
As a first step, we could build a script to do this "manually" and not on PRs (doing it on PRs might be a bit complicated, as we need to run the whole pipeline to be able to train a model)
It would be nice if we could trigger TaskCluster training using the same script with an optional argument. 

Example:
```
python -m scripts.trainer defect --taskcluster
```
As suggested by Marco, we could use [TaskCluster API](https://docs.taskcluster.net/docs/manual/using/api) to replicate creating a task like https://community-tc.services.mozilla.com/tasks/RpQqfhKgT6GTlvOmK9i_vw/runs/0

Resolving patch:
diff --git a/scripts/trainer_extract_args.py b/scripts/trainer_extract_args.py
new file mode 100644
--- /dev/null
+++ b/scripts/trainer_extract_args.py
@@ -0,0 +1,39 @@
+# -*- coding: utf-8 -*-
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this file,
+# You can obtain one at http://mozilla.org/MPL/2.0/.
+
+import logging
+import os
+import re
+
+logging.basicConfig(level=logging.INFO)
+logger = logging.getLogger(__name__)
+
+
+def get_model_name() -> str | None:
+    pr_description = os.environ.get("PR_DESCRIPTION")
+    if not pr_description:
+        logger.error("The PR_DESCRIPTION environment variable does not exist")
+        return None
+
+    match = re.search(r"Train on Taskcluster:\s+([a-z_1-9]+)", pr_description)
+    if not match:
+        logger.error(
+            "Could not identify the model name using the 'Train on Taskcluster' keyword from the Pull Request description"
+        )
+        return None
+
+    model_name = match.group(1)
+
+    return model_name
+
+
+def main():
+    model = get_model_name()
+    if model:
+        print(model)
+
+
+if __name__ == "__main__":
+    main()


Test patch:
Repository: mozilla/bugbug

Issue:
Prepare to use the latest version of scikit-learn
Currently, when upgrading to the latest version of scikit-learn, training of some models takes long and in particular cases hangs. Investigating and resolving the cause of the hang is crucial towards upgrading the dependency. 

Related Issues and PRs:
- #3770 
- #3671 
- #3559 
- #3308


Discussion:
The newly added task to train on task cluster in #3748  could be helpful towards this solving this issue. 

Resolving patch:
diff --git a/bugbug/utils.py b/bugbug/utils.py
--- a/bugbug/utils.py
+++ b/bugbug/utils.py
@@ -71,8 +71,8 @@ def to_array(val):
 
 
 class StructuredColumnTransformer(ColumnTransformer):
-    def _hstack(self, Xs):
-        result = super()._hstack(Xs)
+    def _hstack(self, Xs, n_samples):
+        result = super()._hstack(Xs, n_samples=n_samples)
 
         transformer_names = (name for name, transformer, column in self.transformers_)
         types = []


Test patch:
Repository: mozilla/bugbug

Issue:
Failure in defect_model.classify
```
Traceback (most recent call last):
  File "/usr/local/bin/bugbug-regressor-finder", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.10/site-packages/scripts/regressor_finder.py", line 651, in main
    regressor_finder.find_bug_fixing_commits()
  File "/usr/local/lib/python3.10/site-packages/scripts/regressor_finder.py", line 276, in find_bug_fixing_commits
    classified_bugs = defect_model.classify(bugs_to_classify)
  File "/usr/local/lib/python3.10/site-packages/bugbug/model.py", line 629, in classify
    X = self.extraction_pipeline.transform(lambda: items)
  File "/usr/local/lib/python3.10/site-packages/sklearn/pipeline.py", line 635, in transform
    Xt = transform.transform(Xt)
  File "/usr/local/lib/python3.10/site-packages/bugbug/bug_features.py", line 737, in transform
    first_bug = next(bugs_iter)
TypeError: 'list' object is not an iterator
```

This is a regression from 9bc7dd03b9a577bd6009aeed88d90e8f26f4d0da.


Discussion:

Resolving patch:
diff --git a/bugbug/bug_features.py b/bugbug/bug_features.py
--- a/bugbug/bug_features.py
+++ b/bugbug/bug_features.py
@@ -733,7 +733,7 @@ def fit(self, x, y=None):
     def transform(self, bugs):
         results = []
 
-        bugs_iter = bugs()
+        bugs_iter = iter(bugs())
         first_bug = next(bugs_iter)
         bugs_iter = itertools.chain([first_bug], bugs_iter)
         is_couple = isinstance(first_bug, tuple)


Test patch:
Repository: mozilla/bugbug

Issue:
Integration test task hanging while rollbacking
It is working in production, but failing in the integration task.

Regression from 3ddaf4197b0636c0979be463e1d1ca8a46417522. I'll just revert it for now.


Discussion:

Resolving patch:
diff --git a/bugbug/bug_snapshot.py b/bugbug/bug_snapshot.py
--- a/bugbug/bug_snapshot.py
+++ b/bugbug/bug_snapshot.py
@@ -4,7 +4,6 @@
 # You can obtain one at http://mozilla.org/MPL/2.0/.
 
 from logging import INFO, basicConfig, getLogger
-from multiprocessing.pool import Pool
 
 import dateutil.parser
 from dateutil.relativedelta import relativedelta
@@ -869,21 +868,17 @@ def parse_flag_change(change):
     return bug
 
 
-def _try_rollback(bug):
-    try:
-        return rollback(bug, do_assert=True), True
-    except Exception:
-        logger.exception("Failed to rollback bug %s", bug["id"])
-        return bug, False
+def get_inconsistencies(bugs):
+    inconsistencies = []
 
+    for bug in bugs:
+        try:
+            rollback(bug, do_assert=True)
+        except Exception:
+            logger.exception("Failed to rollback bug %s", bug["id"])
+            inconsistencies.append(bug)
 
-def get_inconsistencies(bugs):
-    with Pool() as p:
-        return [
-            bug
-            for bug, is_successful in p.imap(_try_rollback, bugs, chunksize=1024)
-            if not is_successful
-        ]
+    return inconsistencies
 
 
 if __name__ == "__main__":


Test patch:
Repository: mozilla/bugbug

Issue:
Optimize the `bug_to_types()` functions to avoid redundant the execution of string lowering
The `bug["whiteboard"].lower()` statement is performed multiple times in the function. Instead, we could store the lowercase value in a variable (e.g., `bug_whiteboard`) and reuse it where needed.

Examples:

https://github.com/mozilla/bugbug/blob/0586c6a49732f03781daf43d8726b53e2049e3db/bugbug/models/bugtype.py#L46

https://github.com/mozilla/bugbug/blob/0586c6a49732f03781daf43d8726b53e2049e3db/bugbug/models/bugtype.py#L50



Discussion:
Hello @suhaibmujahid , Could I please work on this issue ?
> Could I please work on this issue ?

@sreeroop10 Yes, you could work on any issue that does not have an open PR linked to it.

Resolving patch:
diff --git a/bugbug/models/bugtype.py b/bugbug/models/bugtype.py
--- a/bugbug/models/bugtype.py
+++ b/bugbug/models/bugtype.py
@@ -62,7 +62,7 @@ def bug_to_types(
         types.add("performance")
 
     if any(
-        f"[{whiteboard_text}" in bug["whiteboard"].lower()
+        f"[{whiteboard_text}" in bug_whiteboard
         for whiteboard_text in ("client-bounty-form", "sec-survey")
     ):
         types.add("security")


Test patch:
Repository: mozilla/bugbug

Issue:
Some logging statements don't use the appropriate % operator 
> This is a valid fix but unrelated to the goal of this PR. We could open a new issue and fix it in a separate PR (with similar cases if there are any)

_Originally posted by @suhaibmujahid in https://github.com/mozilla/bugbug/pull/3822#discussion_r1398670853_
            


Discussion:
Example:
https://github.com/mozilla/bugbug/blob/97a0f86dd68b9e41a5d2640406cfb4ffba68d15e/bugbug/models/regressor.py#L279-L281

We should use `%0.2f` instead of `%d`.

Resolving patch:
diff --git a/bugbug/models/regressor.py b/bugbug/models/regressor.py
--- a/bugbug/models/regressor.py
+++ b/bugbug/models/regressor.py
@@ -278,7 +278,7 @@ def evaluation(self) -> None:
         total_regressions = sum(is_reg for _, is_reg in results)
         average_regression_rate = total_regressions / total_landings
 
-        logger.info("Average risk is %d", average_regression_rate)
+        logger.info("Average risk is %0.2f", average_regression_rate)
 
         MIN_SAMPLE = 200
 


Test patch:
Repository: mozilla/bugbug

Issue:
Train the component model on bugs from Fenix
It is now appropriate to include Fenix in the training dataset since it has been moved to Bugzilla.






Discussion:

Resolving patch:
diff --git a/bugbug/models/component.py b/bugbug/models/component.py
--- a/bugbug/models/component.py
+++ b/bugbug/models/component.py
@@ -27,6 +27,7 @@ class ComponentModel(BugModel):
         "Core",
         "External Software Affecting Firefox",
         "DevTools",
+        "Fenix",
         "Firefox",
         "Toolkit",
         "WebExtensions",
@@ -52,6 +53,7 @@ class ComponentModel(BugModel):
         "External Software Affecting Firefox",
         "WebExtensions",
         "Firefox Build System",
+        "Fenix",
     ]
 
     CONFLATED_COMPONENTS_MAPPING = {
@@ -62,6 +64,7 @@ class ComponentModel(BugModel):
         "External Software Affecting Firefox": "External Software Affecting Firefox::Other",
         "WebExtensions": "WebExtensions::Untriaged",
         "Firefox Build System": "Firefox Build System::General",
+        "Fenix": "Fenix::General",
     }
 
     def __init__(self, lemmatization=False):


Test patch:
Repository: mozilla/bugbug

Issue:
[model:regressor] AttributeError: 'IsotonicRegressionCalibrator' object has no attribute 'n_features_in_'
https://community-tc.services.mozilla.com/tasks/HncpjvKKRcSnxL_GJ8PV9A/runs/0/logs/public/logs/live.log
```
Traceback (most recent call last):
  File "/usr/local/bin/bugbug-train", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.10/site-packages/scripts/trainer.py", line 141, in main
    retriever.go(args)
  File "/usr/local/lib/python3.10/site-packages/scripts/trainer.py", line 41, in go
    metrics = model_obj.train(limit=args.limit)
  File "/usr/local/lib/python3.10/site-packages/bugbug/model.py", line 418, in train
    logger.info("Number of features: %d", self.clf.steps[-1][1].n_features_in_)
AttributeError: 'IsotonicRegressionCalibrator' object has no attribute 'n_features_in_'
```


Discussion:
I think using the `n_features_in_` attribute from the `base_clf` (after fitting) should solve this.
`self.n_features_in_ = self.base_clf.n_features_in_`


Resolving patch:
diff --git a/bugbug/model_calibration.py b/bugbug/model_calibration.py
--- a/bugbug/model_calibration.py
+++ b/bugbug/model_calibration.py
@@ -27,3 +27,7 @@ def predict(self, X):
 
     def predict_proba(self, X):
         return self.calibrated_clf.predict_proba(X)
+
+    @property
+    def n_features_in_(self):
+        return self.base_clf.n_features_in_


Test patch:
Repository: mozilla/bugbug

Issue:
Remove duplicate imports for `dateutil.parser`
> I noticed that `dateutil.parser` was imported twice. I don't think it's necessary, is it?
> 
> https://github.com/mozilla/bugbug/blob/8e4235c7f7b765247e2b305d62d763d293095243/bugbug/bug_features.py#L13-L15

_Originally posted by @PromiseFru in https://github.com/mozilla/bugbug/issues/3907#issuecomment-1854819315_
            


Discussion:

Resolving patch:
diff --git a/bugbug/bug_features.py b/bugbug/bug_features.py
--- a/bugbug/bug_features.py
+++ b/bugbug/bug_features.py
@@ -10,7 +10,6 @@
 from functools import partial
 from multiprocessing.pool import Pool
 
-import dateutil.parser
 import pandas as pd
 from dateutil import parser
 from libmozdata import versions
@@ -538,8 +537,7 @@ def get_time_to_fix(bug):
         return None
 
     return (
-        dateutil.parser.parse(bug["cf_last_resolved"])
-        - dateutil.parser.parse(bug["creation_time"])
+        parser.parse(bug["cf_last_resolved"]) - parser.parse(bug["creation_time"])
     ).total_seconds() / 86400
 
 
@@ -557,8 +555,7 @@ def get_time_to_assign(bug):
                 and change["added"] == "ASSIGNED"
             ):
                 return (
-                    dateutil.parser.parse(history["when"])
-                    - dateutil.parser.parse(bug["creation_time"])
+                    parser.parse(history["when"]) - parser.parse(bug["creation_time"])
                 ).total_seconds() / 86400
 
     return None


Test patch:
Repository: mozilla/bugbug

Issue:
[model:accessibility] Add the model to `http_service` and `data_pipeline`
Depends on merging: #3775


Discussion:

Resolving patch:
diff --git a/http_service/bugbug_http/models.py b/http_service/bugbug_http/models.py
--- a/http_service/bugbug_http/models.py
+++ b/http_service/bugbug_http/models.py
@@ -34,6 +34,7 @@
     "spambug",
     "testlabelselect",
     "testgroupselect",
+    "accessibility",
 ]
 
 DEFAULT_EXPIRATION_TTL = 7 * 24 * 3600  # A week


Test patch:
Repository: mozilla/bugbug

Issue:
UnboundLocalError: local variable 'median_fix_time_text' referenced before assignment when generating quality report
```
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/scripts/generate_landings_risk_report.py", line 1673, in notification
    {median_fix_time_text}
UnboundLocalError: local variable 'median_fix_time_text' referenced before assignment
```


Discussion:

Resolving patch:
diff --git a/scripts/generate_landings_risk_report.py b/scripts/generate_landings_risk_report.py
--- a/scripts/generate_landings_risk_report.py
+++ b/scripts/generate_landings_risk_report.py
@@ -1546,6 +1546,9 @@ def get_top_crashes(team: str, channel: str) -> str | None:
                     fix_time_diff = f"This is {verb} when compared to two weeks ago (median was {prev_median_fix_time} days)."
                 else:
                     fix_time_diff = ""
+            else:
+                median_fix_time_text = ""
+                fix_time_diff = ""
 
             top_intermittent_failures = "\n".join(
                 "{} failures ({}#{} globally{}){}".format(


Test patch: