internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: mozilla/bugbug

Issue:
Make spawn_pipeline not depend on the order of tasks in the yaml file
Currently, if a task is defined in the yaml file before its dependencies, the spawn_pipeline script fails with:
```
Traceback (most recent call last):
  File "/code/spawn_pipeline.py", line 132, in <module>
    main()
  File "/code/spawn_pipeline.py", line 110, in main
    new_dependencies.append(id_mapping[dependency])
KeyError: 'regressor-finder'
```

So things like https://github.com/mozilla/bugbug/commit/aaa67b3b0a1db7530cbf88df644aff076fcd2e4e are needed.

We should make the spawn_pipeline script not depend on the order of definition of tasks in the yaml file.


Discussion:

Resolving patch:
diff --git a/infra/spawn_pipeline.py b/infra/spawn_pipeline.py
--- a/infra/spawn_pipeline.py
+++ b/infra/spawn_pipeline.py
@@ -85,13 +85,19 @@ def main():
         # mapping between an internal ID and the generate ID
 
         task_id = taskcluster.utils.slugId()
-        task_internal_id = task.pop("ID")
+        task_internal_id = task["ID"]
 
         if task_internal_id in id_mapping:
             raise ValueError(f"Conflicting IDs {task_internal_id}")
 
+        # Store each task ID in the id_mapping dictionary before processing dependencies.
+        # This way, tasks can be defined in any order.
         id_mapping[task_internal_id] = task_id
 
+    for task in rendered["tasks"]:
+        task_internal_id = task.pop("ID")
+        task_id = id_mapping[task_internal_id]
+
         for key, value in keys.items():
             task[key] = value
 


Test patch:
Repository: mozilla/bugbug

Issue:
Timeout when the list of commits to analyze is empty
We are analyzing all commits in that case, which makes the job timeout (it would take hours :D).


Discussion:

Resolving patch:
diff --git a/bugbug/repository.py b/bugbug/repository.py
--- a/bugbug/repository.py
+++ b/bugbug/repository.py
@@ -1005,9 +1005,10 @@ def download_commits(
     with hglib.open(repo_dir) as hg:
         if revs is None:
             revs = get_revs(hg, rev_start)
-            if len(revs) == 0:
-                logger.info("No commits to analyze")
-                return tuple()
+
+        if len(revs) == 0:
+            logger.info("No commits to analyze")
+            return tuple()
 
         first_pushdate = get_first_pushdate(repo_dir)
 


Test patch:
diff --git a/tests/test_repository.py b/tests/test_repository.py
--- a/tests/test_repository.py
+++ b/tests/test_repository.py
@@ -557,6 +557,14 @@ def test_download_commits(fake_hg_repo, use_single_process):
     )
     assert len(list(repository.get_commits())) == 2
 
+    os.remove("data/commits.json")
+    shutil.rmtree("data/commit_experiences.lmdb")
+    commits = repository.download_commits(
+        local, revs=[], use_single_process=use_single_process,
+    )
+    assert len(commits) == 0
+    assert len(list(repository.get_commits())) == 0
+
 
 def test_get_directories():
     assert repository.get_directories("") == []
Repository: mozilla/bugbug

Issue:
Set up Sentry for bugbug
Adding it will help investigate issues (like the Mercurial woes).


Discussion:

Resolving patch:
diff --git a/http_service/bugbug_http/worker.py b/http_service/bugbug_http/worker.py
--- a/http_service/bugbug_http/worker.py
+++ b/http_service/bugbug_http/worker.py
@@ -7,10 +7,20 @@
 import os
 import sys
 
+import sentry_sdk
 from redis import Redis
 from rq import Connection, Worker
+from sentry_sdk.integrations.rq import RqIntegration
 
 import bugbug_http.boot
+from bugbug import get_bugbug_version
+
+if os.environ.get("SENTRY_DSN"):
+    sentry_sdk.init(
+        os.environ.get("SENTRY_DSN"),
+        integrations=[RqIntegration()],
+        release=get_bugbug_version(),
+    )
 
 
 def main():


Test patch:
Repository: mozilla/bugbug

Issue:
Submit logged warnings to Sentry
Currently, we only report exceptions to Sentry.
It'd be nice to report warnings too, so we can get a sense of how often they happen.
IIRC in the code-review bot (https://github.com/mozilla/code-review), we are doing that kind of automatically.


Discussion:
This is probably relevant: https://github.com/mozilla/code-review/blob/0451e9285ab98ba54a620cbc37cf58c30ddee9df/tools/code_review_tools/log.py#L47-L77.

Resolving patch:
diff --git a/http_service/bugbug_http/worker.py b/http_service/bugbug_http/worker.py
--- a/http_service/bugbug_http/worker.py
+++ b/http_service/bugbug_http/worker.py
@@ -4,21 +4,29 @@
 # License, v. 2.0. If a copy of the MPL was not distributed with this file,
 # You can obtain one at http://mozilla.org/MPL/2.0/.
 
+import logging
 import os
 import sys
 
 import sentry_sdk
 from redis import Redis
 from rq import Connection, Worker
+from sentry_sdk.integrations.logging import LoggingIntegration
 from sentry_sdk.integrations.rq import RqIntegration
 
 import bugbug_http.boot
 from bugbug import get_bugbug_version
 
 if os.environ.get("SENTRY_DSN"):
+    logging_integration = LoggingIntegration(
+        # Default behaviour: INFO messages will be included as breadcrumbs
+        level=logging.INFO,
+        # Change default behaviour (ERROR messages events)
+        event_level=logging.WARNING,
+    )
     sentry_sdk.init(
-        os.environ.get("SENTRY_DSN"),
-        integrations=[RqIntegration()],
+        dsn=os.environ.get("SENTRY_DSN"),
+        integrations=[RqIntegration(), logging_integration],
         release=get_bugbug_version(),
     )
 


Test patch:
Repository: mozilla/bugbug

Issue:
Add Cyclomatic metric
See https://github.com/mozilla/rust-code-analysis/commit/8722dc5388ef6c25850cbb2caae125796054b435


Discussion:
There are many changes in the latest rust-code-analysis version that we'd need to pick up!

Resolving patch:
diff --git a/bugbug/repository.py b/bugbug/repository.py
--- a/bugbug/repository.py
+++ b/bugbug/repository.py
@@ -447,7 +447,7 @@ def get_metrics(commit, metrics_space):
 
     if metrics_space["kind"] == "function" and not error:
         metrics = metrics_space["metrics"]
-        commit.total_cyclomatic += metrics["cyclomatic"]
+        commit.total_cyclomatic += metrics["cyclomatic"]["sum"]
         commit.total_halstead_n2 += metrics["halstead"]["n2"]
         commit.total_halstead_N2 += metrics["halstead"]["N2"]
         commit.total_halstead_n1 += metrics["halstead"]["n1"]
@@ -456,7 +456,7 @@ def get_metrics(commit, metrics_space):
         commit.total_logical_loc += metrics["loc"]["lloc"]
 
         commit.maximum_cyclomatic = max(
-            commit.maximum_cyclomatic, metrics["cyclomatic"]
+            commit.maximum_cyclomatic, metrics["cyclomatic"]["sum"]
         )
         commit.maximum_halstead_n2 = max(
             commit.maximum_halstead_n2,
@@ -480,7 +480,7 @@ def get_metrics(commit, metrics_space):
         )
 
         commit.minimum_cyclomatic = min(
-            commit.minimum_cyclomatic, metrics["cyclomatic"]
+            commit.minimum_cyclomatic, metrics["cyclomatic"]["sum"]
         )
         commit.minimum_halstead_n2 = min(
             commit.minimum_halstead_n2,
diff --git a/bugbug/rust_code_analysis_server.py b/bugbug/rust_code_analysis_server.py
--- a/bugbug/rust_code_analysis_server.py
+++ b/bugbug/rust_code_analysis_server.py
@@ -45,7 +45,7 @@ def start_process(self):
 
         try:
             self.proc = subprocess.Popen(
-                ["rust-code-analysis", "--serve", "--port", str(self.port)]
+                ["rust-code-analysis-cli", "--serve", "--port", str(self.port)]
             )
         except FileNotFoundError:
             raise Exception("rust-code-analysis is required for code analysis")


Test patch:
Repository: mozilla/bugbug

Issue:
Replace typing.Dict with dict
It is now possible to use `dict` directly instead of `typing.Dict` in type definitions.


Discussion:
@marco-c I would like to work on this.

Few Questions:
- Is this specific to any specific file or the entire project? 
- Any reason as to why all the ```typing.Dict``` needs to be replaced with ```dict```?

> Is this specific to any specific file or the entire project?

The entire project.

> Any reason as to why all the ```typing.Dict``` needs to be replaced with ```dict```?

Simply because of code cleanup.
@marco-c 

I am working on this issue.
Please look at my pull request #2640 .
Some checks are failing.
Please help.

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -65,8 +65,6 @@ def read_requirements(file_):
         ]
     },
     classifiers=[
-        "Programming Language :: Python :: 3.7",
-        "Programming Language :: Python :: 3.8",
         "Programming Language :: Python :: 3.9",
         "Programming Language :: Python :: 3 :: Only",
         "License :: OSI Approved :: Mozilla Public License 2.0 (MPL 2.0)",


Test patch:
Repository: mozilla/bugbug

Issue:
Raise a custom exception when Last-Modified is not available
So we don't have to resort to https://github.com/mozilla/bugbug/blob/fa24337fc17c725c3017bc3ad303b4d8e43c7757/bugbug/phabricator.py#L150 but can directly catch the specific exception we want to catch.


Discussion:

Resolving patch:
diff --git a/bugbug/db.py b/bugbug/db.py
--- a/bugbug/db.py
+++ b/bugbug/db.py
@@ -22,6 +22,10 @@
 logger = logging.getLogger(__name__)
 
 
+class LastModifiedNotAvailable(Exception):
+    pass
+
+
 def register(path, url, version, support_files=[]):
     DATABASES[path] = {"url": url, "version": version, "support_files": support_files}
 
@@ -119,7 +123,7 @@ def last_modified(path):
     last_modified = utils.get_last_modified(url)
 
     if last_modified is None:
-        raise Exception("Last-Modified is not available")
+        raise LastModifiedNotAvailable()
 
     return last_modified
 
diff --git a/bugbug/phabricator.py b/bugbug/phabricator.py
--- a/bugbug/phabricator.py
+++ b/bugbug/phabricator.py
@@ -12,6 +12,7 @@
 from tqdm import tqdm
 
 from bugbug import db
+from bugbug.db import LastModifiedNotAvailable
 
 logger = logging.getLogger(__name__)
 
@@ -146,9 +147,8 @@ def download_revisions(rev_ids: Collection[int]) -> None:
 def download_modified_revisions():
     try:
         last_modified = db.last_modified(REVISIONS_DB)
-    except Exception as e:
-        if str(e) == "Last-Modified is not available":
-            return
+    except LastModifiedNotAvailable:
+        return
 
     modified_revisions = get(modified_start=last_modified)
     modified_revision_ids = set(rev["id"] for rev in modified_revisions)


Test patch:
diff --git a/tests/test_db.py b/tests/test_db.py
--- a/tests/test_db.py
+++ b/tests/test_db.py
@@ -12,6 +12,7 @@
 import responses
 
 from bugbug import db
+from bugbug.db import LastModifiedNotAvailable
 
 
 @pytest.fixture
@@ -179,7 +180,7 @@ def test_download_missing(tmp_path, mock_zst):
     assert not db.download(db_path)
     assert not os.path.exists(db_path)
 
-    with pytest.raises(Exception, match="Last-Modified is not available"):
+    with pytest.raises(LastModifiedNotAvailable):
         db.last_modified(db_path)
Repository: mozilla/bugbug

Issue:
Change needsdiagnosis model to consider moved milestone as `needsdiagnosis = True`
We have observed that needsdiagnosis model classifies certain issues that potentially need diagnosis as `needsdiagnosis = False`. While this is expected, I think it might be getting worse, as the issues data is unbalanced and has much more data points for `needsdiagnosis = False`. We've started a discussion in https://github.com/mozilla/webcompat-team-okrs/issues/256 

It's worth mentioning that in the [recent sample of 22 issues](https://docs.google.com/spreadsheets/d/1F9vcSpLQ_hNBeZinsytGXlfXpJLW6vh7C0BJYtd9hIY/edit?pli=1#gid=1640243023) most of the issues that looked like false negatives didn't end up needing diagnosis (not reproducible,  out of the scope of the project, etc.), so they're true negatives (prediction was correct for 21 of them). We'll continue tracking them to get a more representative sample.

As an example, issues that looked like they need diagnosis, but in the end, they didn't (prediction was correct):
https://github.com/webcompat/web-bugs/issues/100746
https://github.com/webcompat/web-bugs/issues/100676
https://github.com/webcompat/web-bugs/issues/100687

Issues that are false negatives (prediction was incorrect): 
https://github.com/webcompat/web-bugs/issues/100495
https://github.com/webcompat/web-bugs/issues/100645

I was thinking of including an additional set of issues that will contribute to `needsdiagnosis = True` pool. We have recently added a `moved` [milestone](https://github.com/webcompat/web-bugs/issues?q=is%3Aissue+milestone%3Amoved+is%3Aclosed). These issues often don't need diagnosis and are moved to bugzilla or elsewhere, but their content should be contributed to `needsdiagnosis = True`  rather than false. 



Discussion:

Resolving patch:
diff --git a/bugbug/models/needsdiagnosis.py b/bugbug/models/needsdiagnosis.py
--- a/bugbug/models/needsdiagnosis.py
+++ b/bugbug/models/needsdiagnosis.py
@@ -71,9 +71,9 @@ def get_labels(self):
                 continue
 
             for event in issue["events"]:
-                if (
-                    event["event"] == "milestoned"
-                    and event["milestone"]["title"] == "needsdiagnosis"
+                if event["event"] == "milestoned" and (
+                    event["milestone"]["title"] == "needsdiagnosis"
+                    or event["milestone"]["title"] == "moved"
                 ):
                     classes[issue["number"]] = 0
 


Test patch:
Repository: mozilla/bugbug

Issue:
Failing to access the value of `code_analysis_server` when running multi processes on Mac
The tests fail on Mac when running `test_repository.py`:
```
concurrent.futures.process._RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/opt/homebrew/Cellar/python@3.9/3.9.13_3/Frameworks/Python.framework/Versions/3.9/lib/python3.9/concurrent/futures/process.py", line 246, in _process_worker
    r = call_item.fn(*call_item.args, **call_item.kwargs)
  File "/opt/homebrew/Cellar/python@3.9/3.9.13_3/Frameworks/Python.framework/Versions/3.9/lib/python3.9/concurrent/futures/process.py", line 205, in _process_chunk
    return [fn(*args) for args in chunk]
  File "/opt/homebrew/Cellar/python@3.9/3.9.13_3/Frameworks/Python.framework/Versions/3.9/lib/python3.9/concurrent/futures/process.py", line 205, in <listcomp>
    return [fn(*args) for args in chunk]
  File "/Users/smujahid/repos/mozilla/bugbug/bugbug/repository.py", line 839, in _transform
    return transform(HG, REPO_DIR, commit)
  File "/Users/smujahid/repos/mozilla/bugbug/bugbug/repository.py", line 693, in transform
    assert code_analysis_server is not None
AssertionError
"""
```


Discussion:

Resolving patch:
diff --git a/bugbug/models/testselect.py b/bugbug/models/testselect.py
--- a/bugbug/models/testselect.py
+++ b/bugbug/models/testselect.py
@@ -7,6 +7,7 @@
 import concurrent.futures
 import logging
 import math
+import multiprocessing as mp
 import pickle
 import statistics
 from functools import reduce
@@ -835,7 +836,9 @@ def do_eval(
             print(message)
 
         with concurrent.futures.ProcessPoolExecutor(
-            max_workers=utils.get_physical_cpu_count()
+            max_workers=utils.get_physical_cpu_count(),
+            # Fixing https://github.com/mozilla/bugbug/issues/3131
+            mp_context=mp.get_context("fork"),
         ) as executor:
             scenarios = [
                 (None, None, None),
diff --git a/bugbug/repository.py b/bugbug/repository.py
--- a/bugbug/repository.py
+++ b/bugbug/repository.py
@@ -10,6 +10,7 @@
 import json
 import logging
 import math
+import multiprocessing as mp
 import os
 import pickle
 import re
@@ -1354,7 +1355,10 @@ def download_commits(
             code_analysis_server = rust_code_analysis_server.RustCodeAnalysisServer()
 
             with concurrent.futures.ProcessPoolExecutor(
-                initializer=_init_process, initargs=(repo_dir,)
+                initializer=_init_process,
+                initargs=(repo_dir,),
+                # Fixing https://github.com/mozilla/bugbug/issues/3131
+                mp_context=mp.get_context("fork"),
             ) as executor:
                 commits_iter = executor.map(_transform, commits, chunksize=64)
                 commits_iter = tqdm(commits_iter, total=commits_num)


Test patch:
Repository: mozilla/bugbug

Issue:
bugbug check components task failing because of "Component 'Preferences' of product 'Firefox' have 0 bugs or less in it, failure"



Discussion:

Resolving patch:
diff --git a/bugbug/bugzilla.py b/bugbug/bugzilla.py
--- a/bugbug/bugzilla.py
+++ b/bugbug/bugzilla.py
@@ -326,6 +326,9 @@ def get_product_component_count(months: int = 12) -> dict[str, int]:
                 continue
 
             value = int(raw_value)
+            # If there are no bugs, the product/component pair doesn't exist.
+            if value == 0:
+                continue
 
             full_comp = f"{product}::{component}"
             bugs_number[full_comp] = value


Test patch:
Repository: mozilla/bugbug

Issue:
Use information on how a bug is filed as a feature
This could be especially useful for the Spam model.

https://bugzilla.mozilla.org/show_bug.cgi?id=1565403


Discussion:
I would like to attempt this.
After going through `bug_features.py` and some of the bugs in `bugs.json`, I can't seem to find a field that would contain information about how a bug was filed. Am I missing something or is the field under a different name?

Do I have to change the bug mining script?
> Do I have to change the bug mining script?

Yes, if needed. The field on Bugzilla is called `filed_via`. 

I have completed training the spam bug model with the `filed_via` feature included. Should I share the performance reports here or I open a PR?
> I have completed training the spam bug model with the `filed_via` feature included. Should I share the performance reports here or I open a PR?

Feel free to open a PR. It will be helpful to highlight the improvements over what we already have.

Resolving patch:
diff --git a/bugbug/models/spambug.py b/bugbug/models/spambug.py
--- a/bugbug/models/spambug.py
+++ b/bugbug/models/spambug.py
@@ -41,6 +41,7 @@ def __init__(self, lemmatization=False):
             bug_features.has_attachment(),
             bug_features.platform(),
             bug_features.op_sys(),
+            bug_features.filed_via(),
         ]
 
         cleanup_functions = [


Test patch:
Repository: mozilla/bugbug

Issue:
Calibrate xgboost regressor model to make it isotonic
So its output can be interpreted as a confidence level.


Discussion:

Resolving patch:
diff --git a/bugbug/model_calibration.py b/bugbug/model_calibration.py
new file mode 100644
--- /dev/null
+++ b/bugbug/model_calibration.py
@@ -0,0 +1,29 @@
+# -*- coding: utf-8 -*-
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this file,
+# You can obtain one at http://mozilla.org/MPL/2.0/.
+
+from sklearn.base import BaseEstimator, ClassifierMixin
+from sklearn.calibration import CalibratedClassifierCV
+from sklearn.model_selection import train_test_split
+
+
+class IsotonicRegressionCalibrator(BaseEstimator, ClassifierMixin):
+    def __init__(self, base_clf):
+        self.base_clf = base_clf
+        self.calibrated_clf = CalibratedClassifierCV(
+            base_clf, cv="prefit", method="isotonic"
+        )
+
+    def fit(self, X_train, y_train):
+        X_train, X_val, y_train, y_val = train_test_split(
+            X_train, y_train, test_size=0.2, random_state=42
+        )
+        self.base_clf.fit(X_train, y_train)
+        self.calibrated_clf.fit(X_val, y_val)
+
+    def predict(self, X):
+        return self.calibrated_clf.predict(X)
+
+    def predict_proba(self, X):
+        return self.calibrated_clf.predict_proba(X)


Test patch:
Repository: mozilla/bugbug

Issue:
Refactor logging statements to use lazy % formatting
Example of logging statements that we want to refactor:

https://github.com/mozilla/bugbug/blob/d53595391dbd75379bb49bff12dee4821e4b956c/bugbug/github.py#L61

https://github.com/mozilla/bugbug/blob/69972a1684f788319bf5c2944bbe8eeb79428c7d/scripts/regressor_finder.py#L396

More details can be found in the [pylint docs](https://pylint.readthedocs.io/en/latest/user_guide/messages/warning/logging-fstring-interpolation.html).


Discussion:
@suhaibmujahid I would like to help out and work on this issue, would you please assign this to me :blush: 
> @suhaibmujahid I would like to help out and work on this issue, would you please assign this to me 😊

You can work on any issue which is open and has no PR linked to it.

Did you read https://github.com/mozilla/bugbug/issues/1092 or [CONTRIBUTING.md](https://github.com/mozilla/bugbug/blob/master/CONTRIBUTING.md)?
> > @suhaibmujahid I would like to help out and work on this issue, would you please assign this to me blush
> 
> You can work on any issue which is open and has no PR linked to it.
> 
> Did you read #1092 or [CONTRIBUTING.md](https://github.com/mozilla/bugbug/blob/master/CONTRIBUTING.md)?

yes I did

The target is to refactor all cases. Most of the cases still need to be refactored.
Hi @suhaibmujahid, are there any cases still left or is this issue fixed?
> Hi @suhaibmujahid, are there any cases still left or is this issue fixed?

I still see one case:

https://github.com/mozilla/bugbug/blob/f9906057a5281b8913fb5a92edbe73440953581b/scripts/retrieve_training_metrics.py#L98

Resolving patch:
diff --git a/scripts/retrieve_training_metrics.py b/scripts/retrieve_training_metrics.py
--- a/scripts/retrieve_training_metrics.py
+++ b/scripts/retrieve_training_metrics.py
@@ -95,7 +95,7 @@ def get_task_metrics_from_date(model, date, output_directory):
             file_path = abspath(join(output_directory, file_name))
             with open(file_path, "w") as metric_file:
                 metric_file.write(r.text)
-            LOGGER.info(f"Metrics saved to {file_path!r}")
+            LOGGER.info("Metrics saved to %r", file_path)
 
         for namespace in get_namespaces(index, ns_full_uri):
             new_ns = current_ns.copy()


Test patch:
Repository: mozilla/bugbug

Issue:
Use mozilla-unified instead of mozilla-central for the commit classifier
This way we are more likely to be able to find the base revision of the patch.


Discussion:

Resolving patch:
diff --git a/scripts/commit_classifier.py b/scripts/commit_classifier.py
--- a/scripts/commit_classifier.py
+++ b/scripts/commit_classifier.py
@@ -145,7 +145,7 @@ def __init__(
         self.git_repo_dir = git_repo_dir
         if git_repo_dir:
             self.clone_git_repo(
-                "hg::https://hg.mozilla.org/mozilla-central", git_repo_dir
+                "hg::https://hg.mozilla.org/mozilla-unified", git_repo_dir
             )
 
         self.method_defect_predictor_dir = method_defect_predictor_dir
@@ -236,7 +236,9 @@ def clone_git_repo(self, repo_url, repo_dir, rev="origin/branches/default/tip"):
         )
 
     def update_commit_db(self):
-        repository.clone(self.repo_dir, update=True)
+        repository.clone(
+            self.repo_dir, "https://hg.mozilla.org/mozilla-unified", update=True
+        )
 
         assert db.download(repository.COMMITS_DB, support_files_too=True)
 


Test patch: