internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: mozilla/bugbug

Issue:
'Is reporter a Mozillian' as a feature
The first implementation will simply check if an email contains "@mozilla.com" or "@mozilla.org".


Discussion:
I have couple of questions.
1. To add this feature we just need to check in `bugs.json` file that 'creator_detail' contains email that ends with "@mozilla.com" or "@mozilla.org".   
2. After going through some data i found there are email with "@mozilla.bugs". Does this count as `Mozillian reporter` or not?
1. You'll need to add an additional feature extractor in bugbug/bug_features.py.
2. No, I don't think so. Most of them should be automated accounts. Could you list here all the emails that end with "@mozilla.bugs"?
 ` 'wptsync@mozilla.bugs' 
 'omphalos@mozilla.bugs'
 'intermittent-bug-filer@mozilla.bugs' `
This are 3 emails that ends with "@mozilla.bugs".
Yes we can ignore them.

Resolving patch:
diff --git a/bugbug/bug_features.py b/bugbug/bug_features.py
--- a/bugbug/bug_features.py
+++ b/bugbug/bug_features.py
@@ -126,6 +126,11 @@ def __call__(self, bug):
         return bug['component']
 
 
+class is_mozillian(object):
+    def __call__(self, bug):
+        return any(bug['creator_detail']['email'].endswith(domain) for domain in ['@mozilla.com', '@mozilla.org'])
+
+
 def cleanup_url(text):
     text = re.sub(r'http[s]?://(hg.mozilla|searchfox|dxr.mozilla)\S+', '__CODE_REFERENCE_URL__', text)
     return re.sub(r'http\S+', '__URL__', text)


Test patch:
Repository: mozilla/bugbug

Issue:
Unnecessary string formatting when saving the model
https://github.com/mozilla/bugbug/blob/ab007344c5c799c08739b2d56f334afcaf242919/bugbug/model.py#L111 is not needed, we can use the variable directly.


Discussion:
Hi @marco-c 

I am working on it.

Resolving patch:
diff --git a/bugbug/model.py b/bugbug/model.py
--- a/bugbug/model.py
+++ b/bugbug/model.py
@@ -104,7 +104,7 @@ def bugs():
             print(metrics.confusion_matrix(y_test_filter, y_pred_filter, labels=class_names))
             print(classification_report_imbalanced(y_test_filter, y_pred_filter, labels=class_names))
 
-        joblib.dump(self, f'{self.__class__.__name__.lower()}')
+        joblib.dump(self, self.__class__.__name__.lower())
 
     @staticmethod
     def load(model_file_name):


Test patch:
Repository: mozilla/bugbug

Issue:
Add 'priority' feature
In bugbug/bug_features.py


Discussion:
I need to implement a feature extractor class `priority` to extract bug priorities right?
Correct!
I'm not able to run the project locally. I've installed all the requirements but it shows an error:
```
HP@LAPTOP-6V5RSGD0 MINGW64 /f/COMPILED/Git/bugbug (master)
$ python -i run.py --train
Traceback (most recent call last):
  File "run.py", line 25, in <module>
    from bugbug.models.bug import BugModel
  File "F:\COMPILED\Git\bugbug\bugbug\models\bug.py", line 14, in <module>
    from bugbug.model import Model
  File "F:\COMPILED\Git\bugbug\bugbug\model.py", line 7, in <module>
    import shap
  File "C:\Anaconda3\lib\site-packages\shap\__init__.py", line 11, in <module>
    from .plots.summary import summary_plot
  File "C:\Anaconda3\lib\site-packages\shap\plots\summary.py", line 13, in <module>
    from . import colors
  File "C:\Anaconda3\lib\site-packages\shap\plots\colors.py", line 5, in <module>
    import skimage.color
  File "C:\Anaconda3\lib\site-packages\skimage\__init__.py", line 167, in <module>
    from .util.dtype import (img_as_float32,
  File "C:\Anaconda3\lib\site-packages\skimage\util\__init__.py", line 8, in <module>
    from .arraycrop import crop
  File "C:\Anaconda3\lib\site-packages\skimage\util\arraycrop.py", line 8, in <module>
    from numpy.lib.arraypad import _validate_lengths
ImportError: cannot import name '_validate_lengths'
```

Please help.
@sbeesm please open a separate issue, as your problem is unrelated to this issue.
The error was due to the incompatible versions of scikit-image and numpy.
I've opened an issue #120 related to this. 

Resolving patch:
diff --git a/bugbug/bug_features.py b/bugbug/bug_features.py
--- a/bugbug/bug_features.py
+++ b/bugbug/bug_features.py
@@ -162,6 +162,11 @@ def __call__(self, bug):
         return sum((commit['types'] for commit in bug['commits']), [])
 
 
+class priority(object):
+    def __call__(self, bug):
+        return bug['priority']
+
+
 def cleanup_url(text):
     text = re.sub(r'http[s]?://(hg.mozilla|searchfox|dxr.mozilla)\S+', '__CODE_REFERENCE_URL__', text)
     return re.sub(r'http\S+', '__URL__', text)
diff --git a/bugbug/models/tracking.py b/bugbug/models/tracking.py
--- a/bugbug/models/tracking.py
+++ b/bugbug/models/tracking.py
@@ -31,6 +31,7 @@ def __init__(self, lemmatization=False):
             bug_features.patches(),
             bug_features.landings(),
             bug_features.title(),
+            bug_features.priority(),
         ]
 
         cleanup_functions = [


Test patch:
Repository: mozilla/bugbug

Issue:
Add number of touched files as a feature



Discussion:
Could the number of files touched be the length of `types` attribute list in commits data?

`"types": ["JavaScript", ".ini", ".build"]`
No, the types are only the types. If you have ten JavaScript files the length of types will be 1, but we want 10.
As of now, we are just downloading commits data and not parsing it like in `repository.py`, right?

So do we somehow need to parse the bug id related commit data from bugzilla? Something like extracting info from attachments diff?
We can simply change `repository.py` to add the required info.
@marco-c This means that the bug data will now contain a new field called `desc` right? (Obtained from commits.json for each bug element)
@ayush1999 what do you mean?
Will the `number of touched files` feature be extracted from the comments in the bug `bug['comments']`? Or would this data need to be extracted from commits.json file?
We can't extract it from the comments, as they don't contain this information.

It would need to be extracted in `repository.py`, and stored in `commits.json`.

Resolving patch:
diff --git a/bugbug/bug_features.py b/bugbug/bug_features.py
--- a/bugbug/bug_features.py
+++ b/bugbug/bug_features.py
@@ -183,6 +183,11 @@ def __call__(self, bug):
         return bug['priority']
 
 
+class commit_files_modified_num(object):
+    def __call__(self, bug):
+        return sum(commit['files_modified_num'] for commit in bug['commits'])
+
+
 class comment_count(object):
     def __call__(self, bug):
         return field(bug, 'comment_count')
diff --git a/bugbug/repository.py b/bugbug/repository.py
--- a/bugbug/repository.py
+++ b/bugbug/repository.py
@@ -48,6 +48,7 @@ def _transform(commit):
         'bug_id': bug_id,
         'added': 0,
         'deleted': 0,
+        'files_modified_num': 0,
         'types': set(),
     }
 
@@ -74,6 +75,8 @@ def _transform(commit):
             type_ = ext
         obj['types'].add(type_)
 
+    obj['files_modified_num'] = len(patch_data)
+
     # Covert to a list, as a set is not JSON-serializable.
     obj['types'] = list(obj['types'])
 


Test patch:
Repository: mozilla/bugbug

Issue:
Fix linting issue preventing us from updating to flake8 4.7.1
https://github.com/mozilla/bugbug/pull/135 is failing because:
> ./bugbug/model.py:102:21: E117 over-indented

We should fix it, and then we can merge https://github.com/mozilla/bugbug/pull/135.


Discussion:

Resolving patch:
diff --git a/bugbug/model.py b/bugbug/model.py
--- a/bugbug/model.py
+++ b/bugbug/model.py
@@ -99,7 +99,7 @@ def bugs():
 
             print(f'\nTop {len(important_features)} Features :')
             for i, [importance, index, is_positive] in enumerate(important_features):
-                    print(f'{i + 1}. \'{feature_names[int(index)]}\' ({"+" if (is_positive) else "-"}{importance})')
+                print(f'{i + 1}. \'{feature_names[int(index)]}\' ({"+" if (is_positive) else "-"}{importance})')
 
         y_pred = self.clf.predict(X_test)
 


Test patch:
Repository: mozilla/bugbug

Issue:
Print progress bar while downloading bugs
Similar to what we're doing for commits data: 28b83c12c29185c52afb58d94a533a9448969a8a.


Discussion:
Sounds interesting. I am a bit of a newbie, if you could give me some advice on how to do this, I'd be happy to try 👍 
You have an example in the first comment :)
You can just do something similar in the function to download bugs in bugbug/bugzilla.py.
@marco-c are these changes enough?
@akashgosai sorry, the issue is reserved for new contributors (people who haven't done any commit to the project yet).
Okay.

Resolving patch:
diff --git a/bugbug/bugzilla.py b/bugbug/bugzilla.py
--- a/bugbug/bugzilla.py
+++ b/bugbug/bugzilla.py
@@ -8,6 +8,7 @@
 
 import requests
 from libmozdata import bugzilla
+from tqdm import tqdm
 
 from bugbug import db
 
@@ -136,19 +137,17 @@ def download_bugs(bug_ids, products=None, security=False):
 
     new_bug_ids = sorted(list(new_bug_ids))
 
-    total_downloaded = 0
     chunks = (new_bug_ids[i:(i + 500)] for i in range(0, len(new_bug_ids), 500))
-    for chunk in chunks:
-        new_bugs = _download(chunk)
+    with tqdm(total=len(new_bug_ids)) as progress_bar:
+        for chunk in chunks:
+            new_bugs = _download(chunk)
 
-        total_downloaded += len(new_bugs)
+            progress_bar.update(len(chunk))
 
-        print(f'Downloaded {total_downloaded} out of {len(new_bug_ids)} bugs')
+            if not security:
+                new_bugs = {bug_id: bug for bug_id, bug in new_bugs.items() if len(bug['groups']) == 0}
 
-        if not security:
-            new_bugs = {bug_id: bug for bug_id, bug in new_bugs.items() if len(bug['groups']) == 0}
+            if products is not None:
+                new_bugs = {bug_id: bug for bug_id, bug in new_bugs.items() if bug['product'] in products}
 
-        if products is not None:
-            new_bugs = {bug_id: bug for bug_id, bug in new_bugs.items() if bug['product'] in products}
-
-        db.append(BUGS_DB, new_bugs.values())
+            db.append(BUGS_DB, new_bugs.values())


Test patch:
Repository: mozilla/bugbug

Issue:
In some features related to commits, ignore commits that have been backed out
For example, to count the number of lines modified by the commits of a bug, we only want to sum the lines of non-backed out commits.


Discussion:

Resolving patch:
diff --git a/bugbug/bug_features.py b/bugbug/bug_features.py
--- a/bugbug/bug_features.py
+++ b/bugbug/bug_features.py
@@ -228,27 +228,27 @@ def __call__(self, bug, **kwargs):
 
 class commit_added(object):
     def __call__(self, bug, **kwargs):
-        return sum(commit['added'] for commit in bug['commits'])
+        return sum(commit['added'] for commit in bug['commits'] if not commit['ever_backedout'])
 
 
 class commit_deleted(object):
     def __call__(self, bug, **kwargs):
-        return sum(commit['deleted'] for commit in bug['commits'])
+        return sum(commit['deleted'] for commit in bug['commits'] if not commit['ever_backedout'])
 
 
 class commit_types(object):
     def __call__(self, bug, **kwargs):
-        return sum((commit['types'] for commit in bug['commits']), [])
+        return sum((commit['types'] for commit in bug['commits'] if not commit['ever_backedout']), [])
 
 
 class commit_files_modified_num(object):
     def __call__(self, bug, **kwargs):
-        return sum(commit['files_modified_num'] for commit in bug['commits'])
+        return sum(commit['files_modified_num'] for commit in bug['commits'] if not commit['ever_backedout'])
 
 
 class commit_author_experience(object):
     def __call__(self, bug, **kwargs):
-        res = [commit['author_experience'] for commit in bug['commits']]
+        res = [commit['author_experience'] for commit in bug['commits'] if not commit['ever_backedout']]
         return sum(res) / len(res)
 
 
@@ -259,12 +259,12 @@ def __call__(self, bug, **kwargs):
 
 class components_touched(object):
     def __call__(self, bug, **kwargs):
-        return list(set(component for commit in bug['commits'] for component in commit['components']))
+        return list(set(component for commit in bug['commits'] for component in commit['components'] if not commit['ever_backedout']))
 
 
 class components_touched_num(object):
     def __call__(self, bug, **kwargs):
-        return len(set(component for commit in bug['commits'] for component in commit['components']))
+        return len(set(component for commit in bug['commits'] for component in commit['components'] if not commit['ever_backedout']))
 
 
 def cleanup_url(text):


Test patch:
Repository: mozilla/bugbug

Issue:
Author and reviewer experience should ignore backouts
Author and reviewer experience should not be increased for commits that are backed out.


Discussion:

Resolving patch:
diff --git a/bugbug/repository.py b/bugbug/repository.py
--- a/bugbug/repository.py
+++ b/bugbug/repository.py
@@ -139,7 +139,9 @@ def download_commits(repo_dir, date_from):
     global author_experience_90_days
     for commit in commits:
         author_experience[commit] = total_commits_by_author[commit.author]
-        total_commits_by_author[commit.author] += 1
+        # We don't want to consider backed out commits when calculating author/reviewer experience.
+        if not commit.ever_backedout:
+            total_commits_by_author[commit.author] += 1
 
         # Keep only the previous commits from a window of 90 days in the commits_by_author map.
         cut = None
@@ -155,7 +157,8 @@ def download_commits(repo_dir, date_from):
 
         author_experience_90_days[commit] = len(commits_by_author[commit.author])
 
-        commits_by_author[commit.author].append(commit)
+        if not commit.ever_backedout:
+            commits_by_author[commit.author].append(commit)
 
     global COMPONENTS
     r = requests.get('https://index.taskcluster.net/v1/task/gecko.v2.mozilla-central.latest.source.source-bugzilla-info/artifacts/public/components.json')


Test patch:
Repository: mozilla/bugbug

Issue:
In the DevDocNeeded model, consider bugs for which the dev-doc-needed keyword was removed as negative examples
See also #79.


Discussion:
I'm willing to take this up.

Resolving patch:
diff --git a/bugbug/models/devdocneeded.py b/bugbug/models/devdocneeded.py
--- a/bugbug/models/devdocneeded.py
+++ b/bugbug/models/devdocneeded.py
@@ -73,7 +73,14 @@ def get_labels(self):
 
             for entry in bug_data['history']:
                 for change in entry['changes']:
-                    if change['field_name'] == 'keywords' and any(keyword in change['added'] for keyword in ['dev-doc-needed', 'dev-doc-complete']):
+                    # Bugs that get dev-doc-needed removed from them at some point after it's been added (this suggests a false positive among human-analyzed bugs)
+                    if change['field_name'] == 'keywords' and 'dev-doc-needed' in change['removed'] and 'dev-doc-complete' not in change['added']:
+                        classes[bug_id] = 0
+                    # Bugs that go from dev-doc-needed to dev-doc-complete are guaranteed to be good
+                    # Bugs that go from not having dev-doc-needed to having dev-doc-complete are bugs
+                    # that were missed by previous scans through content but someone realized it
+                    # should have been flagged and updated the docs, found the docs already updated.
+                    elif change['field_name'] == 'keywords' and any(keyword in change['added'] for keyword in ['dev-doc-needed', 'dev-doc-complete']):
                         classes[bug_id] = 1
 
             if bug_id not in classes:


Test patch:
Repository: mozilla/bugbug

Issue:
Use 'product' and 'component' features in the models
b7369ea8bf282941ce4b378ad5ad3c832db20668 introduced the features, but we are still not using them.


Discussion:
@marco-c I will try this up. :)
@pradeepgangwar cool, thanks! I would do one PR at a time, one per model. When you make the PR, please write down both the results before adding the features and the results after adding the features.
@marco-c can i take this issue? I am contributing for the first time.
Sure. As described at https://github.com/mozilla/bugbug#contributing, any issue that is open and has no PR linked to it can be considered unassigned.
Hi @marco-c ,i,ve made some changes to use these features, how do i know if these features are being used now and i,ve done things correctly?
Train the model to which you have added the feature before and after adding the feature, and see if there's any change in the metrics.
Hey @marco-c ,  i checked the change in metrics for different models,and found significant change in only bug model.can i make PR?
Sure, but please attach here one text file per model showing your results before/after.
[bug_model.txt](https://github.com/mozilla/bugbug/files/2908732/bug_model.txt)
This is for bugmodel,like this for all?
[uplift_model.txt](https://github.com/mozilla/bugbug/files/2908775/uplift_model.txt)
For Uplift.
Yes, thanks!
[qaneeded_model.txt](https://github.com/mozilla/bugbug/files/2910937/qaneeded_model.txt)
Here is the 'qaneeded' model, no changes at all. I think the rest of the models don't need to use these features and some are already using. So should i only implement this for 'bug' model? 
And Thank you @marco-c  :)
OK, sounds good!

Resolving patch:
diff --git a/bugbug/models/bug.py b/bugbug/models/bug.py
--- a/bugbug/models/bug.py
+++ b/bugbug/models/bug.py
@@ -39,6 +39,8 @@ def __init__(self, lemmatization=False):
             bug_features.blocked_bugs_number(),
             bug_features.ever_affected(),
             bug_features.affected_then_unaffected(),
+            bug_features.product(),
+            bug_features.component(),
         ]
 
         cleanup_functions = [


Test patch:
Repository: mozilla/bugbug

Issue:
Add number of bug dependencies as a feature



Discussion:
I would like to tackle this! Should I just create another class in the bug_feature.py file? If I manage this I will solve similar problems as well. 
Correct!

Resolving patch:
diff --git a/bugbug/bug_features.py b/bugbug/bug_features.py
--- a/bugbug/bug_features.py
+++ b/bugbug/bug_features.py
@@ -64,6 +64,11 @@ def __call__(self, bug, **kwargs):
         return field(bug, 'severity')
 
 
+class number_of_bug_dependencies(object):
+    def __call__(self, bug, **kwargs):
+        return len(bug['depends_on'])
+
+
 class is_coverity_issue(object):
     def __call__(self, bug, **kwargs):
         return re.search('[CID ?[0-9]+]', bug['summary']) is not None or re.search('[CID ?[0-9]+]', bug['whiteboard']) is not None
diff --git a/bugbug/models/tracking.py b/bugbug/models/tracking.py
--- a/bugbug/models/tracking.py
+++ b/bugbug/models/tracking.py
@@ -45,6 +45,7 @@ def __init__(self, lemmatization=False):
             bug_features.comment_count(),
             bug_features.comment_length(),
             bug_features.reporter_experience(),
+            bug_features.number_of_bug_dependencies()
         ]
 
         cleanup_functions = [


Test patch:
Repository: mozilla/bugbug

Issue:
Use the bug snapshot transform in the "uplift" model
Depends on #5.


Discussion:
Hi.
I would like to work on this issue.
Previously I have not made any contributions to any open source community.
It would be great if you could help me get started.
To fix this issue, we'll need to do something similar to 2a440299d66d5a039bd683a353b80326dd5751df, but for the `uplift` model. We'd need to rollback to the time the uplift request was accepted or rejected.
@marco-c  How to test if i've made correct changes for this issue?
@chidauri make sure you can run the model, and write some tests similar to those in #189.
@marco-c The rollback should be done to the time when anything beginning with `approval-mozilla` was added to the change, right?
We have to rollback to the time when the uplift was accepted (approval-mozilla-XXX == +) or rejected (approval-mozilla-XXX == -), but we don't have to rollback to when the uplift was requested (approval-mozilla-XXX == ?).

Resolving patch:
diff --git a/bugbug/models/uplift.py b/bugbug/models/uplift.py
--- a/bugbug/models/uplift.py
+++ b/bugbug/models/uplift.py
@@ -43,7 +43,7 @@ def __init__(self, lemmatization=False):
         ]
 
         self.extraction_pipeline = Pipeline([
-            ('bug_extractor', bug_features.BugExtractor(feature_extractors, cleanup_functions)),
+            ('bug_extractor', bug_features.BugExtractor(feature_extractors, cleanup_functions, rollback=True, rollback_when=self.rollback)),
             ('union', ColumnTransformer([
                 ('data', DictVectorizer(), 'data'),
 
@@ -56,6 +56,9 @@ def __init__(self, lemmatization=False):
         self.clf = xgboost.XGBClassifier(n_jobs=16)
         self.clf.set_params(predictor='cpu_predictor')
 
+    def rollback(self, change):
+        return (change['field_name'] == 'flagtypes.name' and change['added'].startswith('approval-mozilla-') and (change['added'].endswith('+') or change['added'].endswith('-')))
+
     def get_labels(self):
         classes = {}
 


Test patch:
Repository: mozilla/bugbug

Issue:
Add assignee model to run.py
See also #61.


Discussion:

Resolving patch:
diff --git a/run.py b/run.py
--- a/run.py
+++ b/run.py
@@ -21,7 +21,7 @@
     parser.add_argument('--train', help='Perform training', action='store_true')
     parser.add_argument('--goal',
                         help='Goal of the classifier',
-                        choices=['bug', 'regression', 'tracking', 'qaneeded', 'uplift', 'component', 'devdocneeded', 'defectfeaturetask'],
+                        choices=['bug', 'regression', 'tracking', 'qaneeded', 'uplift', 'component', 'devdocneeded', 'defectfeaturetask', 'assignee'],
                         default='bug')
     parser.add_argument('--classifier', help='Type of the classifier', choices=['default', 'nn'], default='default')
     parser.add_argument('--classify', help='Perform evaluation', action='store_true')
@@ -61,6 +61,9 @@
     elif args.goal == 'devdocneeded':
         from bugbug.models.devdocneeded import DevDocNeededModel
         model_class = DevDocNeededModel
+    elif args.goal == 'assignee':
+        from bugbug.models.assignee import AssigneeModel
+        model_class = AssigneeModel
 
     if args.train:
         db.download()


Test patch:
Repository: mozilla/bugbug

Issue:
Add 'has bug ever had enhancement severity' as a feature
Especially useful for the bug and regression models.


Discussion:
Seems that I oversimplified the issue. :p Shouldn't it just check whether 'severity' field is equal to 'enhancement' or not? Or is all the difficulty in word "ever" - do we need to check all history of a bug for the 'severity' component?
The 'enhancement' severity has been removed lately, so yes, we need to check in the history :)
(but it is still easy enough)
Since going forward bugs won't have enhancement severity, we can only use this for archeology (we should probably have two different models, one to analyze old bugs, one to analyze new bugs).

Resolving patch:
diff --git a/bugbug/bug_features.py b/bugbug/bug_features.py
--- a/bugbug/bug_features.py
+++ b/bugbug/bug_features.py
@@ -287,6 +287,16 @@ def __call__(self, bug, author_ids, **kwargs):
         return bug_reporter()(bug).strip() in author_ids
 
 
+class had_severity_enhancement(object):
+    def __call__(self, bug, **kwargs):
+        for history in bug['history']:
+            for change in history['changes']:
+                if change['field_name'] == 'severity' and change['added'] == 'enhancement':
+                    return True
+
+        return False
+
+
 def cleanup_url(text):
     text = re.sub(r'http[s]?://(hg.mozilla|searchfox|dxr.mozilla)\S+', '__CODE_REFERENCE_URL__', text)
     return re.sub(r'http\S+', '__URL__', text)


Test patch: