Repository: mozilla/bugbug

Issue:
Detect bugs that impact MDN Web Docs aka "dev-doc-needed"
For the last decade or so, the MDN Web Docs team relies on people adding "dev-doc-needed" to Bugzilla bugs, so that we can document if Firefox implemented a new Web platform feature or if changes to Web platform features happened in Firefox.

It would be amazing if we wouldn't have to rely so much on humans adding this keyword anymore.
I don't know if this project can help with this, but filing this issue as an idea :)

Prior related bug about this https://bugzilla.mozilla.org/show_bug.cgi?id=1376393


Discussion:
The "positive" training data for this can be past bugs where somebody set "dev-doc-needed", and the "negative" training data can be past bugs where "dev-doc-needed" was never set. The "positive" training data should be pretty good, the "negative" one not so much (there could be bugs where documentation was needed, but for some reason nobody ever set "dev-doc-needed").
We can definitely try, building a simple model like the already existing ones should be easy enough.

We might have better results if we also include some VCS features (e.g. which kinds of files are modified).

That sounds fantastic! Note that for the positive training there is also "dev-doc-complete" (that means bugs where we finished the doc work).
The results are not too bad. The model has quite a bit of false positives (but this is partly to be expected, given that there could be bugs in the dataset which we should have documented but didn't), but it almost doesn't miss any bug.

If we set a confidence threshold to limit the false positives, out of 1933 bugs, 63 are suggested for dev-doc-needed (16 are actually dev-doc-needed), 1870 are not suggested for dev-doc-needed, 0 are dev-doc-needed but not suggested (so, all dev-doc-needed are suggested).

The top features the model looks at to decide if a bug should be dev-doc-needed:
1. 'comments__mozilla' (0.06737037003040314)
2. 'title__implement' (0.057237956672906876)
3. 'comments__spec' (0.04483399540185928)
4. 'data__patches' (0.03969449922442436)
5. 'data__keywords-site-compat=True' (0.037555694580078125)
6. 'comments____code_reference_url__' (0.03449741005897522)
7. 'comments__implement' (0.031212590634822845)
8. 'comments____url__' (0.03115002065896988)
9. 'data__landings' (0.02904007025063038)
10. 'comments__10' (0.027795393019914627)
11. 'comments__chrome' (0.02690601721405983)
12. 'comments__api' (0.02548757568001747)
13. 'comments__details' (0.02436738647520542)
14. 'comments__support' (0.016782119870185852)
15. 'comments__build' (0.016499998047947884)
16. 'comments__windows' (0.016358932480216026)
17. 'comments__implemented' (0.01507305447012186)
18. 'title__enable' (0.013755886815488338)
19. 'comments__regression' (0.013395934365689754)
20. 'title__intermittent' (0.013026916421949863)

To me, most of the features look reasonable.
N.B.: Some features are positive (pull the model towards thinking it is a dev-doc-needed bug), some are negative (pull the model towards thinking it is not a dev-doc-needed bug).

If you want, we could setup an automated email (daily? weekly?) with a list of the suggested bugs, so you can manually evaluate the results. Or, I could create a sheet with the results and you can take a look at them.
> If you want, we could setup an automated email (daily? weekly?) with a list of the suggested bugs, so you can manually evaluate the results. 

Yes, this sounds super useful! Weekly would be fine, I'm already getting component watching summary emails weekly.
Other factors that could be considered:

* Bugs that get dev-doc-needed removed from them at some point after it's been added (this suggests a false positive among human-analyzed bugs)
* Bugs that go from dev-doc-needed to dev-doc-complete are guaranteed to be good
* Bugs that go from not having dev-doc-needed to having dev-doc-complete are bugs that were missed by previous scans through content but someone realized it should have been flagged and updated the docs, found the docs already updated.
* I suspect the triage information can be made useful too. Once a bug is triaged and prioritized, that priority could potentially help to calculate the odds something needs to be documented.
* May be worth analyzing the documentation request issues/bugs as well, eventually.

This is a really clever project idea. I'm eager to see how it plays out. The idea of using ML to analyze bugs like this is a very good one!
> If you want, we could setup an automated email (daily? weekly?) with a list of the suggested bugs, so you can manually evaluate the results. Or, I could create a sheet with the results and you can take a look at them.

I'd like this too.

Resolving patch:
diff --git a/bugbug/models/devdocneeded.py b/bugbug/models/devdocneeded.py
new file mode 100644
--- /dev/null
+++ b/bugbug/models/devdocneeded.py
@@ -0,0 +1,82 @@
+# -*- coding: utf-8 -*-
+# This Source Code Form is subject to the terms of the Mozilla Public
+# License, v. 2.0. If a copy of the MPL was not distributed with this file,
+# You can obtain one at http://mozilla.org/MPL/2.0/.
+
+import xgboost
+from sklearn.compose import ColumnTransformer
+from sklearn.feature_extraction import DictVectorizer
+from sklearn.pipeline import Pipeline
+
+from bugbug import bug_features
+from bugbug import bugzilla
+from bugbug.model import Model
+
+
+class DevDocNeededModel(Model):
+    def __init__(self, lemmatization=False):
+        Model.__init__(self, lemmatization)
+
+        feature_extractors = [
+            bug_features.has_str(),
+            bug_features.has_regression_range(),
+            bug_features.severity(),
+            bug_features.keywords({'dev-doc-needed', 'dev-doc-complete'}),
+            bug_features.is_coverity_issue(),
+            bug_features.has_crash_signature(),
+            bug_features.has_url(),
+            bug_features.has_w3c_url(),
+            bug_features.has_github_url(),
+            bug_features.whiteboard(),
+            bug_features.patches(),
+            bug_features.landings(),
+            bug_features.title(),
+            bug_features.product(),
+            bug_features.component(),
+
+            bug_features.commit_added(),
+            bug_features.commit_deleted(),
+            bug_features.commit_types(),
+        ]
+
+        cleanup_functions = [
+            bug_features.cleanup_fileref,
+            bug_features.cleanup_url,
+            bug_features.cleanup_synonyms,
+        ]
+
+        self.extraction_pipeline = Pipeline([
+            ('bug_extractor', bug_features.BugExtractor(feature_extractors, cleanup_functions, rollback=True, rollback_when=self.rollback, commit_data=True)),
+            ('union', ColumnTransformer([
+                ('data', DictVectorizer(), 'data'),
+
+                ('title', self.text_vectorizer(stop_words='english'), 'title'),
+
+                ('comments', self.text_vectorizer(stop_words='english'), 'comments'),
+            ])),
+        ])
+
+        self.clf = xgboost.XGBClassifier(n_jobs=16)
+        self.clf.set_params(predictor='cpu_predictor')
+
+    def rollback(self, change):
+        return change['field_name'] == 'keywords' and any(keyword in change['added'] for keyword in ['dev-doc-needed', 'dev-doc-complete'])
+
+    def get_labels(self):
+        classes = {}
+
+        for bug_data in bugzilla.get_bugs():
+            bug_id = int(bug_data['id'])
+
+            for entry in bug_data['history']:
+                for change in entry['changes']:
+                    if change['field_name'] == 'keywords' and any(keyword in change['added'] for keyword in ['dev-doc-needed', 'dev-doc-complete']):
+                        classes[bug_id] = 1
+
+            if bug_id not in classes:
+                classes[bug_id] = 0
+
+        return classes
+
+    def get_feature_names(self):
+        return self.extraction_pipeline.named_steps['union'].get_feature_names()
diff --git a/run.py b/run.py
--- a/run.py
+++ b/run.py
@@ -15,7 +15,7 @@
     parser = argparse.ArgumentParser()
     parser.add_argument('--lemmatization', help='Perform lemmatization (using spaCy)', action='store_true')
     parser.add_argument('--train', help='Perform training', action='store_true')
-    parser.add_argument('--goal', help='Goal of the classifier', choices=['bug', 'regression', 'tracking', 'qaneeded', 'uplift', 'component'], default='bug')
+    parser.add_argument('--goal', help='Goal of the classifier', choices=['bug', 'regression', 'tracking', 'qaneeded', 'uplift', 'component', 'devdocneeded'], default='bug')
     parser.add_argument('--classify', help='Perform evaluation', action='store_true')
     args = parser.parse_args()
 
@@ -39,6 +39,9 @@
     elif args.goal == 'component':
         from bugbug.models.component import ComponentModel
         model_class = ComponentModel
+    elif args.goal == 'devdocneeded':
+        from bugbug.models.devdocneeded import DevDocNeededModel
+        model_class = DevDocNeededModel
 
     if args.train:
         db.download()


Test patch: