Repository: mozilla/bugbug

Issue:
Create a classifier to detect bugs that need QA
Needed for https://github.com/mozilla/relman-auto-nag/issues/227.

To do this, we'll need to collect some labels.
We can automatically create some positive labels by getting bugs that have the `qawanted` keyword or that have `qe-verify` flag.
We can't automatically create negative labels because we can't be sure that QA was not needed when the keyword/flag was not set.


Discussion:
As a first implementation, we can just simplify and assume that bugs don't need QA when qawanted and qe-verify are not set.

In order to implement this, we'll need to add a new function in bugbug/labels.py to create the labels for this problem. Then, we need to add the classifier to the possible choices of 'goal' in `run.py`.

The label creation will be similar to `get_tracking_labels`. In this case though, we need to have `True` when the bug has had qawanted or qe-verify, and `False` otherwise.

https://github.com/marco-c/bugbug/issues/5 will be needed to fully fix this, but in the meantime we can just use the bug data and not the snapshot at the time of qawanted/qe-verify setting.
#22 was the first step here, adding a function to create the labels. Now we need to add some code to run.py to use it.
@marco-c  The `args.goal` in run.py would need to have a new QA option right? Also, are there some models present at the moment to test these changes? (Or would I have to train one myself?)
@ayush1999 yes, there should be a new `qaneeded` option. There isn't a model present yet, since you're just adding the code to train now :smile: 
#27 added the code to train.

The result seems too good to be true though, so we need to figure out if there is some feature that should be not considered in the 'qaneeded' case.
@marco-c Currently we are considering labels that have `qawanted` or `qe-verify` keyword. What should we be experimenting with to check the validity of the model? Should we be checking for a specific keyword that might cause the condition to be false? (So we'd have to remove that condition too)
It's possible that we are using those keywords as features, which clearly would make the problem too easy :smile: 
We have to remove these keywords from the training for this 'qaneeded' case. Currently it's not easy to do it, so this will have to wait for the refactoring from https://github.com/marco-c/bugbug/issues/6.
#6 is done, and c95a6f809d45991489591066ebf5ff0b86ff4d2f made it possible to make the feature extractors configurable.
For the QA wanted case, we should clearly remove `qawanted` from the keywords extractor.
Here's an example for the regression model: https://github.com/marco-c/bugbug/commit/e9d291aaae297c885688db7583a4bfc6200ede8f#diff-5f2351779b955b08e775eb4234c3f2ebR28.

We should do something similar for the qaneeded model, with the only difference being that in that case we want to ignore the `qawanted` keyword.

Resolving patch:
diff --git a/bugbug/models/qaneeded.py b/bugbug/models/qaneeded.py
--- a/bugbug/models/qaneeded.py
+++ b/bugbug/models/qaneeded.py
@@ -24,7 +24,7 @@ def __init__(self, lemmatization=False):
             bug_features.has_str(),
             bug_features.has_regression_range(),
             bug_features.severity(),
-            bug_features.keywords(),
+            bug_features.keywords(set(['qawanted'])),
             bug_features.is_coverity_issue(),
             bug_features.has_crash_signature(),
             bug_features.has_url(),


Test patch: