Repository: mozilla/bugbug

Issue:
Add number of previous commits touching the same modules as a feature
This is similar to https://github.com/mozilla/bugbug/commit/a002e0ce53080699a78cada08a95fd2355953c42#diff-fd96da16451dabb38a55be1721e8be9e, but for components.

It's a little harder than author or reviewer experience, as we need to parse the patch in order to get the components, and we're currently parsing the patches in multiple processes.


Discussion:
Now that we have added the components as a feature in #115, we don't need to parse the patch again, right?
So can it be done along similar lines as this: a002e0c?
> Now that we have added the components as a feature in #115, we don't need to parse the patch again, right?

It's still the same problem as before. We are parsing patches in multiple processes, so not in order. If you want to have a "component experience" variable, you need to do things in order and you need to have a shared memory between processes.

>  So can it be done along similar lines as this: a002e0c?

Yes, as I said earlier, it is similar to a002e0c in theory, but it is much harder.
Can we use the `Future` object from `concurrent.futures` so that processes wait for a particular process to finish first and then maybe use its result?
Sure, we can use Future, but how exactly do you want to use it to implement this?
I was thinking about using Future objects to check if a process has finished and then continue with the next process. This would require to `submit` each process to get the returned future instead of using `map`. But this again has the problem of ordering of the processes.

And I'm a little confused about this. If we want the processes to execute in order, won't that be against the idea of multiprocessing? As we want the processes to execute sequentially/in order but multiprocessing would execute multiple processes at the same time.
So how can we share some variable(returned by a process running along with multiple other processes) among different processes running(which need to modify and subsequently use that variable) at the same time?
Indeed, if we use Future like you proposed, we lose the advantage of multiprocessing. If so, then we just remove completely the multiprocessing, it doesn't make sense to spawn multiple processes if then you only run them one after the other.

We don't want to lose the advantage of multiprocessing (unless strictly necessary), so we need an alternative solution (this is why I say this issue is not so simple and requires some thinking).
Maybe we can use the `Condition` object from multiprocessing module to synchronize parts of the workflow.

In `_transform`, we can first call a function which would just parse components from the patch and update a global dictionary(while other processes wait over the Condition). This function can then `notify` the next process to execute.
Once we parse just the component data sequentially, we can execute the(remaining) `_transform` function in parallel for all the processes.

And we can use a `multiprocessing.Manager` proxy for sharing the data between different processes.
This solution would still make the most expensive part of the whole workflow synchronous, defeating the purpose of multiprocessing.
I think doing this inside the `_transform` function is a really difficult task.

Even if we try to execute `_transform` in parallel first (to do all the computationally expensive task) and `wait` just before it returns (and maybe then check the previous commits and return sequentially), then also we must have some id for each process(to know their order) as to which process was indeed the first one which might then signal the second process and so on.

Can we instead do this after the commits are returned by the executor.map?
Indeed, in the current architecture it would be really really hard to do it in _transform.

With a962e0694bef08090bd6c4810e5c4c770154f99c, which made the hg.log part faster, we can now just use the `files` hg template parameter, so we have this information directly at the beginning.

You can use `{join(files,"|")}` for files and `{join(file_copies,"|")}` for file copies (when a file is renamed, you'll need to move the "# prev commits" value from the old path to the new path).
For some reason, `repository.py` is not mining commits. It shows:

```
Mining 0 commits using 4 processes...
0it [00:00, ?it/s]
Analyzing 0 patches...
Mining commits using 4 processes...
0it [00:00, ?it/s]
```

As suggested here: https://github.com/mozilla/bugbug/issues/186#issuecomment-469019713, I tried running `hg pull central` and `hg up central`. 

But `hg up central` says:
`$ hg up central
abort: unknown revision 'central'!`
Please open a separate issue for this problem.
`{join(files,"|")}` should be added to `_hg_log` template right?

And can you please suggest how to access these files?
> {join(files,"|")} should be added to _hg_log template right?

Yes.

> And can you please suggest how to access these files?

Try to figure it out yourself, it's very easy. You are getting the log of a repository, where could the files be?
The files can be accessed using rev in `hglib.util.grouper`.
I was looking in the wrong place initially. 😅

Now that we have the files modified by a commit, a problem is that we have multiple files and so multiple components touched by a particular commit.
So do we need to add the # previous commits for all the components?
Yes, the sum.
Also, for `{join(file_copies,"|")}`, this is one of the results I got:

`tools/profiler/core/ProfiledThreadData.cpp (tools/profiler/core/ThreadInfo.cpp)`

In this, do I need to change the file before parenthesis in `{join(files,"|")}` with the one in parenthesis here? 
I don't remember which is the original and which is the copy, you'll have to check.

When there is a copy, you'll have to move the "# of past commits" to the new path.
With `{join(file_copies,"|")}`, if one of the files has been renamed, then it's info shouldn't be in the `COMPONENTS` dict, right? But the COMPONENTS dictionary shows components for both the files (original and the copy).

Is there any other way of checking for the copy?
Also, some files in `commit.files` give KeyError in `COMPONENTS` dictionary.

Example: this file `devtools/client/netmonitor/test/components/filter-buttons.test.js` shows KeyError even though there are no files in the `file_copies` (indicating that the file wasn't renamed) for that particular commit.

Is this behavior expected?
> With {join(file_copies,"|")}, if one of the files has been renamed, then it's info shouldn't be in the COMPONENTS dict, right? But the COMPONENTS dictionary shows components for both the files (original and the copy).

Can you show me an example? (path before, path after, changeset hash)

> Is there any other way of checking for the copy?

Why? Is there a problem with this way?

> Also, some files in commit.files give KeyError in COMPONENTS dictionary.
> Example: this file devtools/client/netmonitor/test/components/filter-buttons.test.js shows KeyError even though there are no files in the file_copies (indicating that the file wasn't renamed) for that particular commit.
> Is this behavior expected?

Yes, unfortunately COMPONENTS dictionary is built from the current state of the repository, so there might be files that were once present and later removed that are not in COMPONENTS.
> Why? Is there a problem with this way?

If both the files in `file_copies` are there in COMPONENTS, how would we know which one of them is the new file? Aren't the old files supposed to be gone from the COMPONENTS?
> Can you show me an example? (path before, path after, changeset hash)

For some very strange reason, I'm not able to mine that commit again which showed this behavior. But when I first tried this, I saved the files:

```
files => b'tools/profiler/core/ProfiledThreadData.cpp|tools/profiler/core/ProfiledThreadData.h|tools/profiler/core/RegisteredThread.cpp|tools/profiler/core/RegisteredThread.h|tools/profiler/core/ThreadInfo.cpp|tools/profiler/core/ThreadInfo.h|tools/profiler/core/platform-linux-android.cpp|tools/profiler/core/platform-macos.cpp|tools/profiler/core/platform-win32.cpp|tools/profiler/core/platform.cpp|tools/profiler/moz.build|tools/profiler/tests/gtest/ThreadProfileTest.cpp'


file_copies => b'tools/profiler/core/ProfiledThreadData.cpp (tools/profiler/core/ThreadInfo.cpp)|tools/profiler/core/ProfiledThreadData.h (tools/profiler/core/ThreadInfo.h)|tools/profiler/core/RegisteredThread.cpp (tools/profiler/core/ThreadInfo.cpp)|tools/profiler/core/RegisteredThread.h (tools/profiler/core/ThreadInfo.h)'
```
So unfortunately, I don't have the commit hash, but for these files in `file_copies`, both, the files in parenthesis and the file before them are there in COMPONENTS:

```
tools/profiler/core/ProfiledThreadData.h (tools/profiler/core/ThreadInfo.h)
tools/profiler/core/RegisteredThread.h (tools/profiler/core/ThreadInfo.h)
```

Also, files in `file_copies` when checked for a COMPONENT show different behavior:

```
Commit: 2f0342644d576df1d1c0bda70a629c019187e6f4:

Both these files in file_copies don't exist in COMPONENTS:
third_party/rust/rayon-0.8.2/src/iter/README.md (third_party/rust/rayon/src/iter/README.md)
```

```
Commit: 61b168663a54d4b5da46de34d318da43cc10a71c

the second file is not in COMPONENTS:
security/manager/tools/crtshToIdentifyingStruct/crtshToIdentifyingStruct.py (security/manager/tools/crtshToDNStruct/crtshToDNStruct.py)
```

```
Commit: dece9cd799b531d92fb7afaca547bc7d5d73fcf6

the second file is in the COMPONENTS but the first one is not:
security/certverifier/TrustOverride-AppleGoogleData.inc (security/certverifier/TrustOverride-AppleGoogleDigiCertData.inc)
```
> If both the files in file_copies are there in COMPONENTS, how would we know which one of them is the new file?

There is a rule. Either the first is the original and the second is the copy, or the other way around. Of course, it's always the same order in every commit, or file_copies would be useless. So, you just need to figure out for one example commit which order it is, and then you know.

> Aren't the old files supposed to be gone from the COMPONENTS?

People can remove and re-add files over time, so you can't expect that files that were moved three years ago can't reappear later.
> For some very strange reason, I'm not able to mine that commit again which showed this behavior

If you are not able to mine it again, maybe you made a mistake the first time?
> Also, files in file_copies when checked for a COMPONENT show different behavior:

This is not a problem.
> If you are not able to mine it again, maybe you made a mistake the first time?

This might be the case, I probably did something unusual the first time because I'm not able to see that commit again.


And from [here](https://stackoverflow.com/questions/16040503/mercurial-log-how-to-show-renames), it looks like the first file is the copy and the second one is original.
> And from here, it looks like the first file is the copy and the second one is original.

You can double check very easily, just take a commit where there is a copy, run `hg export -r COMMIT_HASH` and see which file was renamed.
Just checked using `hg export -r 61b168663a54d4b5da46de34d318da43cc10a71c`, the second file is renamed to the first file.

So the first file is the copy and the second one is the original.

Resolving patch:
diff --git a/bugbug/repository.py b/bugbug/repository.py
--- a/bugbug/repository.py
+++ b/bugbug/repository.py
@@ -23,13 +23,16 @@
 COMMITS_DB = 'data/commits.json'
 db.register(COMMITS_DB, 'https://www.dropbox.com/s/mz3afgncx0siijc/commits.json.xz?dl=1', 'v1')
 
-COMPONENTS = {}
+path_to_component = {}
 
-Commit = namedtuple('Commit', ['node', 'author', 'desc', 'date', 'bug', 'backedoutby', 'author_email'])
+Commit = namedtuple('Commit', ['node', 'author', 'desc', 'date', 'bug', 'backedoutby', 'author_email', 'files', 'file_copies'])
 
 author_experience = {}
 author_experience_90_days = {}
 
+components_touched_prev = defaultdict(int)
+components_touched_prev_90_days = defaultdict(int)
+
 
 def get_commits():
     return db.read(COMMITS_DB)
@@ -63,9 +66,11 @@ def _transform(commit):
         'files_modified_num': 0,
         'types': set(),
         'components': list(),
-        'author_experience': author_experience[commit],
-        'author_experience_90_days': author_experience_90_days[commit],
+        'author_experience': author_experience[commit.node],
+        'author_experience_90_days': author_experience_90_days[commit.node],
         'author_email': commit.author_email.decode('utf-8'),
+        'components_touched_prev': components_touched_prev[commit.node],
+        'components_touched_prev_90_days': components_touched_prev_90_days[commit.node],
     }
 
     patch = HG.export(revs=[commit.node], git=True)
@@ -103,13 +108,13 @@ def _transform(commit):
     # Covert to a list, as a set is not JSON-serializable.
     obj['types'] = list(obj['types'])
 
-    obj['components'] = list(set('::'.join(COMPONENTS[fl]) for fl in patch_data.keys() if COMPONENTS.get(fl)))
+    obj['components'] = list(set(path_to_component[path] for path in patch_data.keys() if path_to_component.get(path)))
 
     return obj
 
 
 def _hg_log(revs):
-    template = '{node}\\0{author}\\0{desc}\\0{date}\\0{bug}\\0{backedoutby}\\0{author|email}\\0'
+    template = '{node}\\0{author}\\0{desc}\\0{date}\\0{bug}\\0{backedoutby}\\0{author|email}\\0{join(files,"|")}\\0{join(file_copies,"|")}\\0'
 
     args = hglib.util.cmdbuilder(b'log', template=template, no_merges=True, rev=revs[0] + b':' + revs[-1], branch='central')
     x = HG.rawcommand(args)
@@ -120,6 +125,16 @@ def _hg_log(revs):
         posixtime = float(rev[3].split(b'.', 1)[0])
         dt = datetime.fromtimestamp(posixtime)
 
+        file_copies = {}
+        for file_copy in rev[8].decode('utf-8').split('|'):
+            if not file_copy:
+                continue
+
+            parts = file_copy.split(' (')
+            copied = parts[0]
+            orig = parts[1][:-1]
+            file_copies[orig] = copied
+
         revs.append(Commit(
             node=rev[0],
             author=rev[1],
@@ -128,6 +143,8 @@ def _hg_log(revs):
             bug=rev[4],
             backedoutby=rev[5],
             author_email=rev[6],
+            files=rev[7].decode('utf-8').split('|'),
+            file_copies=file_copies,
         ))
 
     return revs
@@ -189,7 +206,7 @@ def download_commits(repo_dir, date_from):
     global author_experience
     global author_experience_90_days
     for commit in commits:
-        author_experience[commit] = total_commits_by_author[commit.author]
+        author_experience[commit.node] = total_commits_by_author[commit.author]
         # We don't want to consider backed out commits when calculating author/reviewer experience.
         if not commit.backedoutby:
             total_commits_by_author[commit.author] += 1
@@ -206,15 +223,58 @@ def download_commits(repo_dir, date_from):
         if cut is not None:
             commits_by_author[commit.author] = commits_by_author[commit.author][cut + 1:]
 
-        author_experience_90_days[commit] = len(commits_by_author[commit.author])
+        author_experience_90_days[commit.node] = len(commits_by_author[commit.author])
 
         if not commit.backedoutby:
             commits_by_author[commit.author].append(commit)
 
-    global COMPONENTS
+    global path_to_component
     r = requests.get('https://index.taskcluster.net/v1/task/gecko.v2.mozilla-central.latest.source.source-bugzilla-info/artifacts/public/components.json')
     r.raise_for_status()
-    COMPONENTS = r.json()
+    path_to_component = r.json()
+    path_to_component = {path: '::'.join(component) for path, component in path_to_component.items()}
+
+    global components_touched_prev
+    global components_touched_prev_90_days
+
+    components_touched = defaultdict(int)
+    prev_commits_90_days = []
+    for commit in commits:
+        components = set(path_to_component[path] for path in commit.files if path in path_to_component)
+
+        for component in components:
+            components_touched_prev[commit.node] += components_touched[component]
+
+            components_touched[component] += 1
+
+        if len(commit.file_copies) > 0:
+            for orig, copied in commit.file_copies.items():
+                if orig in path_to_component and copied in path_to_component:
+                    components_touched[path_to_component[copied]] = components_touched[path_to_component[orig]]
+
+        for i, prev_commit in enumerate(prev_commits_90_days):
+            if (commit.date - prev_commit.date).days <= 90:
+                break
+
+            cut = i
+
+        if cut is not None:
+            prev_commits_90_days = prev_commits_90_days[cut + 1:]
+
+        components_touched_90_days = defaultdict(int)
+        for prev_commit in prev_commits_90_days:
+            components_prev = set(path_to_component[path] for path in prev_commit.files if path in path_to_component)
+
+            for component_prev in components_prev:
+                components_touched_90_days[component_prev] += 1
+
+            if len(prev_commit.file_copies) > 0:
+                for orig, copied in prev_commit.file_copies.items():
+                    if orig in path_to_component and copied in path_to_component:
+                        components_touched_90_days[path_to_component[copied]] = components_touched_90_days[path_to_component[orig]]
+
+        components_touched_prev_90_days[commit.node] = sum(components_touched_90_days[component] for component in components)
+        prev_commits_90_days.append(commit)
 
     print(f'Mining commits using {multiprocessing.cpu_count()} processes...')
 


Test patch: