Repository: ishepard/pydriller
Pull request #47 (2019-06-10 09:20:26)

Use git hyper-blame, if available, to improve SZZ results

lgtm-com[bot]: This pull request **introduces 2 alerts** when merging f6a0bf6eeba121d522cd97d7997665ec5e9f690c into cea1795876d048d4a87deae4853d1a621017b7f9 - [view on LGTM.com](https://lgtm.com/projects/g/ishepard/pydriller/rev/pr-83bf881600614600bd1865e3e9b0c85a83b61a18)

**new alerts:**

* 1 for Unused local variable
* 1 for Except block handles &#39;BaseException&#39;
lgtm-com[bot]: This pull request **introduces 1 alert** when merging 1cd6e484e896a5f9395d286f45f7a35802f1c843 into cea1795876d048d4a87deae4853d1a621017b7f9 - [view on LGTM.com](https://lgtm.com/projects/g/ishepard/pydriller/rev/pr-53783443c258af61122dc25bb10c15ee3cc716e7)

**new alerts:**

* 1 for Unused local variable
marco-c: Getting tests to pass on Travis and Appveyor is proving a pain :P
marco-c: Ah, the issue is simply that depot_tools still requires Python 2... I didn't notice that.
If we want to test this on Travis, we need to install Python 2 too just for depot_tools, and keep testing with Python3.
ishepard: I am gonna have a look at it tomorrow morning, let’s see if I can help somehow 😁
marco-c: > I am gonna have a look at it tomorrow morning, let’s see if I can help somehow grin

Thanks, I'm too tired to try again today ;)

I think the solution is "simply" to install Python 2 alongside Python 3, keep running tests with Python 3 but run depot_tools with Python 2. The current code is trying to do that by installing python 2 with apt-get and creating a symlink so that `python` -> `python2`, but Travis is hanging when running `python --version`.
ishepard: @marco-c, can you grant me access to your fork repository? So I can commit directly in the PR as well 
marco-c: Done!
ishepard: So, this approach doesn't work if the user uses `virtualenv`, like in my case. So, what I am doing now, is taking the high road: I took the hyper-blame command of chromium and I am making it compatible for python3. 
The disadvantage is that we will create a new file inside Pydriller just for this command, and I will have to learn how it works (kind of at least). 
The advantage is that once it works, it will work for tests, production and everything. 

marco-c: >  The disadvantage is that we will create a new file inside Pydriller just for this command, and I will have to learn how it works (kind of at least).

The other disadvantage is that we need to sync them (in case they fix an issue, we need to cherry-pick the fix too). It doesn't seem to be updated so often though, so maybe it won't be a big issue.

I had another idea that we could try to force the script to run with Python 2.
After cloning the repo, replace `#!/usr/bin/env python` with `#!/usr/bin/env python2` in the `git_hyper_blame.py`.
ishepard: Good catch. My solution of "creating" hyper-blame from scratch is not impossible, but for sure requires a lot of time and it's gonna be error-prone at the beginning. I am gonna insert it in my TODO list though! 

Right now, we can apply your suggestion and it will work 👍 
marco-c: It worked with Python 3.7 and 3.6, that's a start ;)
marco-c: The issue on 3.5 and 3.4 is much easier to solve, I just need to use tmpdir_factory instead of tmp_path_factory, as the latter is not available.
codecov[bot]: # [Codecov](https://codecov.io/gh/ishepard/pydriller/pull/47?src=pr&el=h1) Report
> Merging [#47](https://codecov.io/gh/ishepard/pydriller/pull/47?src=pr&el=desc) into [master](https://codecov.io/gh/ishepard/pydriller/commit/cea1795876d048d4a87deae4853d1a621017b7f9?src=pr&el=desc) will **decrease** coverage by `50.46%`.
> The diff coverage is `88.88%`.

[![Impacted file tree graph](https://codecov.io/gh/ishepard/pydriller/pull/47/graphs/tree.svg?width=650&token=hDdmGET6SJ&height=150&src=pr)](https://codecov.io/gh/ishepard/pydriller/pull/47?src=pr&el=tree)

```diff
@@             Coverage Diff             @@
##           master      #47       +/-   ##
===========================================
- Coverage   99.39%   48.92%   -50.47%     
===========================================
  Files           4        4               
  Lines         494      511       +17     
===========================================
- Hits          491      250      -241     
- Misses          3      261      +258
```


| [Impacted Files](https://codecov.io/gh/ishepard/pydriller/pull/47?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pydriller/git\_repository.py](https://codecov.io/gh/ishepard/pydriller/pull/47/diff?src=pr&el=tree#diff-cHlkcmlsbGVyL2dpdF9yZXBvc2l0b3J5LnB5) | `64.9% <88.88%> (-35.1%)` | :arrow_down: |
| [pydriller/repository\_mining.py](https://codecov.io/gh/ishepard/pydriller/pull/47/diff?src=pr&el=tree#diff-cHlkcmlsbGVyL3JlcG9zaXRvcnlfbWluaW5nLnB5) | `17.64% <0%> (-82.36%)` | :arrow_down: |
| [pydriller/domain/developer.py](https://codecov.io/gh/ishepard/pydriller/pull/47/diff?src=pr&el=tree#diff-cHlkcmlsbGVyL2RvbWFpbi9kZXZlbG9wZXIucHk=) | `36.36% <0%> (-63.64%)` | :arrow_down: |
| [pydriller/domain/commit.py](https://codecov.io/gh/ishepard/pydriller/pull/47/diff?src=pr&el=tree#diff-cHlkcmlsbGVyL2RvbWFpbi9jb21taXQucHk=) | `58.21% <0%> (-40.38%)` | :arrow_down: |


Patch:
diff --git a/.travis.yml b/.travis.yml
index 602f6ef..0b11bb7 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -11,14 +11,14 @@ matrix:
       dist: xenial
       sudo: required
 before_install:
+  - sudo apt-get update
+  - sudo apt-get install -y python
   - unzip test-repos.zip
-  - cd test-repos
-  - git clone https://github.com/apache/hadoop.git
-  - cd ..
+  - git clone https://github.com/apache/hadoop.git test-repos/hadoop
 install:
   - pip install -r test-requirements.txt
   - pip install codecov pytest-cov
 script:
   - pytest tests/ --cov=pydriller/
 after_success:
-  - codecov
\ No newline at end of file
+  - codecov
diff --git a/pydriller/git_repository.py b/pydriller/git_repository.py
index af1a393..af91bbd 100644
--- a/pydriller/git_repository.py
+++ b/pydriller/git_repository.py
@@ -48,6 +48,19 @@ class GitRepository:
         self.main_branch = None
         self.lock = Lock()
 
+        # Try running 'git hyper-blame' on a file in the repo to check if the command is available.
+        try:
+            for f in os.listdir(path):
+                if f == '.git':
+                    continue
+
+                break
+
+            self.git.execute(["git", "hyper-blame", f])
+            self.hyper_blame_available = True
+        except GitCommandError as e:
+            self.hyper_blame_available = False
+
     @property
     def git(self):
         """
@@ -250,7 +263,8 @@ class GitRepository:
         return delete_line_number, additions_line_number
 
     def get_commits_last_modified_lines(self, commit: Commit,
-                                        modification: Modification = None) \
+                                        modification: Modification = None,
+                                        hashes_to_ignore_path: str = None) \
             -> Set[str]:
         """
         Given the Commit object, returns the set of commits that last
@@ -270,8 +284,13 @@ class GitRepository:
 
         :param Commit commit: the commit to analyze
         :param Modification modification: single modification to analyze
+        :param str hashes_to_ignore_path: path to a file containing hashes of
+               commits to ignore
         :return: the set containing all the bug inducing commits
         """
+        if hashes_to_ignore_path is not None:
+            assert self.hyper_blame_available, "Can't ignore hashes if hyper-blame is not available. Install it by cloning depot_tools and adding it to your PATH."
+
         buggy_commits = set()
 
         if modification is not None:
@@ -287,8 +306,14 @@ class GitRepository:
 
             deleted_lines = self.parse_diff(mod.diff)['deleted']
             try:
-                blame = self.git.blame('-w', commit.hash + '^',
-                                       '--', path).split('\n')
+                if not self.hyper_blame_available:
+                    blame = self.git.blame('-w', commit.hash + '^',
+                                           '--', path).split('\n')
+                else:
+                    cmd = ["git", "hyper-blame", commit.hash + '^', path]
+                    if hashes_to_ignore_path is not None:
+                        cmd.append("--ignore-file={}".format(hashes_to_ignore_path))
+                    blame = self.git.execute(cmd).split('\n')
                 for num_line, line in deleted_lines:
                     if not self._useless_line(line.strip()):
                         buggy_commit = blame[num_line - 1].split(' ')[


Test patch:
diff --git a/tests/test_git_repository.py b/tests/test_git_repository.py
index e7d262c..99ff750 100644
--- a/tests/test_git_repository.py
+++ b/tests/test_git_repository.py
@@ -11,10 +11,13 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
+import os
+import platform
 from datetime import datetime, timezone, timedelta
 from pathlib import Path
 
 import pytest
+from git import Repo
 
 from pydriller.domain.commit import ModificationType
 from pydriller.git_repository import GitRepository
@@ -437,3 +440,45 @@ def test_get_tagged_commits_wo_tags():
     tagged_commits = gr.get_tagged_commits()
 
     assert len(tagged_commits) == 0
+
+
+@pytest.fixture(scope="session")
+def depot_tools(tmpdir_factory):
+    gr = GitRepository('test-repos/test5/')
+    if not gr.hyper_blame_available:
+        tmpdir = tmpdir_factory.mktemp("depot_tools")
+        Repo.clone_from(url="https://chromium.googlesource.com/chromium/tools/depot_tools.git", to_path=tmpdir)
+
+        with open(os.path.join(str(tmpdir), 'git_hyper_blame.py'), 'r') as f:
+            git_hyper_blame_script = f.read()
+
+        with open(os.path.join(str(tmpdir), 'git_hyper_blame.py'), 'w') as f:
+            f.write(git_hyper_blame_script.replace('#!/usr/bin/env python', '#!/usr/bin/env python2'))
+
+        if platform.system() == "Windows":
+            os.environ["PATH"] = str(tmpdir) + os.pathsep + os.environ["PATH"]
+        else:
+            os.environ["PATH"] += os.pathsep + str(tmpdir)
+
+
+@pytest.mark.skipif(platform.system() == "Windows", reason="depot_tools is not easy to install on Windows CI")
+def test_get_commits_last_modified_lines_hyper_blame(depot_tools):
+    gr = GitRepository('test-repos/test5/')
+
+    buggy_commits = gr.get_commits_last_modified_lines(gr.get_commit('e6d3b38a9ef683e8184eac10a0471075c2808bbd'))
+
+    assert len(buggy_commits) == 1
+    assert '540c7f31c18664a38190fafb6721b5174ff4a166' in buggy_commits
+
+
+@pytest.mark.skipif(platform.system() == "Windows", reason="depot_tools is not easy to install on Windows CI")
+def test_get_commits_last_modified_lines_hyper_blame_ignore_hash(depot_tools, tmpdir):
+    with open(os.path.join(str(tmpdir), "ignore"), "w") as f:
+        f.write("540c7f31c18664a38190fafb6721b5174ff4a166")
+
+    gr = GitRepository('test-repos/test5/')
+
+    buggy_commits = gr.get_commits_last_modified_lines(gr.get_commit('e6d3b38a9ef683e8184eac10a0471075c2808bbd'), hashes_to_ignore_path=tmpdir / "ignore")
+
+    assert len(buggy_commits) == 1
+    assert '22505e97dca6f843549b3a484b3609be4e3acf17' in buggy_commits