Repository: vmware/tern
Pull request #606 (2020-03-25 08:57:17)

Find Git Project URL
Adds git project url to the comments.

In tern/analyze/common.py
Adds function find_git_url which parses the output of 'git remote
-v' to get all the fecth labels containing a http or https url and
returns them as string.

Add test for this function.

Works towards #585.
Signed-off-by: WangJL <hazard15020@gmail.com>

rnjudge: Hi @ForgetMe17 -- Thanks for your PR. The purpose of issue #585 is to enhance the Dockerfile lock capability, which means that somehow we need to update the dockerfile object that we print out when the user calls the `tern lock` command. While your PR creates a function to find the git URL, it doesn't actually append it as a comment to the dockerfile object `ADD` line. Because you have already done the hard part of finding the URL, updating the dfobj is just a matter of calling your `get_git_url` function in the already existing `expand_add_command` or in the `find_git_info` function (your choice), both in `Dockerfile.py` and appending the found URL to the comment line.
ForgetMe17: Thanks for your reviews!
rnjudge: > @rnjudge Can you look this over? It looks good to me.

I had two small nits -- one question for Nisha and one change request. Other than that looks good. Thank you very much @ForgetMe17!

Patch:
diff --git a/docs/adding-to-command-library.md b/docs/adding-to-command-library.md
index 0d7e37f..30dc6ec 100644
--- a/docs/adding-to-command-library.md
+++ b/docs/adding-to-command-library.md
@@ -78,7 +78,7 @@ OS container, we can run:
 $ dpkg --get-selections | cut -f1 -d':' | awk '{print $1}'
 ```
 which is what we would normally do on a regular Debian OS (granted there is some
-fenagling of the output to just print out the names but hey, who hasn't done fancy
+finagling of the output to just print out the names but hey, who hasn't done fancy
 shell-fu to get a desired output? No joke - I found this on stackoverflow).
 
 You can try this yourself on a debian:jessie container:
diff --git a/docs/architecture.md b/docs/architecture.md
index 0fd27d2..df08015 100644
--- a/docs/architecture.md
+++ b/docs/architecture.md
@@ -26,7 +26,7 @@ This is the database where filesystem identifiers can be queried against to retr
 ### The Command Library
 This is a database of shell commands that may be used to create a container's layer filesystem. There are two types of shell commands - one for system wide package managers and one for custom shell commands or install scripts. The library is split in this way to account for situations where whole root filesystems are imported in order to create a new container.
 
-For example, in [this Dockerfile sample](../samples/debian_vim/Dockerfile), the use of FROM debian:jessie imports the base debian:jessie image from Dockerhub. This image is in turn created with a [Dockerfile](https://github.com/debuerreotype/docker-debian-artifacts/blob/b024a792c752a5c6ccc422152ab0fd7197ae8860/jessie/Dockerfile) that has the FROM scratch directive and adds an entire root filesystem created from an external build an release pipeline not related to Docker. The base library is meant to keep track of such images. On the other hand, the snippets library is used to keep track of standalone shell commands that can be used to install packages, once you hava a base image to work on.
+For example, in [this Dockerfile sample](../samples/debian_vim/Dockerfile), the use of FROM debian:jessie imports the base debian:jessie image from Dockerhub. This image is in turn created with a [Dockerfile](https://github.com/debuerreotype/docker-debian-artifacts/blob/b024a792c752a5c6ccc422152ab0fd7197ae8860/jessie/Dockerfile) that has the FROM scratch directive and adds an entire root filesystem created from an external build an release pipeline not related to Docker. The base library is meant to keep track of such images. On the other hand, the snippets library is used to keep track of standalone shell commands that can be used to install packages, once you have a base image to work on.
 
 Some acknowledgement should be made here that this is not the only way to create a container. However, it seems to be the most prevalent way right now. In the future, the community may move away from the Copy on Write filesystems and instead use one filesystem. If this were the case, Tern's job might be a bit easier, but this is just speculation.
 
@@ -43,7 +43,7 @@ The analyzer will collate the metadata it can get in Image objects which encapsu
 Tern's main purpose is to produce reports, either as an aid for understanding a container image or as a manifest for something else to consume. The default is the verbose report explaining where in the container the list of packages came from and what commands created them. The type of reports supported can be found in the project README. Also, take a look at the [custom report format creation process](./creating-custom-templates.md) to get a better understanding of how Tern supports multiple formats.
 
 ## Objects
-Tern uses Object Oriented Programming concepts to encapsulate the data that it will be referencing during thr course of execution. They can be found in the `tern/classes` directory. The general format is that an object of type Image contains a list of type ImageLayer and each ImageLayer contains a list of type Package. On top of that each of those objects contain an object of type Origins. The Origins object contains a list of type NoticeOrigin which contains a list of type Notice.
+Tern uses Object Oriented Programming concepts to encapsulate the data that it will be referencing during the course of execution. They can be found in the `tern/classes` directory. The general format is that an object of type Image contains a list of type ImageLayer and each ImageLayer contains a list of type Package. On top of that each of those objects contain an object of type Origins. The Origins object contains a list of type NoticeOrigin which contains a list of type Notice.
 
 A Notice object is where notice messages and the notice levels are recorded. Right now there are 4 notice levels: 'error', 'warning', 'hint' and 'info'. The 'hint' is not so much as to indicate the gravity of the notice but to inform the user on the best way to remove the notice.
 
diff --git a/docs/data-model.md b/docs/data-model.md
index a00d559..b90d7da 100644
--- a/docs/data-model.md
+++ b/docs/data-model.md
@@ -4,7 +4,7 @@ Tern stores metadata about the image and messages during operation in objects de
 
 ![Tern data model](./img/tern_data_model.png)
 
-The main class is `Image` and its derived classes. This class contains a list of type `ImageLayer`. `ImageLayer` contains a list of type `Package`. `Image`, `ImageLayer` and `Package` contain a property called `origins` which is an object of type `Origins`. This class is used to record notes while Tern operates on an image such as what tools were used to retrieve the metadata or if the filesystem is of unknown content. `Origins` contains a list of type `NoticeOrigin` which contains a string and a list of type `Notice`. The `Notice` objects are where messages get recorded. You can easily record a message in the `origins` property of the `Image`, `ImageLayer` and `Package` types of objects by using the `add_notice_to_origins` method which just adds a `Notice` object to the `NoticeOrigin` object containing the origin string you give it ("origin_str" is basically a string indicating where in the image or analysis stage an event that you want recorded occured).
+The main class is `Image` and its derived classes. This class contains a list of type `ImageLayer`. `ImageLayer` contains a list of type `Package`. `Image`, `ImageLayer` and `Package` contain a property called `origins` which is an object of type `Origins`. This class is used to record notes while Tern operates on an image such as what tools were used to retrieve the metadata or if the filesystem is of unknown content. `Origins` contains a list of type `NoticeOrigin` which contains a string and a list of type `Notice`. The `Notice` objects are where messages get recorded. You can easily record a message in the `origins` property of the `Image`, `ImageLayer` and `Package` types of objects by using the `add_notice_to_origins` method which just adds a `Notice` object to the `NoticeOrigin` object containing the origin string you give it ("origin_str" is basically a string indicating where in the image or analysis stage an event that you want recorded occurred).
 
 You will also see a class called `Template`. This is an abstract base class used to make custom formats. To learn more see the [documentation on creating custom formats](./creating-custom-templates.md).
 
diff --git a/docs/faq.md b/docs/faq.md
index 5e46abc..e70601e 100644
--- a/docs/faq.md
+++ b/docs/faq.md
@@ -1,12 +1,12 @@
 # FAQ (Work in Progress)
 
 ## Why Tern?
-Tern was created to help developers meet open source compliance requirements for containers. Open source software compliance is a hard problem in general but it gets harder with containers due to the ability to reuse diff filesystems. How those filesystems were created is still an ad hoc process. If you happen to have an lwn subscription you can read an article about it [here](https://lwn.net/Articles/752982/).
+Tern was created to help developers meet open source compliance requirements for containers. Open source software compliance is a hard problem in general but it gets harder with containers due to the ability to reuse diff filesystems. How those filesystems were created is still an ad hoc process. If you happen to have a LWN subscription you can read an article about it [here](https://lwn.net/Articles/752982/).
 
 The first step in meeting open source compliance obligations is knowing your container's Bill of Materials or BoM. This knowledge gives you some added benefits like debugging integration and build errors and identifying vulnerable packages in your running containers.
 
 ## Why not filesystem scanning?
-Static analysis is a reasonable approach to find software components and there are tools like [clair](https://github.com/coreos/clair) that create a BoM as part of vulnerability scanning. Some things to consider when using static analysis is the number of false positives that are detected, the time it takes to scan numerous files (some of which may not even be needed for an application to work) and the reliance on data that may not be open sourced. Tern is not meant to be a replacement for static analysis but simply a tool that automates some of the methods that developers and sysadmins use anyway.
+Static analysis is a reasonable approach to find software components and there are tools like [clair](https://github.com/coreos/clair) that create a BoM as part of vulnerability scanning. Some things to consider when using static analysis are the number of false positives that are detected, the time it takes to scan numerous files (some of which may not even be needed for an application to work) and the reliance on data that may not be open sourced. Tern is not meant to be a replacement for static analysis but simply a tool that automates some of the methods that developers and sysadmins use anyway.
 
 ## Why Python?
 Python is well suited for easy string formatting which is most of the work that Tern does.
diff --git a/tern/analyze/common.py b/tern/analyze/common.py
index 84db2ca..c5f1517 100644
--- a/tern/analyze/common.py
+++ b/tern/analyze/common.py
@@ -563,9 +563,54 @@ def get_git_sha(path_to_toplevel):
     return sha_info
 
 
+def get_git_url(dockerfile_path):
+    '''Given a dockerfile_path, return url of git project which contains
+    the dockerfile in a form of list.'''
+    # get the path of the folder containing the dockerfile
+    dockerfile_folder_path = os.path.dirname(dockerfile_path)
+    command = ['git', 'remote', '-v']
+    try:
+        output = subprocess.check_output(  # nosec
+            command, stderr=subprocess.DEVNULL, cwd=dockerfile_folder_path)
+        if isinstance(output, bytes):
+            lines = output.decode('utf-8').split('\n')
+            # pop the last line which is an empty line
+            lines.pop()
+            url_list = set()
+            for line in lines:
+                extract_url = extract_git_url_from_line(line)
+                if extract_url:
+                    url_list.add(extract_url)
+    except subprocess.CalledProcessError:
+        logger.debug("Cannot find git repo url, path is %s",
+                     dockerfile_folder_path)
+    return url_list
+
+
+def extract_git_url_from_line(line):
+    '''Given a line of git remote -v output, parse the url
+    line structure is '<label>\t<url> (fetch)'''
+    split_line = line.split(' ')
+    extract_url = ''
+    # use fetch url
+    if split_line[1] == '(fetch)':
+        split_line = split_line[0].split('\t')
+        # use https or git@ type
+        full_url = split_line[1]
+        if full_url.startswith('https://'):
+            extract_url = full_url.lstrip('https://')
+        elif full_url.startswith('http://'):
+            extract_url = full_url.lstrip('http://')
+        elif full_url.startswith('git'):
+            extract_url = full_url.replace('git@github.com:', 'github.com/', 1)
+        extract_url = extract_url.rstrip('.git')
+    return extract_url
+
+
 def get_git_toplevel(path):
-    '''Give a path, return the absolute path to the top level directory if it
-    is in a git repository. Empty string will be returned if not.'''
+    '''Given a path, return the absolute path to the top level directory if it
+    is in a git repository. Empty string will be returned if not.
+    Path should be a path to a directory not to a file.'''
     command = ['git', 'rev-parse', '--show-toplevel']
     path_to_toplevel = ''
     try:
diff --git a/tern/analyze/docker/dockerfile.py b/tern/analyze/docker/dockerfile.py
index fdc65b6..c23be12 100644
--- a/tern/analyze/docker/dockerfile.py
+++ b/tern/analyze/docker/dockerfile.py
@@ -254,6 +254,11 @@ def find_git_info(line, dockerfile_path):
     logger.debug('Parsed src_path is %s', src_path)
     # get the git project info
     comment_line = common.check_git_src(dockerfile_path)
+    # get the git project link
+    url_list = common.get_git_url(dockerfile_path)
+    if url_list:
+        comment_url = ', '.join(url_list)
+        comment_line += ', project url(s): ' + comment_url
     return comment_line
 
 


Test patch:
diff --git a/tests/test_analyze_common.py b/tests/test_analyze_common.py
index 10ea86c..e9397cc 100644
--- a/tests/test_analyze_common.py
+++ b/tests/test_analyze_common.py
@@ -5,6 +5,7 @@
 
 import unittest
 from unittest.mock import Mock
+import re
 
 from tern.analyze import common
 from test_fixtures import TestImage
@@ -21,12 +22,14 @@ class TestAnalyzeCommon(unittest.TestCase):
         self.image = TestImage('5678efgh')
         self.file = FileData('README.txt', '/home/test')
         cache.cache = {}
+        self.test_dockerfile = 'tests/dockerfiles/buildpack_deps_jessie_curl'
 
     def tearDown(self):
         del self.image
         del self.command1
         del self.command2
         cache.cache = {}
+        del self.test_dockerfile
 
     def testGetShellCommands(self):
         command = common.get_shell_commands("yum install nfs-utils")
@@ -248,6 +251,19 @@ class TestAnalyzeCommon(unittest.TestCase):
         self.assertEqual(
             len(common.get_installed_package_names(self.command2)), 0)
 
+    def testGetGitURL(self):
+        ''' we check the url to be the following form:
+        github.com/<username>/tern'''
+        url_list = common.get_git_url(self.test_dockerfile)
+        check_num = len(url_list)
+        pass_num = 0
+        git_username_reg = r'([a-zA-Z\d_-]{0,38})'
+        pattern = re.compile(r'github.com/'+git_username_reg+r'/tern')
+        for url in url_list:
+            if pattern.match(url):
+                pass_num += 1
+        self.assertEqual(pass_num, check_num)
+
 
 if __name__ == '__main__':
     unittest.main()