Repository: public-apis/public-apis
Pull request: Add check for checking duplicated links in README file
Add extra check into validate_links.py file to check if there are some duplicated links in the README file

I create an ignored list for some duplicated links which are not related to any API link. There is this link `https://github.com/davemachado/public-api` in the README listed twice. Should we ignore it too?

Closes #1592



Problem statement:
Add extra validation to check if more than two links are the same
There should be a check in the validation to check if more than two links are the same. 
If they exist in the README the "merge" button should be blocked to merge the PR.

Hints from issue discussion:

Commit message: Add check for checking duplicated links in README file

Closes #1592

Patch to review:
diff --git a/build/validate_links.py b/build/validate_links.py
index ab36b0c840..c4eafce954 100755
--- a/build/validate_links.py
+++ b/build/validate_links.py
@@ -5,6 +5,11 @@
 import socket
 import sys
 
+ignored_links = [
+    'https://github.com/public-apis/public-apis/actions?query=workflow%3A%22Run+tests%22', 
+    'https://github.com/public-apis/public-apis/workflows/Validate%20links/badge.svg?branch=master', 
+    'https://github.com/public-apis/public-apis/actions?query=workflow%3A%22Validate+links%22',
+]
 
 def parse_links(filename):
     """Returns a list of URLs from text file"""
@@ -16,6 +21,29 @@ def parse_links(filename):
     links = [raw_link[0] for raw_link in raw_links]
     return links
 
+def dup_links(links):
+    """Check for duplicated links"""
+    print(f'Checking for duplicated links...')
+    hasError = False
+    seen = {}
+    dupes = []
+
+    for x in links:
+        if x in ignored_links:
+            continue
+
+        if x not in seen:
+            seen[x] = 1
+        else:
+            if seen[x] == 1:
+                dupes.append(x)
+
+    if not dupes:
+        print(f"No duplicated links")
+    else:
+        print(f"Found duplicated links: {dupes}")  
+        hasError = True  
+    return hasError
 
 def validate_links(links):
     """Checks each entry in JSON file for live link"""
@@ -58,6 +86,9 @@ def validate_links(links):
     if num_args < 2:
         print("No .md file passed")
         sys.exit(1)
-    hasError = validate_links(parse_links(sys.argv[1]))
+    links = parse_links(sys.argv[1])
+    hasError = dup_links(links)
+    if not hasError:
+        hasError = validate_links(links)
     if hasError:
         sys.exit(1)


Code review comments:
diff_hunk: @@ -16,6 +21,29 @@ def parse_links(filename):
     links = [raw_link[0] for raw_link in raw_links]
     return links
 
+def dup_links(links):
+    """Check for duplicated links"""
+    print(f'Checking for duplicated links...')
+    hasError = False
+    seen = {}
+    dupes = []
+
+    for x in links:
+        if x in ignored_links:
+            continue
+
+        if x not in seen:
+            seen[x] = 1
+        else:
+            if seen[x] == 1:
+                dupes.append(x)
path: build/validate_links.py
text: @user1:
Many URLs contain the `/` character at the end. This section does not consider them.

This is returning at the moment:
``` bash
Found duplicated links: ['https://github.com/davemachado/public-api']
```

If you consider the character `/` at the end:
``` bash
Found duplicated links: ['https://mailboxlayer.com', 'https://github.com/davemachado/public-api', 'https://languagelayer.com']
```

The `str.rstrip()` function solves this.

```suggestion
        x = x.rstrip('/')
        if x in ignored_links:
            continue

        if x not in seen:
            seen[x] = 1
        else:
            if seen[x] == 1:
                dupes.append(x)
```

@author:
@user1 Thanks for reviewing my PR and spotting an extra check with `/` character.

diff_hunk: @@ -16,6 +21,29 @@ def parse_links(filename):
     links = [raw_link[0] for raw_link in raw_links]
     return links
 
+def dup_links(links):
+    """Check for duplicated links"""
+    print(f'Checking for duplicated links...')
+    hasError = False
+    seen = {}
+    dupes = []
+
+    for x in links:
path: build/validate_links.py
text: @user1:
Isn't it better to make it explicit?
```suggestion
    for link in links:
```