Repository: spotdl/spotify-downloader
Issue: Having duplicates in track file messes up deletion of tracks from track file

- [x] Using latest version as provided on the [master branch](https://github.com/ritiek/spotify-downloader/tree/master)
- [x] [Searched](https://github.com/ritiek/spotify-downloader/issues?utf8=%E2%9C%93&q=is%3Aissue) for similar issues including closed ones

#### What is the purpose of your *issue*?
- [x] Bug
- [ ] Feature Request
- [ ] Question
- [ ] Other

### Description
As the title says. Say we have a `track_file.txt`with (duplicate) tracks as below:
```
https://open.spotify.com/track/2DGa7iaidT5s0qnINlwMjJ
https://open.spotify.com/track/2DGa7iaidT5s0qnINlwMjJ
https://open.spotify.com/track/3SipFlNddvL0XNZRLXvdZD
```

On downloading these tracks we get:
```
$ spotdl -l=track_file.txt -d 
INFO: Preparing to download 2 songs

INFO: 1. Alan Walker ‒ The Spectre (Lyrics / Lyrics Video)[HD] (http://www.youtube.com/watch?v=8lPa0963LEc)

INFO: 2. Janji - Heroes Tonight (feat. Johnning) [Lyrics] (http://www.youtube.com/watch?v=7bI36Iu6M6s)
```

```
$ cat track_file.txt
https://open.spotify.com/track/3SipFlNddvL0XNZRLXvdZD
```

spotdl internally removes the duplicates from the track file but does not rewrite the unique tracks back to `track_file.txt`. So, when it successfully downloads the third track from the file in the above case, it deletes the second track instead of the third track and the third track lives on in `track_file.txt`

I can think of three solutions here:

- Fix this by filtering duplicates from track file and re-writing track file with unique tracks so that tracks from track file are deleted in proper order.

- Initially creating a new list of duplicate tracks in file and then take extra care to skip this track and remove it from file if it has been already downloaded once.

- Do nothing and let the script figure out on its own going with the `--overwrite` argument when the same track is being downloaded for the second time.

Changed files:
spotdl/spotdl.py
test/test_internals.py
spotdl/internals.py

diff --git a/spotdl/internals.py b/spotdl/internals.py
index ca8b6e44..76fc6d9d 100755
--- a/spotdl/internals.py
+++ b/spotdl/internals.py
@@ -158,6 +158,23 @@ def get_splits(url):
     return splits
 
 
+def get_unique_tracks(text_file):
+    """
+    Returns a list of unique tracks given a path to a
+    file containing tracks.
+    """
+
+    with open(text_file, 'r') as listed:
+        # Read tracks into a list and remove any duplicates
+        lines = listed.read().splitlines()
+
+    # Remove blank and strip whitespaces from lines (if any)
+    lines = [line.strip() for line in lines if line.strip()]
+    lines = remove_duplicates(lines)
+
+    return lines
+
+
 # a hacky way to user's localized music directory
 # (thanks @linusg, issue #203)
 def get_music_dir():
diff --git a/spotdl/spotdl.py b/spotdl/spotdl.py
index dc66fe70..b9cfbc03 100755
--- a/spotdl/spotdl.py
+++ b/spotdl/spotdl.py
@@ -65,15 +65,13 @@ def check_exists(music_file, raw_song, meta_tags):
 
 def download_list(text_file):
     """ Download all songs from the list. """
-    with open(text_file, 'r') as listed:
-        # read tracks into a list and remove any duplicates
-        lines = listed.read().splitlines()
-        lines = internals.remove_duplicates(lines)
-    # ignore blank lines in text_file (if any)
-    try:
-        lines.remove('')
-    except ValueError:
-        pass
+
+    log.info('Checking and removing any duplicate tracks')
+    lines = internals.get_unique_tracks(text_file)
+
+    # override file with unique tracks
+    with open(text_file, 'w') as listed:
+        listed.write('\\n'.join(lines))
 
     log.info(u'Preparing to download {} songs'.format(len(lines)))
     downloaded_songs = []
diff --git a/test/test_internals.py b/test/test_internals.py
index 89e63808..2fcd1c8f 100644
--- a/test/test_internals.py
+++ b/test/test_internals.py
@@ -80,3 +80,39 @@ class TestGetSeconds:
             internals.get_sec('10*05')
         with pytest.raises(ValueError):
             internals.get_sec('02,28,46')
+
+
+duplicate_tracks_test_table = [
+    (('https://open.spotify.com/track/2DGa7iaidT5s0qnINlwMjJ',
+      'https://open.spotify.com/track/2DGa7iaidT5s0qnINlwMjJ'),
+     ('https://open.spotify.com/track/2DGa7iaidT5s0qnINlwMjJ',)),
+
+    (('https://open.spotify.com/track/2DGa7iaidT5s0qnINlwMjJ',
+      '',
+      'https://open.spotify.com/track/3SipFlNddvL0XNZRLXvdZD'),
+     ('https://open.spotify.com/track/2DGa7iaidT5s0qnINlwMjJ',
+      'https://open.spotify.com/track/3SipFlNddvL0XNZRLXvdZD')),
+
+    (('ncs fade',
+      'https://open.spotify.com/track/2DGa7iaidT5s0qnINlwMjJ',
+      '',
+      'ncs fade'),
+     ('ncs fade',
+      'https://open.spotify.com/track/2DGa7iaidT5s0qnINlwMjJ')),
+
+    (('ncs spectre ',
+      '  https://open.spotify.com/track/2DGa7iaidT5s0qnINlwMjJ',
+      ''),
+     ('ncs spectre',
+      'https://open.spotify.com/track/2DGa7iaidT5s0qnINlwMjJ'))
+]
+
+
+@pytest.mark.parametrize("duplicates, expected", duplicate_tracks_test_table)
+def test_get_unique_tracks(tmpdir, duplicates, expected):
+    file_path = os.path.join(str(tmpdir), 'test_duplicates.txt')
+    with open(file_path, 'w') as tin:
+        tin.write('\\n'.join(duplicates))
+
+    unique_tracks = internals.get_unique_tracks(file_path)
+    assert tuple(unique_tracks) == expected