Repository: Cog-Creators/Red-DiscordBot

Issue:
[V2] Audio.py - Cache manager appears to remove youtubedl partial downloads.
### Type:

- [ ] Suggestion
- [X] Bug

### Brief description of the problem
Looks like we might have a possible race condition between youtube-dl and the cache manager in audio.py
Youtube-dl downloads to a .part then renames, cache manager seems to be dumping .part files (`_cache_required_files()` doesn't seem to include these)

https://github.com/Cog-Creators/Red-DiscordBot/blob/a3bd757120ee99429a59893db7e07dcdb04e9671/cogs/audio.py#L538-L568
There's the cache dump

Here's where the file's required are defined:
https://github.com/Cog-Creators/Red-DiscordBot/blob/a3bd757120ee99429a59893db7e07dcdb04e9671/cogs/audio.py#L374-L383

I'm not 100% sure I've properly tracked the issue back though, and would definitely appreciate a second set of eyes on this before I make a PR of any sort to audio.
### Expected behavior
Things still needed in the cache shouldn't get removed until they are no longer needed.

### Actual behavior
(copied from the support channel in discord)
```
Feb 10 19:06:11 inxa-bot python3.5[25727]: ERROR: unable to rename file: [Errno 2] No such file or directory: 'data/audio/cache/xvJKMyMzjos.part' -> 'data/audio/cache/xvJKMyMzjos'
Feb 10 19:06:12 inxa-bot python3.5[25727]: data/audio/cache/xvJKMyMzjos: No such file or directory
```


Discussion:
I'm experiencing this exact issue right now. Typically seems to happen when the download is larger than normal.
Same issue as #844. Not sure if this is a large enough issue to revisit a fix for. Try setting your max cache size to a larger number.

Resolving patch:
diff --git a/cogs/audio.py b/cogs/audio.py
--- a/cogs/audio.py
+++ b/cogs/audio.py
@@ -360,14 +360,15 @@ def _addleft_to_queue(self, server, url, channel):
         self.queue[server.id][QueueKey.QUEUE].appendleft(queued_song)
 
     def _cache_desired_files(self):
-        filelist = []
+        filelist = set()
+
         for server in self.downloaders:
             song = self.downloaders[server].song
             try:
-                filelist.append(song.id)
+                filelist.add(song.id)
             except AttributeError:
                 pass
-        shuffle(filelist)
+
         return filelist
 
     def _cache_max(self):
@@ -380,13 +381,15 @@ def _cache_min(self):
 
     def _cache_required_files(self):
         queue = copy.deepcopy(self.queue)
-        filelist = []
+        filelist = set()
+
         for server in queue:
             now_playing = queue[server].get(QueueKey.NOW_PLAYING)
             try:
-                filelist.append(now_playing.id)
+                filelist.add(now_playing.id)
             except AttributeError:
                 pass
+
         return filelist
 
     def _cache_size(self):
@@ -552,16 +555,22 @@ def _dump_cache(self, ignore_desired=False):
         prev_size = self._cache_size()
 
         for file in os.listdir(self.cache_path):
-            if file not in reqd:
-                if ignore_desired or file not in opt:
-                    try:
-                        os.remove(os.path.join(self.cache_path, file))
-                    except OSError:
-                        # A directory got in the cache?
-                        pass
-                    except WindowsError:
-                        # Removing a file in use, reqd failed
-                        pass
+            if file in reqd:
+                continue
+            elif not ignore_desired:
+                if file in opt:
+                    continue
+                elif file.endswith('.part') and file[:-5] in opt:
+                    continue
+
+            try:
+                os.remove(os.path.join(self.cache_path, file))
+            except OSError:
+                # A directory got in the cache?
+                pass
+            except WindowsError:
+                # Removing a file in use, reqd failed
+                pass
 
         post_size = self._cache_size()
         dumped = prev_size - post_size


Test patch: