Repository: streamlink/streamlink

Issue:
Fallback selection for --stream-sorting-excludes
### Checklist

- [ ] This is a bug report.
- [x] This is a feature request.
- [ ] This is a plugin (improvement) request.
- [ ] I have read the contribution guidelines.

### Description

Related issue:
https://github.com/streamlink/streamlink-twitch-gui/issues/481

In certain situations, the usage of `--stream-sorting-excludes` can lead to an empty stream selection. This is an issue for applications which try to map the available streams to a custom list of qualities.

The Streamlink Twitch GUI needs to do this stream quality mapping because of Twitch's inconsistent quality naming. This is an example of trying to find the "low" quality:

```
# available: 160p, 360p, 480p, 720p, 1080p
streamlink --stream-sorting-excludes ">360p" twitch.tv/CHANNEL low,best
# will open 360p

# available: low, medium, high, source
streamlink --stream-sorting-excludes ">360p" twitch.tv/CHANNEL low,best
# will open low

# available: 720p
streamlink --stream-sorting-excludes ">360p" twitch.tv/CHANNEL low,best
# returns an empty selection (channel doesn't have re-encoded qualities)
```

Having a fallback parameter (eg. `--stream-sorting-excludes-fallback`) with a list of streams that will be opened if no matching stream could be found would fix this situation.

Another solution could be having a different notation in the actual quality selection, but this is probably a bit too complex.

```
# 1. try to find the quality labeled "medium"
# 2. look for qualities named by video resolution matching the defined range
# 3. fall back to lower qualities
# 4. use any available quality
streamlink twitch.tv/CHANNEL "medium||>360p&&<=480p||<=360p||best"
streamlink twitch.tv/CHANNEL "medium,>360p<=480p,<=360p,best"
```

Discussion:
bastimeyer: Something on GH seems to be broken right now, oh boy... https://status.github.com/messages :confused: 
codecov[bot]: # [Codecov](https://codecov.io/gh/streamlink/streamlink/pull/2127?src=pr&el=h1) Report
> Merging [#2127](https://codecov.io/gh/streamlink/streamlink/pull/2127?src=pr&el=desc) into [master](https://codecov.io/gh/streamlink/streamlink/commit/b21df07689023f1f6fc79b228da4faa76c617667?src=pr&el=desc) will **decrease** coverage by `0.16%`.
> The diff coverage is `100%`.


```diff
@@            Coverage Diff             @@
##           master    #2127      +/-   ##
==========================================
- Coverage   51.31%   51.14%   -0.17%     
==========================================
  Files         232      232              
  Lines       14184    14189       +5     
==========================================
- Hits         7278     7257      -21     
- Misses       6906     6932      +26
```

bastimeyer: > will decrease coverage by 0.16%

?????
https://codecov.io/gh/streamlink/streamlink/commit/0aad083fdc23ed649b736f29cccab1b2ae25e5be
https://codecov.io/gh/streamlink/streamlink/compare/b21df07689023f1f6fc79b228da4faa76c617667...0aad083fdc23ed649b736f29cccab1b2ae25e5be/changes
gravyboat: Talk about an edge case. This looks good to me @bastimeyer. I'll leave it open for a bit longer in case anyone else wants to weigh in, but if not we'll get it merged. 
bastimeyer: Let's tag it as enhancement...

> edge case

The main problem and motivation for this change is that people who are using the Twitch GUI can set different stream quality selections which filter out unwanted streams by using the stream-sorting-excludes parameter. This has to be done because Twitch doesn't provide an API for knowing the available qualities, so you have to define the selection beforehand (or manually parse the HLS playlist in a couple of redundant requests, which is kinda stupid when using Streamlink).
Now If a user has selected a medium quality and thus filters out everything above the specified medium qualities and a streamer on Twitch doesn't have a full partnership and only a certain quality above "medium" is available, this stream-sorting-excludes filter fails. By having fallback stream name synonyms, this won't happen.
bastimeyer: Interesting... There's something going on with parsing the `vod_alt` stream name on Python<=3.5. This is unrelated to the PR because I was using the suggested stream names by @back-to.

Patch:
diff --git a/src/streamlink/plugin/plugin.py b/src/streamlink/plugin/plugin.py
index 1f5513d7..ef38ac47 100644
--- a/src/streamlink/plugin/plugin.py
+++ b/src/streamlink/plugin/plugin.py
@@ -384,6 +384,7 @@ class Plugin(object):
 
         stream_names = filter(stream_weight_only, streams.keys())
         sorted_streams = sorted(stream_names, key=stream_weight_only)
+        unfiltered_sorted_streams = sorted_streams
 
         if isinstance(sorting_excludes, list):
             for expr in sorting_excludes:
@@ -402,6 +403,11 @@ class Plugin(object):
             worst = sorted_streams[0]
             final_sorted_streams["worst"] = streams[worst]
             final_sorted_streams["best"] = streams[best]
+        elif len(unfiltered_sorted_streams) > 0:
+            best = unfiltered_sorted_streams[-1]
+            worst = unfiltered_sorted_streams[0]
+            final_sorted_streams["worst-unfiltered"] = streams[worst]
+            final_sorted_streams["best-unfiltered"] = streams[best]
 
         return final_sorted_streams
 
diff --git a/src/streamlink/plugins/bilibili.py b/src/streamlink/plugins/bilibili.py
index 8ddb95cd..4e3c068e 100644
--- a/src/streamlink/plugins/bilibili.py
+++ b/src/streamlink/plugins/bilibili.py
@@ -1,12 +1,15 @@
+import hashlib
 import re
+import time
 
 from requests.adapters import HTTPAdapter
 from streamlink.plugin import Plugin
 from streamlink.plugin.api import validate, useragents
 from streamlink.stream import HTTPStream
 
-API_URL = "https://api.live.bilibili.com/room/v1/Room/playUrl?cid={0}&quality=0&platform=web"
+API_URL = "http://live.bilibili.com/api/playurl?cid={0}&player=1&quality=0&sign={1}&otype=json"
 ROOM_API = "https://api.live.bilibili.com/room/v1/Room/room_init?id={}"
+API_SECRET = "95acd7f6cc3392f3"
 SHOW_STATUS_OFFLINE = 0
 SHOW_STATUS_ONLINE = 1
 SHOW_STATUS_ROUND = 2
@@ -29,15 +32,6 @@ _room_id_schema = validate.Schema(
     validate.get("data")
 )
 
-_room_stream_list_schema = validate.Schema(
-    {
-        "data": validate.any(None, {
-            "durl": [{"url": validate.url()}]
-        })
-    },
-    validate.get("data")
-)
-
 
 class Bilibili(Plugin):
     @classmethod
@@ -62,8 +56,11 @@ class Bilibili(Plugin):
         if room_id_json['live_status'] != SHOW_STATUS_ONLINE:
             return
 
-        res = self.session.http.get(API_URL.format(room_id))
-        room = self.session.http.json(res, schema=_room_stream_list_schema)
+        ts = int(time.time() / 60)
+        sign = hashlib.md5(("{0}{1}".format(channel, API_SECRET, ts)).encode("utf-8")).hexdigest()
+
+        res = self.session.http.get(API_URL.format(room_id, sign))
+        room = self.session.http.json(res)
         if not room:
             return
 
diff --git a/src/streamlink/plugins/huomao.py b/src/streamlink/plugins/huomao.py
index e0d5770f..aee762b3 100644
--- a/src/streamlink/plugins/huomao.py
+++ b/src/streamlink/plugins/huomao.py
@@ -4,8 +4,8 @@ simply extracts the videos stream_id, stream_url and stream_quality by
 scraping the HTML and JS of one of Huomaos mobile webpages.
 
 When viewing a stream on huomao.com, the base URL references a room_id. This
-room_id is mapped one-to-one to a stream_id which references the actual .m3u8
-file. Both stream_id, stream_url and stream_quality can be found in the
+room_id is mapped one-to-one to a stream_id which references the actual .flv
+video. Both stream_id, stream_url and stream_quality can be found in the
 HTML and JS source of the mobile_page. Since one stream can occur in many
 different qualities, we scrape all stream_url and stream_quality occurrences
 and return each option to the user.
@@ -14,7 +14,7 @@ and return each option to the user.
 import re
 
 from streamlink.plugin import Plugin
-from streamlink.stream import HLSStream
+from streamlink.stream import HTTPStream
 
 # URL pattern for recognizing inputed Huomao.tv / Huomao.com URL.
 url_re = re.compile(r"""
@@ -35,15 +35,18 @@ mobile_url = "http://www.huomao.com/mobile/mob_live/{0}"
 #   <input id="html_stream" value="efmrCH" type="hidden">
 stream_id_pattern = re.compile(r'id=\"html_stream\" value=\"(?P<stream_id>\w+)\"')
 
-# Pattern for extracting each stream_url and
+# Pattern for extracting each stream_url, stream_quality_url and a prettified
 # stream_quality_name used for quality naming.
 #
 # Example from HTML:
-#   src="http://live-ws-hls.huomaotv.cn/live/<stream_id>_720/playlist.m3u8"
+#   "2: 'http://live-ws.huomaotv.cn/live/'+stream+'_720/playlist.m3u8'"
 stream_info_pattern = re.compile(r"""
-    (?P<stream_url>(?:[\w\/\.\-:]+)
-    \/[^_\"]+(?:_(?P<stream_quality_name>\d+))
-    ?/playlist.m3u8)
+    [1-9]:
+    \s+
+    '(?P<stream_url>(?:\w|\.|:|-|/)+)
+    '\+stream\+'
+    (?P<stream_quality_url>_?(?P<stream_quality_name>\d*))
+    /playlist.m3u8'
 """, re.VERBOSE)
 
 
@@ -62,11 +65,11 @@ class Huomao(Plugin):
         return stream_id.group("stream_id")
 
     def get_stream_info(self, html):
-        """
-        Returns a nested list of different stream options.
+        """Returns a nested list of different stream options.
 
-        Each entry in the list will contain a stream_url and stream_quality_name
-        for each stream occurrence that was found in the JS.
+        Each entry in the list will contain a stream_url, stream_quality_url
+        and stream_quality_name for each stream occurrence that was found in
+        the JS.
         """
         stream_info = stream_info_pattern.findall(html)
 
@@ -77,8 +80,8 @@ class Huomao(Plugin):
         # list and reassigning.
         stream_info_list = []
         for info in stream_info:
-            if not info[1]:
-                stream_info_list.append([info[0], "source"])
+            if not info[2]:
+                stream_info_list.append([info[0], info[1], "source"])
             else:
                 stream_info_list.append(list(info))
 
@@ -92,8 +95,8 @@ class Huomao(Plugin):
 
         streams = {}
         for info in stream_info:
-            if stream_id in info[0]:
-                streams[info[1]] = HLSStream(self.session, info[0])
+            streams[info[2]] = HTTPStream(self.session,
+                                          info[0] + stream_id + info[1] + ".flv")
 
         return streams
 
diff --git a/src/streamlink_cli/argparser.py b/src/streamlink_cli/argparser.py
index 0c77adb3..6af91976 100644
--- a/src/streamlink_cli/argparser.py
+++ b/src/streamlink_cli/argparser.py
@@ -107,7 +107,7 @@ def build_parser():
         help="""
         Stream to play.
 
-        Use "best" or "worst" for selecting the highest or lowest available
+        Use ``best`` or ``worst`` for selecting the highest or lowest available
         quality.
 
         Fallback streams can be specified by using a comma-separated list:
@@ -523,7 +523,7 @@ def build_parser():
         help="""
         Stream to play.
 
-        Use "best" or "worst" for selecting the highest or lowest available
+        Use ``best`` or ``worst`` for selecting the highest or lowest available
         quality.
 
         Fallback streams can be specified by using a comma-separated list:
@@ -590,13 +590,17 @@ def build_parser():
         metavar="STREAMS",
         type=comma_list,
         help="""
-        Fine tune best/worst synonyms by excluding unwanted streams.
+        Fine tune the ``best`` and ``worst`` stream name synonyms by excluding unwanted streams.
+
+        If all of the available streams get excluded, ``best`` and ``worst`` will become
+        inaccessible and new special stream synonyms ``best-unfiltered`` and ``worst-unfiltered``
+        can be used as a fallback selection method.
 
         Uses a filter expression in the format:
 
           [operator]<value>
 
-        Valid operators are >, >=, < and <=. If no operator is specified then
+        Valid operators are ``>``, ``>=``, ``<`` and ``<=``. If no operator is specified then
         equality is tested.
 
         For example this will exclude streams ranked higher than "480p":
diff --git a/src/streamlink_cli/constants.py b/src/streamlink_cli/constants.py
index 05ec5dba..45eb7894 100644
--- a/src/streamlink_cli/constants.py
+++ b/src/streamlink_cli/constants.py
@@ -28,7 +28,7 @@ else:
     ]
     PLUGINS_DIR = os.path.expanduser(XDG_CONFIG_HOME + "/streamlink/plugins")
 
-STREAM_SYNONYMS = ["best", "worst"]
+STREAM_SYNONYMS = ["best", "worst", "best-unfiltered", "worst-unfiltered"]
 STREAM_PASSTHROUGH = ["hls", "http", "rtmp"]
 
 __all__ = [
diff --git a/src/streamlink_cli/main.py b/src/streamlink_cli/main.py
index b43175a6..3fe315ce 100644
--- a/src/streamlink_cli/main.py
+++ b/src/streamlink_cli/main.py
@@ -57,14 +57,10 @@ def check_file_output(filename, force):
     log.debug("Checking file output")
 
     if os.path.isfile(filename) and not force:
-        if sys.stdin.isatty():
-            answer = console.ask("File {0} already exists! Overwrite it? [y/N] ",
-                                 filename)
+        answer = console.ask("File {0} already exists! Overwrite it? [y/N] ",
+                             filename)
 
-            if answer.lower() != "y":
-                sys.exit()
-        else:
-            log.error("File {0} already exists, use --force to overwrite it.".format(filename))
+        if answer.lower() != "y":
             sys.exit()
 
     return FileOutput(filename)
@@ -326,7 +322,7 @@ def read_stream(stream, output, prebuffer, chunk_size=8192):
     is_player = isinstance(output, PlayerOutput)
     is_http = isinstance(output, HTTPServer)
     is_fifo = is_player and output.namedpipe
-    show_progress = isinstance(output, FileOutput) and output.fd is not stdout and sys.stdout.isatty()
+    show_progress = isinstance(output, FileOutput) and output.fd is not stdout
 
     stream_iterator = chain(
         [prebuffer],


Test patch:
diff --git a/tests/plugins/test_huomao.py b/tests/plugins/test_huomao.py
index 1261680f..a27efdcb 100644
--- a/tests/plugins/test_huomao.py
+++ b/tests/plugins/test_huomao.py
@@ -15,12 +15,15 @@ class TestPluginHuomao(unittest.TestCase):
         #   room_id             = 123456
         #   stream_id           = 9qsvyF24659
         #   stream_url          = http://live-ws.huomaotv.cn/live/
+        #   stream_quality      = source, _720 and _480
         #   stream_quality_name = source, 720 and 480
         self.mock_html = """
             <input id="html_stream" value="9qsvyF24659" type="hidden">
-            <source  src="http://live-ws-hls.huomaotv.cn/live/9qsvyF24659/playlist.m3u8">
-            <source  src="http://live-ws-hls.huomaotv.cn/live/9qsvyF24659_720/playlist.m3u8">
-            <source  src="http://live-ws-hls.huomaotv.cn/live/9qsvyF24659_480/playlist.m3u8">
+            <!--            urls:{-->
+            <!--                1: 'http://live-ws.huomaotv.cn/live/'+stream+'/playlist.m3u8',-->
+            <!--                2: 'http://live-ws.huomaotv.cn/live/'+stream+'_720/playlist.m3u8',-->
+            <!--                3: 'http://live-ws.huomaotv.cn/live/'+stream+'_480/playlist.m3u8'-->
+            <!--            },-->
         """
 
         # Create a mock Huomao object.
@@ -40,9 +43,9 @@ class TestPluginHuomao(unittest.TestCase):
         # Assert that the stream_url, stream_quality and stream_quality_name
         # is correctly extracted from the mock HTML.
         self.assertEqual(self.mock_huomao.get_stream_info(self.mock_html), [
-            ["http://live-ws-hls.huomaotv.cn/live/9qsvyF24659/playlist.m3u8", "source"],
-            ["http://live-ws-hls.huomaotv.cn/live/9qsvyF24659_720/playlist.m3u8", "720"],
-            ["http://live-ws-hls.huomaotv.cn/live/9qsvyF24659_480/playlist.m3u8", "480"]
+            ["http://live-ws.huomaotv.cn/live/", "", "source"],
+            ["http://live-ws.huomaotv.cn/live/", "_720", "720"],
+            ["http://live-ws.huomaotv.cn/live/", "_480", "480"]
         ])
 
     def test_can_handle_url(self):
diff --git a/tests/plugins/testplugin.py b/tests/plugins/testplugin.py
index 06b693c4..66822ef4 100644
--- a/tests/plugins/testplugin.py
+++ b/tests/plugins/testplugin.py
@@ -37,6 +37,14 @@ class TestPlugin(Plugin):
     def _get_streams(self):
         if "empty" in self.url:
             return
+
+        if "UnsortableStreamNames" in self.url:
+            def gen():
+                for i in range(3):
+                    yield "vod", HTTPStream(self.session, "http://test.se/stream")
+
+            return gen()
+
         if "NoStreamsError" in self.url:
             raise NoStreamsError(self.url)
 
diff --git a/tests/test_cli_main.py b/tests/test_cli_main.py
index dccd30a7..86c3b9c4 100644
--- a/tests/test_cli_main.py
+++ b/tests/test_cli_main.py
@@ -3,8 +3,9 @@ import os.path
 import tempfile
 
 import streamlink_cli.main
-from streamlink_cli.main import resolve_stream_name, check_file_output
+from streamlink_cli.main import resolve_stream_name, format_valid_streams, check_file_output
 from streamlink_cli.output import FileOutput
+from streamlink.plugin.plugin import Plugin
 import unittest
 from tests.mock import Mock, patch
 
@@ -18,25 +19,12 @@ class TestCLIMain(unittest.TestCase):
         tmpfile = tempfile.NamedTemporaryFile()
         try:
             streamlink_cli.main.console = console = Mock()
-            streamlink_cli.main.sys.stdin = stdin = Mock()
-            stdin.isatty.return_value = True
             console.ask.return_value = "y"
             self.assertTrue(os.path.exists(tmpfile.name))
             self.assertIsInstance(check_file_output(tmpfile.name, False), FileOutput)
         finally:
             tmpfile.close()
 
-    def test_check_file_output_exists_notty(self):
-        tmpfile = tempfile.NamedTemporaryFile()
-        try:
-            streamlink_cli.main.console = console = Mock()
-            streamlink_cli.main.sys.stdin = stdin = Mock()
-            stdin.isatty.return_value = False
-            self.assertTrue(os.path.exists(tmpfile.name))
-            self.assertRaises(SystemExit, check_file_output, tmpfile.name, False)
-        finally:
-            tmpfile.close()
-
     def test_check_file_output_exists_force(self):
         tmpfile = tempfile.NamedTemporaryFile()
         try:
@@ -59,19 +47,71 @@ class TestCLIMain(unittest.TestCase):
             tmpfile.close()
 
     def test_resolve_stream_name(self):
-        high = Mock()
-        medium = Mock()
-        low = Mock()
+        a = Mock()
+        b = Mock()
+        c = Mock()
+        d = Mock()
+        e = Mock()
         streams = {
-            "low": low,
-            "medium": medium,
-            "high": high,
-            "worst": low,
-            "best": high
+            "160p": a,
+            "360p": b,
+            "480p": c,
+            "720p": d,
+            "1080p": e,
+            "worst": b,
+            "best": d,
+            "worst-unfiltered": a,
+            "best-unfiltered": e
         }
-        self.assertEqual("high", resolve_stream_name(streams, "best"))
-        self.assertEqual("low", resolve_stream_name(streams, "worst"))
-        self.assertEqual("medium", resolve_stream_name(streams, "medium"))
-        self.assertEqual("high", resolve_stream_name(streams, "high"))
-        self.assertEqual("low", resolve_stream_name(streams, "low"))
 
+        self.assertEqual(resolve_stream_name(streams, "unknown"), "unknown")
+        self.assertEqual(resolve_stream_name(streams, "160p"), "160p")
+        self.assertEqual(resolve_stream_name(streams, "360p"), "360p")
+        self.assertEqual(resolve_stream_name(streams, "480p"), "480p")
+        self.assertEqual(resolve_stream_name(streams, "720p"), "720p")
+        self.assertEqual(resolve_stream_name(streams, "1080p"), "1080p")
+        self.assertEqual(resolve_stream_name(streams, "worst"), "360p")
+        self.assertEqual(resolve_stream_name(streams, "best"), "720p")
+        self.assertEqual(resolve_stream_name(streams, "worst-unfiltered"), "160p")
+        self.assertEqual(resolve_stream_name(streams, "best-unfiltered"), "1080p")
+
+    def test_format_valid_streams(self):
+        class FakePlugin:
+            @classmethod
+            def stream_weight(cls, stream):
+                return Plugin.stream_weight(stream)
+        a = Mock()
+        b = Mock()
+        c = Mock()
+
+        streams = {
+            "audio": a,
+            "720p": b,
+            "1080p": c,
+            "worst": b,
+            "best": c
+        }
+        self.assertEqual(
+            format_valid_streams(FakePlugin, streams),
+            ", ".join([
+                "audio",
+                "720p (worst)",
+                "1080p (best)"
+            ])
+        )
+
+        streams = {
+            "audio": a,
+            "720p": b,
+            "1080p": c,
+            "worst-unfiltered": b,
+            "best-unfiltered": c
+        }
+        self.assertEqual(
+            format_valid_streams(FakePlugin, streams),
+            ", ".join([
+                "audio",
+                "720p (worst-unfiltered)",
+                "1080p (best-unfiltered)"
+            ])
+        )
diff --git a/tests/test_session.py b/tests/test_session.py
index 573aae37..e923ab53 100644
--- a/tests/test_session.py
+++ b/tests/test_session.py
@@ -99,12 +99,23 @@ class TestSession(unittest.TestCase):
         self.assertTrue(isinstance(streams["480p"], RTMPStream))
         self.assertTrue(isinstance(streams["480p_http"], HTTPStream))
 
-    def test_plugin_stream_sorted_excludes(self):
+    def test_plugin_stream_sorting_excludes(self):
         channel = self.session.resolve_url("http://test.se/channel")
-        streams = channel.streams(sorting_excludes=["1080p", "3000k"])
 
+        streams = channel.streams(sorting_excludes=[])
         self.assertTrue("best" in streams)
         self.assertTrue("worst" in streams)
+        self.assertFalse("best-unfiltered" in streams)
+        self.assertFalse("worst-unfiltered" in streams)
+        self.assertTrue(streams["worst"] is streams["350k"])
+        self.assertTrue(streams["best"] is streams["1080p"])
+
+        streams = channel.streams(sorting_excludes=["1080p", "3000k"])
+        self.assertTrue("best" in streams)
+        self.assertTrue("worst" in streams)
+        self.assertFalse("best-unfiltered" in streams)
+        self.assertFalse("worst-unfiltered" in streams)
+        self.assertTrue(streams["worst"] is streams["350k"])
         self.assertTrue(streams["best"] is streams["1500k"])
 
         streams = channel.streams(sorting_excludes=[">=1080p", ">1500k"])
@@ -113,6 +124,24 @@ class TestSession(unittest.TestCase):
         streams = channel.streams(sorting_excludes=lambda q: not q.endswith("p"))
         self.assertTrue(streams["best"] is streams["3000k"])
 
+        streams = channel.streams(sorting_excludes=lambda q: False)
+        self.assertFalse("best" in streams)
+        self.assertFalse("worst" in streams)
+        self.assertTrue("best-unfiltered" in streams)
+        self.assertTrue("worst-unfiltered" in streams)
+        self.assertTrue(streams["worst-unfiltered"] is streams["350k"])
+        self.assertTrue(streams["best-unfiltered"] is streams["1080p"])
+
+        channel = self.session.resolve_url("http://test.se/UnsortableStreamNames")
+        streams = channel.streams()
+        self.assertFalse("best" in streams)
+        self.assertFalse("worst" in streams)
+        self.assertFalse("best-unfiltered" in streams)
+        self.assertFalse("worst-unfiltered" in streams)
+        self.assertTrue("vod" in streams)
+        self.assertTrue("vod_alt" in streams)
+        self.assertTrue("vod_alt2" in streams)
+
     def test_plugin_support(self):
         channel = self.session.resolve_url("http://test.se/channel")
         streams = channel.streams()