Repository: streamlink/streamlink
Pull request #1675 (2018-05-21 16:09:43)

URL builder utils
As part of the solution for the issues raised in #1519 by @amurzeau, I have started adding some URL utils that can be used in plugins to better manipulate URLs. 

As well as a useful method for iterating through HTML tags, using regex in a brute force kind of way - like most of the plugins do. 

`update_qsd` and `url_concat` are the two URL manipulation methods. `url_concat` will join together URL parts to make a URL, and `update_qsd` can be used to add, remove or update query string parameters in a URL. 

I updated a couple of plugins as examples. 

codecov[bot]: # [Codecov](https://codecov.io/gh/streamlink/streamlink/pull/1675?src=pr&el=h1) Report
> Merging [#1675](https://codecov.io/gh/streamlink/streamlink/pull/1675?src=pr&el=desc) into [master](https://codecov.io/gh/streamlink/streamlink/commit/a5536e1d50679b721f13f1d7030e947916fbe913?src=pr&el=desc) will **decrease** coverage by `0.06%`.
> The diff coverage is `86.27%`.


```diff
@@            Coverage Diff             @@
##           master    #1675      +/-   ##
==========================================
- Coverage    33.4%   33.34%   -0.07%     
==========================================
  Files         230      231       +1     
  Lines       12960    12990      +30     
==========================================
+ Hits         4329     4331       +2     
- Misses       8631     8659      +28
```

gravyboat: This fell through and I didn't ever merge it, sorry @beardypig. Do you want to rebase and we can get this in?
beardypig: I cannot work out why it's failing on 3.7 right now ... When I have some time I will look at it, unless @back-to is able to work it out :) 
back-to: I know why it fails

```
re.finditer
Changed in version 3.7: Non-empty matches can now start just after a previous empty match.
```

https://docs.python.org/3/library/re.html#re.finditer
https://github.com/python/cpython/pull/4471
https://bugs.python.org/issue25054

https://docs.python.org/3/whatsnew/3.7.html#changes-in-the-python-api

---

here is something similar https://bugs.python.org/issue33585
might be the new expected behavior

> If you don't want to find an empty string, change you patter so that it will not match an empty string: ".+".

that has maybe something todo with it,
**but not sure whats the best way to fix it.**

```py
import re
test_html = """
<title>Title</title>
<meta property="og:type" content= "website" />
<meta property="og:url" content="http://test.se/"/>
<meta property="og:site_name" content="Test" />
<script src="https://test.se/test.js"></script>
<link rel="stylesheet" type="text/css" href="https://test.se/test.css">
<script>Tester.ready(function () {
alert("Hello, world!"); });</script>
<a
href="http://test.se/foo">bar</a>
"""
tag_re = re.compile(r'''(?=<(?P<tag>[a-zA-Z]+)(?P<attr>.*?)(?P<end>/)?>(?:(?P<inner>.*?)</\s*(?P=tag)\s*>)?)''', re.MULTILINE | re.DOTALL)
print([m.groups() for m in tag_re.finditer(test_html)])
```

Python 3.7

```py
[('title', '', None, 'Title'),
('meta', ' property="og:type" content= "website" ', '/', 'Title'),
('meta', ' property="og:url" content="http://test.se/"', '/', None),
('meta', ' property="og:site_name" content="Test" ', '/', None),
('script', ' src="https://test.se/test.js"', '/', ''),
('link', ' rel="stylesheet" type="text/css" href="https://test.se/test.css"', None, ''),
('script', '', None, 'Tester.ready(function () {\nalert("Hello, world!"); });'),
('a', '\nhref="http://test.se/foo"', None, 'bar')]
```

Python 3.6

```py
[('title', '', None, 'Title'),
('meta', ' property="og:type" content= "website" ', '/', None),
('meta', ' property="og:url" content="http://test.se/"', '/', None),
('meta', ' property="og:site_name" content="Test" ', '/', None),
('script', ' src="https://test.se/test.js"', None, ''),
('link', ' rel="stylesheet" type="text/css" href="https://test.se/test.css"', None, None),
('script', '', None, 'Tester.ready(function () {\nalert("Hello, world!"); });'),
('a', '\nhref="http://test.se/foo"', None, 'bar')]
```
beardypig: I must be missing something - I cannot see why the `inner` group would be `"Title"` for the second match. It seems like the `inner` group is not being reset, and gets the previous value for some reason (same for when it is `''`). 
back-to: @beardypig 

with removing the `(?= ... )` part, the tests will pass on 3.7

```diff
diff --git a/src/streamlink/plugin/api/utils.py b/src/streamlink/plugin/api/utils.py
index f6f3f12..00445e2 100644
--- a/src/streamlink/plugin/api/utils.py
+++ b/src/streamlink/plugin/api/utils.py
@@ -7,9 +7,9 @@ from ...utils import parse_qsd as parse_query, parse_json, parse_xml
 __all__ = ["parse_json", "parse_xml", "parse_query"]
 
 
-tag_re = re.compile('''(?=<(?P<tag>[a-zA-Z]+)(?P<attr>.*?)(?P<end>/)?>(?:(?P<inner>.*?)</\s*(?P=tag)\s*>)?)''',
+tag_re = re.compile(r'''<(?P<tag>[a-zA-Z]+)(?P<attr>.*?)(?P<end>/)?>(?:(?P<inner>.*?)</\s*(?P=tag)\s*>)?''',
                     re.MULTILINE | re.DOTALL)
-attr_re = re.compile('''\s*(?P<key>[\w-]+)\s*(?:=\s*(?P<quote>["']?)(?P<value>.*?)(?P=quote)\s*)?''')
+attr_re = re.compile(r'''\s*(?P<key>[\w-]+)\s*(?:=\s*(?P<quote>["']?)(?P<value>.*?)(?P=quote)\s*)?''')
 Tag = namedtuple("Tag", "tag attributes text")
 
 
@@ -26,4 +26,3 @@ def itertags(html, tag):
         if match.group("tag") == tag:
             attrs = dict((a.group("key").lower(), a.group("value")) for a in attr_re.finditer(match.group("attr")))
             yield Tag(match.group("tag"), attrs, match.group("inner"))
-
```

---

```
platform linux -- Python 3.7.0, pytest-3.6.3, py-1.5.4, pluggy-0.6.0
rootdir: /run/media/ka/N/github/back-to/streamlink, inifile:
plugins: requests-mock-1.5.2, cov-2.5.1
collected 5 items                                                                                                                                            

tests/test_plugin_utils.py .....
```
beardypig: Not sure that the positive look ahead is even required, might have been left over from a previous `iter_tags` implementation (I had a few goes) :) 
back-to: @beardypig 

`itertags` is actually broken now, because the **test html code** is missing the `<html>` tag,
which will work for the tests but not for real websites.

it can be tested with the `gardenersworld` plugin as it is broken with the current changes.

`<>` must be ignored for inner tags or there will be only one tag `html`

here is a posible fix

https://github.com/back-to/streamlink/commit/ab7d6bf25345a1a866e1ef6411b493fc074fc9a2
https://travis-ci.org/back-to/streamlink/builds/410256162



Patch:
diff --git a/src/streamlink/plugin/api/useragents.py b/src/streamlink/plugin/api/useragents.py
index e9760098..584ec86f 100644
--- a/src/streamlink/plugin/api/useragents.py
+++ b/src/streamlink/plugin/api/useragents.py
@@ -1,24 +1,29 @@
-ANDROID = ("Mozilla/5.0 (Linux; Android 4.4.2; Nexus 4 Build/KOT49H) "
-           "AppleWebKit/537.36 (KHTML, like Gecko) Chrome/34.0.1847.114 Mobile Safari/537.36")
+ANDROID = ("Mozilla/5.0 (Linux; Android 7.1.1; SM-J510FN Build/NMF26X) "
+           "AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.87 Mobile Safari/537.36")
 CHROME = ("Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
-          "AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36")
-CHROME_OS = ("Mozilla/5.0 (X11; CrOS armv7l 4537.56.0) "
-             "AppleWebKit/537.36 (KHTML, like Gecko) Chrome/30.0.1599.38 Safari/537.36")
-IE_11 = "Mozilla/5.0 (MSIE 10.0; Windows NT 6.1; Trident/5.0)"
+          "AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.75 Safari/537.36")
+CHROME_OS = ("Mozilla/5.0 (X11; CrOS armv7l 10718.34.0) "
+             "AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.40 Safari/537.36")
+EDGE = ("Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
+        "AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.140 Safari/537.36 Edge/17.17134")
+FIREFOX = "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:61.0) Gecko/20100101 Firefox/61.0"
+FIREFOX_MAC = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.13; rv:61.0) Gecko/20100101 Firefox/61.0"
 IE_6 = "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; WOW64; Trident/4.0; SLCC1)"
 IE_7 = "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; WOW64; Trident/4.0; SLCC1)"
 IE_8 = "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0; WOW64; Trident/4.0; SLCC1)"
 IE_9 = "Mozilla/5.0 (MSIE 9.0; Windows NT 6.1; Trident/5.0)"
-IPHONE_6 = ("Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) "
-            "AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25")
-IPAD = ("Mozilla/5.0 (iPad; CPU OS 6_0 like Mac OS X) "
-        "AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25")
-WINDOWS_PHONE_8 = ("Mozilla/5.0 (compatible; MSIE 10.0; Windows Phone 8.0; "
-                   "Trident/6.0; IEMobile/10.0; ARM; Touch; NOKIA; Lumia 920)")
+IE_11 = "Mozilla/5.0 (Windows NT 10.0; WOW64; Trident/7.0; rv:11.0) like Gecko"
+IPAD = ("Mozilla/5.0 (iPad; CPU OS 11_4 like Mac OS X) "
+        "AppleWebKit/605.1.15 (KHTML, like Gecko) Version/11.0 Mobile/15E148 Safari/604.1")
+IPHONE = IPHONE_6 = ("Mozilla/5.0 (iPhone; CPU iPhone OS 11_4 like Mac OS X) "
+                     "AppleWebKit/605.1.15 (KHTML, like Gecko) Version/11.0 Mobile/15E148 Safari/604.1")
+OPERA = ("Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
+         "AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.87 Safari/537.36 OPR/54.0.2952.64")
+SAFARI = ("Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) "
+          "AppleWebKit/603.3.8 (KHTML, like Gecko) Version/10.1.2 Safari/603.3.8")
 SAFARI_8 = ("Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_4) "
             "AppleWebKit/600.7.12 (KHTML, like Gecko) Version/8.0.7 Safari/600.7.12")
 SAFARI_7 = ("Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_3) "
             "AppleWebKit/537.75.14 (KHTML, like Gecko) Version/7.0.3 Safari/7046A194A")
-FIREFOX = "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:52.0) Gecko/20100101 Firefox/52.0"
-FIREFOX_MAC = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10; rv:33.0) Gecko/20100101 Firefox/33.0"
-OPERA = "Opera/9.80 (Windows NT 6.0) Presto/2.12.388 Version/12.14"
+WINDOWS_PHONE_8 = ("Mozilla/5.0 (compatible; MSIE 10.0; Windows Phone 8.0; "
+                   "Trident/6.0; IEMobile/10.0; ARM; Touch; NOKIA; Lumia 920)")
diff --git a/src/streamlink/plugin/api/utils.py b/src/streamlink/plugin/api/utils.py
index ea4b1843..f6f3f121 100644
--- a/src/streamlink/plugin/api/utils.py
+++ b/src/streamlink/plugin/api/utils.py
@@ -17,6 +17,7 @@ def itertags(html, tag):
     """
     Brute force regex based HTML tag parser. This is a rough-and-ready searcher to find HTML tags when
     standards compliance is not required. Will find tags that are commented out, or inside script tag etc.
+
     :param html: HTML page
     :param tag: tag name to find
     :return: generator with Tags
@@ -25,3 +26,4 @@ def itertags(html, tag):
         if match.group("tag") == tag:
             attrs = dict((a.group("key").lower(), a.group("value")) for a in attr_re.finditer(match.group("attr")))
             yield Tag(match.group("tag"), attrs, match.group("inner"))
+
diff --git a/src/streamlink/plugins/facebook.py b/src/streamlink/plugins/facebook.py
index 1b5976e2..537b4a5a 100644
--- a/src/streamlink/plugins/facebook.py
+++ b/src/streamlink/plugins/facebook.py
@@ -7,7 +7,7 @@ from streamlink.utils import parse_json
 
 
 class Facebook(Plugin):
-    _url_re = re.compile(r"https?://(?:www\.)?facebook\.com/[^/]+/videos")
+    _url_re = re.compile(r"https?://(?:www\.)?facebook\.com/[^/]+/(posts|videos)")
     _src_re = re.compile(r'''(sd|hd)_src["']?\s*:\s*(?P<quote>["'])(?P<url>.+?)(?P=quote)''')
     _playlist_re = re.compile(r'''video:\[({url:".+?}\])''')
     _plurl_re = re.compile(r'''url:"(.*?)"''')
diff --git a/src/streamlink/plugins/gardenersworld.py b/src/streamlink/plugins/gardenersworld.py
index f212740c..17342e39 100644
--- a/src/streamlink/plugins/gardenersworld.py
+++ b/src/streamlink/plugins/gardenersworld.py
@@ -4,12 +4,12 @@ import re
 
 from streamlink import NoPluginError
 from streamlink.plugin import Plugin
+from streamlink.plugin.api.utils import itertags
 from streamlink.utils import update_scheme
 
 
 class GardenersWorld(Plugin):
     url_re = re.compile(r"https?://(?:www\.)?gardenersworld\.com/")
-    iframe_re = re.compile('''iframe src=(?P<quote>["'])(?P<url>.*?)(?P=quote)''')
 
     @classmethod
     def can_handle_url(cls, url):
@@ -17,13 +17,13 @@ class GardenersWorld(Plugin):
 
     def _get_streams(self):
         page = self.session.http.get(self.url)
-        iframe = self.iframe_re.search(page.text)
-        if iframe:
-            self.logger.debug("Handing off of {0}".format(iframe.group("url")))
+        for iframe in itertags(page.text, u"iframe"):
+            url = iframe.attributes["src"]
+            self.logger.debug("Handing off of {0}".format(url))
             try:
-                return self.session.streams(update_scheme(self.url, iframe.group("url")))
+                return self.session.streams(update_scheme(self.url, url))
             except NoPluginError:
-                self.logger.error("Handing off of {0} failed".format(iframe.group("url")))
+                self.logger.error("Handing off of {0} failed".format(url))
                 return None
 
 
diff --git a/src/streamlink/plugins/mjunoon.py b/src/streamlink/plugins/mjunoon.py
index ebf874fc..d4121018 100644
--- a/src/streamlink/plugins/mjunoon.py
+++ b/src/streamlink/plugins/mjunoon.py
@@ -3,6 +3,7 @@ import re
 
 from streamlink.compat import urlparse, parse_qsl
 from streamlink.plugin import Plugin
+from streamlink.plugin.api import useragents
 from streamlink.plugin.api.utils import itertags
 from streamlink.stream import HLSStream
 
@@ -10,13 +11,14 @@ log = logging.getLogger(__name__)
 
 
 class Mjunoon(Plugin):
-    url_re = re.compile(r"""https?://(?:www.)?mjunoon.tv/""", re.VERBOSE)
+    url_re = re.compile(r"https?://(?:www\.)?mjunoon\.tv/")
 
     @classmethod
     def can_handle_url(cls, url):
         return cls.url_re.match(url) is not None
 
     def _get_streams(self):
+        self.session.http.headers.update({"User-Agent": useragents.FIREFOX})
         res = self.session.http.get(self.url)
         for script in itertags(res.text, 'script'):
             if script.attributes.get("id") == "playerScript":
diff --git a/src/streamlink/plugins/tf1.py b/src/streamlink/plugins/tf1.py
index 072bb889..7c81d54f 100644
--- a/src/streamlink/plugins/tf1.py
+++ b/src/streamlink/plugins/tf1.py
@@ -6,6 +6,7 @@ from streamlink.plugin import Plugin
 from streamlink.plugin.api import useragents
 from streamlink.stream import HDSStream
 from streamlink.stream import HLSStream
+from streamlink.utils.url import update_qsd
 
 
 class TF1(Plugin):
@@ -50,9 +51,8 @@ class TF1(Plugin):
 
         m = self.embed_re.search(embed_page.text)
         if m:
-            o = urlparse(m.group(1))
-            prms = dict(parse_qsl(o.query))
-            hls_stream_url = "{0}://{1}{2}?hdnea={3}".format(o.scheme, o.netloc, o.path, prms["hdnea"])
+            # remove all query string arguments except hdnea
+            hls_stream_url = update_qsd(m.group(1), {"hdnea": None}, remove="*")
             try:
                 for s in HLSStream.parse_variant_playlist(self.session, hls_stream_url).items():
                     yield s
diff --git a/src/streamlink/utils/__init__.py b/src/streamlink/utils/__init__.py
index 6be6e1d7..01b67a06 100644
--- a/src/streamlink/utils/__init__.py
+++ b/src/streamlink/utils/__init__.py
@@ -13,6 +13,7 @@ from streamlink.exceptions import PluginError
 from streamlink.utils.named_pipe import NamedPipe
 from streamlink.utils.lazy_formatter import LazyFormatter
 from streamlink.utils.encoding import get_filesystem_encoding, maybe_decode, maybe_encode
+from streamlink.utils.url import update_scheme, url_equal
 
 
 def swfdecompress(data):
@@ -142,52 +143,6 @@ def rtmpparse(url):
     return tcurl, playpath
 
 
-def update_scheme(current, target):
-    """
-    Take the scheme from the current URL and applies it to the
-    target URL if the target URL startswith // or is missing a scheme
-    :param current: current URL
-    :param target: target URL
-    :return: target URL with the current URLs scheme
-    """
-    target_p = urlparse(target)
-    if not target_p.scheme and target_p.netloc:
-        return "{0}:{1}".format(urlparse(current).scheme,
-                                urlunparse(target_p))
-    elif not target_p.scheme and not target_p.netloc:
-        return "{0}://{1}".format(urlparse(current).scheme,
-                                  urlunparse(target_p))
-    else:
-        return target
-
-
-def url_equal(first, second, ignore_scheme=False, ignore_netloc=False, ignore_path=False, ignore_params=False,
-              ignore_query=False, ignore_fragment=False):
-    """
-    Compare two URLs and return True if they are equal, some parts of the URLs can be ignored
-    :param first: URL
-    :param second: URL
-    :param ignore_scheme: ignore the scheme
-    :param ignore_netloc: ignore the netloc
-    :param ignore_path: ignore the path
-    :param ignore_params: ignore the params
-    :param ignore_query: ignore the query string
-    :param ignore_fragment: ignore the fragment
-    :return: result of comparison
-    """
-    # <scheme>://<netloc>/<path>;<params>?<query>#<fragment>
-
-    firstp = urlparse(first)
-    secondp = urlparse(second)
-
-    return ((firstp.scheme == secondp.scheme or ignore_scheme) and
-            (firstp.netloc == secondp.netloc or ignore_netloc) and
-            (firstp.path == secondp.path or ignore_path) and
-            (firstp.params == secondp.params or ignore_params) and
-            (firstp.query == secondp.query or ignore_query) and
-            (firstp.fragment == secondp.fragment or ignore_fragment))
-
-
 def memoize(obj):
     cache = obj.cache = {}
 
@@ -247,7 +202,6 @@ def load_module(name, path=None):
             if fd:
                 fd.close()
 
-
 #####################################
 # Deprecated functions, do not use. #
 #####################################
@@ -351,4 +305,4 @@ __all__ = ["urlopen", "urlget", "urlresolve", "swfdecompress", "swfverify",
            "verifyjson", "absolute_url", "parse_qsd", "parse_json", "res_json",
            "parse_xml", "res_xml", "rtmpparse", "prepend_www", "NamedPipe",
            "escape_librtmp", "LazyFormatter", "get_filesystem_encoding",
-           "maybe_decode", "maybe_encode"]
+           "maybe_decode", "maybe_encode"]
\ No newline at end of file
diff --git a/src/streamlink/utils/url.py b/src/streamlink/utils/url.py
new file mode 100644
index 00000000..9afcde8a
--- /dev/null
+++ b/src/streamlink/utils/url.py
@@ -0,0 +1,98 @@
+from collections import OrderedDict
+
+from streamlink.compat import urljoin, urlparse, urlunparse, parse_qsl, urlencode
+
+
+def update_scheme(current, target):
+    """
+    Take the scheme from the current URL and applies it to the
+    target URL if the target URL startswith // or is missing a scheme
+    :param current: current URL
+    :param target: target URL
+    :return: target URL with the current URLs scheme
+    """
+    target_p = urlparse(target)
+    if not target_p.scheme and target_p.netloc:
+        return "{0}:{1}".format(urlparse(current).scheme,
+                                urlunparse(target_p))
+    elif not target_p.scheme and not target_p.netloc:
+        return "{0}://{1}".format(urlparse(current).scheme,
+                                  urlunparse(target_p))
+    else:
+        return target
+
+
+def url_equal(first, second, ignore_scheme=False, ignore_netloc=False, ignore_path=False, ignore_params=False,
+              ignore_query=False, ignore_fragment=False):
+    """
+    Compare two URLs and return True if they are equal, some parts of the URLs can be ignored
+    :param first: URL
+    :param second: URL
+    :param ignore_scheme: ignore the scheme
+    :param ignore_netloc: ignore the netloc
+    :param ignore_path: ignore the path
+    :param ignore_params: ignore the params
+    :param ignore_query: ignore the query string
+    :param ignore_fragment: ignore the fragment
+    :return: result of comparison
+    """
+    # <scheme>://<netloc>/<path>;<params>?<query>#<fragment>
+
+    firstp = urlparse(first)
+    secondp = urlparse(second)
+
+    return ((firstp.scheme == secondp.scheme or ignore_scheme) and
+            (firstp.netloc == secondp.netloc or ignore_netloc) and
+            (firstp.path == secondp.path or ignore_path) and
+            (firstp.params == secondp.params or ignore_params) and
+            (firstp.query == secondp.query or ignore_query) and
+            (firstp.fragment == secondp.fragment or ignore_fragment))
+
+
+def url_concat(base, *parts, **kwargs):
+    """
+    Join extra paths to a URL, does not join absolute paths
+    :param base: the base URL
+    :param parts: a list of the parts to join
+    :param allow_fragments: include url fragments
+    :return: the joined URL
+    """
+    allow_fragments = kwargs.get("allow_fragments", True)
+    for part in parts:
+        base = urljoin(base.rstrip("/") + "/", part.strip("/"), allow_fragments)
+    return base
+
+
+def update_qsd(url, qsd=None, remove=None):
+    """
+    Update or remove keys from a query string in a URL
+
+    :param url: URL to update
+    :param qsd: dict of keys to update, a None value leaves it unchanged
+    :param remove: list of keys to remove, or "*" to remove all
+                   note: updated keys are never removed, even if unchanged
+    :return: updated URL
+    """
+    qsd = qsd or {}
+    remove = remove or []
+
+    # parse current query string
+    parsed = urlparse(url)
+    current_qsd = OrderedDict(parse_qsl(parsed.query))
+
+    # * removes all possible keys
+    if remove == "*":
+        remove = list(current_qsd.keys())
+
+    # remove keys before updating, but leave updated keys untouched
+    for key in remove:
+        if key not in qsd:
+            del current_qsd[key]
+
+    # and update the query string
+    for key, value in qsd.items():
+        if value:
+            current_qsd[key] = value
+
+    return parsed._replace(query=urlencode(current_qsd)).geturl()
+


Test patch:
diff --git a/tests/plugins/test_facebook.py b/tests/plugins/test_facebook.py
index b6b171a1..76e72dff 100644
--- a/tests/plugins/test_facebook.py
+++ b/tests/plugins/test_facebook.py
@@ -9,6 +9,7 @@ class TestPluginFacebook(unittest.TestCase):
         self.assertTrue(Facebook.can_handle_url("https://www.facebook.com/nos/videos/1725546430794241/"))
         self.assertTrue(Facebook.can_handle_url("https://www.facebook.com/nytfood/videos/1485091228202006/"))
         self.assertTrue(Facebook.can_handle_url("https://www.facebook.com/SporTurkTR/videos/798553173631138/"))
+        self.assertTrue(Facebook.can_handle_url("https://www.facebook.com/119555411802156/posts/500665313691162/"))
 
         # shouldn't match
         self.assertFalse(Facebook.can_handle_url("https://www.facebook.com"))
diff --git a/tests/test_plugin_utils.py b/tests/test_plugin_utils.py
new file mode 100644
index 00000000..a64b8c77
--- /dev/null
+++ b/tests/test_plugin_utils.py
@@ -0,0 +1,92 @@
+from __future__ import unicode_literals
+import sys
+
+import pytest
+
+from streamlink.plugin.api.utils import itertags
+
+if sys.version_info[0:2] == (2, 6):
+    import unittest2 as unittest
+else:
+    import unittest
+
+
+class TestPluginUtil(unittest.TestCase):
+    test_html = """
+<!doctype html>
+<html lang="en" class="no-js">
+<title>Title</title>
+<meta property="og:type" content= "website" />
+<meta property="og:url" content="http://test.se/"/>
+<meta property="og:site_name" content="Test" />
+<script src="https://test.se/test.js"></script>
+<link rel="stylesheet" type="text/css" href="https://test.se/test.css">
+<script>Tester.ready(function () {
+alert("Hello, world!"); });</script>
+<p>
+<a 
+href="http://test.se/foo">bar</a>
+</p>
+</html>
+        """
+
+    def test_itertags_single_text(self):
+        title = list(itertags(self.test_html, "title"))
+        self.assertTrue(len(title), 1)
+        self.assertEqual(title[0].tag, "title")
+        self.assertEqual(title[0].text, "Title")
+        self.assertEqual(title[0].attributes, {})
+
+    def test_itertags_attrs_text(self):
+        script = list(itertags(self.test_html, "script"))
+        self.assertTrue(len(script), 2)
+        self.assertEqual(script[0].tag, "script")
+        self.assertEqual(script[0].text, "")
+        self.assertEqual(script[0].attributes, {"src": "https://test.se/test.js"})
+
+        self.assertEqual(script[1].tag, "script")
+        self.assertEqual(script[1].text.strip(), """Tester.ready(function () {\nalert("Hello, world!"); });""")
+        self.assertEqual(script[1].attributes, {})
+
+
+    @pytest.mark.xfail(sys.version_info >= (3, 7),
+                       reason="python3.7 issue, see bpo-34294")
+    def test_itertags_multi_attrs(self):
+        metas = list(itertags(self.test_html, "meta"))
+        self.assertTrue(len(metas), 3)
+        self.assertTrue(all(meta.tag == "meta" for meta in metas))
+
+        self.assertEqual(metas[0].text, None)
+        self.assertEqual(metas[1].text, None)
+        self.assertEqual(metas[2].text, None)
+
+        self.assertEqual(metas[0].attributes, {"property": "og:type", "content": "website"})
+        self.assertEqual(metas[1].attributes, {"property": "og:url", "content": "http://test.se/"})
+        self.assertEqual(metas[2].attributes, {"property": "og:site_name", "content": "Test"})
+
+    def test_multi_line_a(self):
+        anchor = list(itertags(self.test_html, "a"))
+        self.assertTrue(len(anchor), 1)
+        self.assertEqual(anchor[0].tag, "a")
+        self.assertEqual(anchor[0].text, "bar")
+        self.assertEqual(anchor[0].attributes, {"href": "http://test.se/foo"})
+
+    @pytest.mark.xfail(sys.version_info >= (3, 7),
+                       reason="python3.7 issue, see bpo-34294")
+    def test_no_end_tag(self):
+        links = list(itertags(self.test_html, "link"))
+        self.assertTrue(len(links), 1)
+        self.assertEqual(links[0].tag, "link")
+        self.assertEqual(links[0].text, None)
+        self.assertEqual(links[0].attributes, {"rel": "stylesheet",
+                                               "type": "text/css",
+                                               "href": "https://test.se/test.css"})
+
+    def test_tag_inner_tag(self):
+        links = list(itertags(self.test_html, "p"))
+        self.assertTrue(len(links), 1)
+        self.assertEqual(links[0].tag, "p")
+        self.assertEqual(links[0].text.strip(), '<a \nhref="http://test.se/foo">bar</a>')
+        self.assertEqual(links[0].attributes, {})
+
+
diff --git a/tests/test_utils.py b/tests/test_utils.py
index c33253c9..b9827864 100644
--- a/tests/test_utils.py
+++ b/tests/test_utils.py
@@ -97,36 +97,6 @@ class TestUtil(unittest.TestCase):
             {"test": "1", "foo": "bar"},
             parse_qsd("test=1&foo=bar", schema=validate.Schema({"test": validate.text, "foo": "bar"})))
 
-    def test_update_scheme(self):
-        self.assertEqual(
-            "https://example.com/foo",  # becomes https
-            update_scheme("https://other.com/bar", "//example.com/foo")
-        )
-        self.assertEqual(
-            "http://example.com/foo",  # becomes http
-            update_scheme("http://other.com/bar", "//example.com/foo")
-        )
-        self.assertEqual(
-            "http://example.com/foo",  # remains unchanged
-            update_scheme("https://other.com/bar", "http://example.com/foo")
-        )
-        self.assertEqual(
-            "https://example.com/foo",  # becomes https
-            update_scheme("https://other.com/bar", "example.com/foo")
-        )
-
-    def test_url_equal(self):
-        self.assertTrue(url_equal("http://test.com/test", "http://test.com/test"))
-        self.assertFalse(url_equal("http://test.com/test", "http://test.com/test2"))
-
-        self.assertTrue(url_equal("http://test.com/test", "http://test.com/test2", ignore_path=True))
-        self.assertTrue(url_equal("http://test.com/test", "https://test.com/test", ignore_scheme=True))
-        self.assertFalse(url_equal("http://test.com/test", "https://test.com/test"))
-
-        self.assertTrue(url_equal("http://test.com/test", "http://test.com/test#hello", ignore_fragment=True))
-        self.assertTrue(url_equal("http://test.com/test", "http://test2.com/test", ignore_netloc=True))
-        self.assertFalse(url_equal("http://test.com/test", "http://test2.com/test1", ignore_netloc=True))
-
     def test_rtmpparse(self):
         self.assertEquals(
             ("rtmp://testserver.com:1935/app", "playpath?arg=1"),
diff --git a/tests/test_utils_url.py b/tests/test_utils_url.py
new file mode 100644
index 00000000..b77dcba4
--- /dev/null
+++ b/tests/test_utils_url.py
@@ -0,0 +1,50 @@
+import sys
+from streamlink.utils.url import *
+
+if sys.version_info[0:2] == (2, 6):
+    import unittest2 as unittest
+else:
+    import unittest
+
+
+class TestUtilURL(unittest.TestCase):
+    def test_update_scheme(self):
+        self.assertEqual(
+            "https://example.com/foo",  # becomes https
+            update_scheme("https://other.com/bar", "//example.com/foo")
+        )
+        self.assertEqual(
+            "http://example.com/foo",  # becomes http
+            update_scheme("http://other.com/bar", "//example.com/foo")
+        )
+        self.assertEqual(
+            "http://example.com/foo",  # remains unchanged
+            update_scheme("https://other.com/bar", "http://example.com/foo")
+        )
+        self.assertEqual(
+            "https://example.com/foo",  # becomes https
+            update_scheme("https://other.com/bar", "example.com/foo")
+        )
+
+    def test_url_equal(self):
+        self.assertTrue(url_equal("http://test.com/test", "http://test.com/test"))
+        self.assertFalse(url_equal("http://test.com/test", "http://test.com/test2"))
+
+        self.assertTrue(url_equal("http://test.com/test", "http://test.com/test2", ignore_path=True))
+        self.assertTrue(url_equal("http://test.com/test", "https://test.com/test", ignore_scheme=True))
+        self.assertFalse(url_equal("http://test.com/test", "https://test.com/test"))
+
+        self.assertTrue(url_equal("http://test.com/test", "http://test.com/test#hello", ignore_fragment=True))
+        self.assertTrue(url_equal("http://test.com/test", "http://test2.com/test", ignore_netloc=True))
+        self.assertFalse(url_equal("http://test.com/test", "http://test2.com/test1", ignore_netloc=True))
+
+    def test_url_concat(self):
+        self.assertEqual(url_concat("http://test.se", "one", "two", "three"), "http://test.se/one/two/three")
+        self.assertEqual(url_concat("http://test.se", "/one", "/two", "/three"), "http://test.se/one/two/three")
+        self.assertEqual(url_concat("http://test.se/one", "../two", "three"), "http://test.se/two/three")
+        self.assertEqual(url_concat("http://test.se/one", "../two", "../three"), "http://test.se/three")
+
+    def test_update_qsd(self):
+        self.assertEqual(update_qsd("http://test.se?one=1&two=3", {"two": 2}), "http://test.se?one=1&two=2")
+        self.assertEqual(update_qsd("http://test.se?one=1&two=3", remove=["two"]), "http://test.se?one=1")
+        self.assertEqual(update_qsd("http://test.se?one=1&two=3", {"one": None}, remove="*"), "http://test.se?one=1")