Repository: streamlink/streamlink
Pull request #1595 (2018-04-09 22:27:41)

[ABweb] New plugin for BIS Livestreams of french AB Groupe
> AB Groupe
> The AB is a French business group in the field of broadcasting.
> https://en.wikipedia.org/wiki/AB_Groupe

**Requires a login and a subscription.**

Username/Password is only required for the first login every 24h,
it will cache the required cookies and force a relog every 24h.

```
$ streamlink "http://www.abweb.com/BIS-TV-online/bistvo-tele-universal.aspx?chn=ab1" best
[cli][info] Found matching plugin abweb for URL
http://www.abweb.com/BIS-TV-online/bistvo-tele-universal.aspx?chn=ab1
[plugin.abweb][info] Attempting to authenticate using cached cookies
[cli][info] Available streams: 75k (worst), 240p, 360p, 432p, 720p (best)
[cli][info] Opening stream: 720p (hls)
[cli][info] Starting player: /usr/bin/vlc
[cli][info] Player closed
[cli][info] Stream ended
[cli][info] Closing currently open stream...
```

http://www.abweb.com/BIS-TV-Online/
http://www.abweb.com/BIS-TV-Online/bistvo-tele-universal.aspx

closes https://github.com/streamlink/streamlink/issues/1592

---

Ping @NeySlim

would be nice if you can test it, the easiest way without changing all files is

sideloading the file `src/streamlink/plugins/abweb.py`

https://streamlink.github.io/cli.html#sideloading-plugins
https://streamlink.github.io/cli.html#cmdoption-plugin-dirs

and change this two lines from

```py
    options = PluginOptions({
        'username': None,
        'password': None,
        'purge_credentials': None
    })
```

to

```py
    options = PluginOptions({
        'username': 'YOUREMAIL',
        'password': 'YOURPASSWORD',
        'purge_credentials': None
    })
```


codecov[bot]: # [Codecov](https://codecov.io/gh/streamlink/streamlink/pull/1595?src=pr&el=h1) Report
> Merging [#1595](https://codecov.io/gh/streamlink/streamlink/pull/1595?src=pr&el=desc) into [master](https://codecov.io/gh/streamlink/streamlink/commit/2f994acbe5bac2642c0267638fbdfdb7a48a275a?src=pr&el=desc) will **decrease** coverage by `<.01%`.
> The diff coverage is `33.06%`.


```diff
@@            Coverage Diff             @@
##           master    #1595      +/-   ##
==========================================
- Coverage   33.29%   33.28%   -0.01%     
==========================================
  Files         225      226       +1     
  Lines       12619    12743     +124     
==========================================
+ Hits         4201     4242      +41     
- Misses       8418     8501      +83
```

back-to: Not sure why you got an error at the first time there

It could be some _requests session error_, but **i can't reproduce it.**

---

can you reproduce it with every new session?

**delete** `~/.cache/streamlink/plugin-cache.json` to reset the cached session cookies

---

**if you can reproduce it**

change your file with this one and run the command with `-l debug` so it will print your installed versions.

https://gist.githubusercontent.com/back-to/98b56c02cd4459f2bfb02a0cb97fc185/raw/1bf5c426ab5a22445f2099adcdf8f0cbf58a368a/abweb.py

https://gist.github.com/back-to/98b56c02cd4459f2bfb02a0cb97fc185

it will print the whole website source to your terminal and some other informations.

could you upload the output as a **SECRET** file on https://gist.github.com/
and send me an email with the link.
NeySlim: Can't reproduce it. I think it may be a network issue, since it's an iframe, it may lead to another domain that was not resolved correctly or something like that.
NeySlim: There was an update on the ab website and streamlink cannot find playlist anymore.
I wanted to correct myself, but I have to admit, I suck.
This is still the same manifest, still the iframe, maybe the it's just the regex to catch it ?

If you lead me where the url is parsed, I can try to correct myself. Otherwise, credentials you've got are still available.
back-to: the iframe website changed,
with the new update it should work again.
NeySlim: Dude.
You rock.

Patch:
diff --git a/.travis.yml b/.travis.yml
index e76d8195..b7fe249f 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -63,14 +63,6 @@ deploy:
     on:
       branch: master
       condition: $BUILD_INSTALLER == yes && $TRAVIS_EVENT_TYPE == cron
-  - provider: bintray
-    file: build/bintray-latest.json
-    user: "${BINTRAY_USER}"
-    key: "${BINTRAY_KEY}"
-    skip_cleanup: true
-    on:
-      branch: master
-      condition: $BUILD_INSTALLER == yes && $TRAVIS_EVENT_TYPE == cron
 
 after_deploy:
   - if [[ "$BUILD_INSTALLER" == 'yes' && "$TRAVIS_EVENT_TYPE" == 'cron' ]]; then ./script/bintrayupdate.sh; fi
diff --git a/docs/plugin_matrix.rst b/docs/plugin_matrix.rst
index b5eec16a..1c45530d 100644
--- a/docs/plugin_matrix.rst
+++ b/docs/plugin_matrix.rst
@@ -12,6 +12,7 @@ is limited.
 =================== ==================== ===== ===== ===========================
 Name                URL(s)               Live  VOD   Notes
 =================== ==================== ===== ===== ===========================
+abweb               abweb.com            Yes   No    Requires a login and a subscription.
 adultswim           adultswim.com        Yes   Yes   Streams may be geo-restricted, some VOD streams are protected by DRM.
 afreeca             afreecatv.com        Yes   No
 afreecatv           afreeca.tv           Yes   No
diff --git a/script/bintrayconfig.sh b/script/bintrayconfig.sh
index f4e64ab4..33759533 100755
--- a/script/bintrayconfig.sh
+++ b/script/bintrayconfig.sh
@@ -21,37 +21,6 @@ else
     STREAMLINK_INSTALLER="streamlink-${STREAMLINK_VERSION}"
 fi
 
-cat > "${build_dir}/bintray-latest.json" <<EOF
-{
-  "package": {
-    "subject": "streamlink",
-    "repo": "streamlink-nightly",
-    "name": "streamlink"
-  },
-
-  "version": {
-    "name": "latest",
-    "released": "$(date +'%Y-%m-%d')",
-    "desc": "Latest version of the installer (${STREAMLINK_VERSION})"
-  },
-
-  "files": [
-    {
-      "includePattern": "${dist_dir}/${STREAMLINK_INSTALLER}.exe",
-      "uploadPattern": "streamlink-latest.exe",
-      "matrixParams": {
-        "override": 1,
-        "publish": 1
-      }
-    }
-  ],
-
-  "publish": true
-}
-EOF
-
-echo "Wrote Bintray config to: ${build_dir}/bintray-latest.json"
-
 cat > "${build_dir}/bintray-nightly.json" <<EOF
 {
   "package": {
diff --git a/script/bintrayupdate.sh b/script/bintrayupdate.sh
index 74e16e7d..5ce7321b 100755
--- a/script/bintrayupdate.sh
+++ b/script/bintrayupdate.sh
@@ -1,8 +1,21 @@
 #!/usr/bin/env bash
+set -e
 # Update the Bintray release
 
+old_date=$(date +"%Y.%m.%d" -d "30 days ago")
 build_dir="$(pwd)/build"
-current_api_url="https://api.bintray.com/packages/streamlink/streamlink-nightly/streamlink/versions/$(date +'%Y.%m.%d')/release_notes"
-latest_api_url="https://api.bintray.com/packages/streamlink/streamlink-nightly/streamlink/versions/latest/release_notes"
-curl -u "${BINTRAY_USER}:${BINTRAY_KEY}" -H "Content-Type: application/json" -d @"${build_dir}/bintray-changelog.json" ${current_api_url}
-curl -u "${BINTRAY_USER}:${BINTRAY_KEY}" -H "Content-Type: application/json" -d @"${build_dir}/bintray-changelog.json" ${latest_api_url}
+api_base="https://api.bintray.com/packages/streamlink/streamlink-nightly/streamlink"
+current_api_url="${api_base}/versions/$(date +'%Y.%m.%d')/release_notes"
+files_api_url="${api_base}/files"
+versions_api_url="${api_base}/versions"
+
+curl -u "${BINTRAY_USER}:${BINTRAY_KEY}" -H "Content-Type: application/json" -d @"${build_dir}/bintray-changelog.json" "${current_api_url}"
+
+echo "Deleting versions older than ${old_date}..."
+versions=$(curl -s -u "${BINTRAY_USER}:${BINTRAY_KEY}" "${files_api_url}" | jq -r '.[].version')
+
+for version in $versions; do
+    if [[ "$version" < "$old_date" ]]; then
+        curl -s -u "${BINTRAY_USER}:${BINTRAY_KEY}" -X DELETE "${versions_api_url}/${version}" -o /dev/null && echo "Deleted version: ${version}"
+    fi
+done
diff --git a/src/streamlink/plugins/abweb.py b/src/streamlink/plugins/abweb.py
new file mode 100644
index 00000000..1141886f
--- /dev/null
+++ b/src/streamlink/plugins/abweb.py
@@ -0,0 +1,162 @@
+import re
+import time
+
+from streamlink.cache import Cache
+from streamlink.exceptions import PluginError
+from streamlink.plugin import Plugin
+from streamlink.plugin import PluginOptions
+from streamlink.plugin.api import http
+from streamlink.plugin.api import useragents
+from streamlink.stream import HLSStream
+from streamlink.utils import update_scheme
+
+
+class ABweb(Plugin):
+    '''BIS Livestreams of french AB Groupe
+       http://www.abweb.com/BIS-TV-Online/
+    '''
+
+    login_url = 'http://www.abweb.com/BIS-TV-Online/Default.aspx'
+
+    _url_re = re.compile(r'https?://(?:www\.)?abweb\.com/BIS-TV-Online/bistvo-tele-universal.aspx', re.IGNORECASE)
+    _hls_re = re.compile(r'''["']file["']:\s?["'](?P<url>[^"']+\.m3u8[^"']+)["']''')
+    _iframe_re = re.compile(r'''<iframe[^>]+src=["'](?P<url>[^"']+)["']''')
+
+    _input_re = re.compile(r'''(<input[^>]+>)''')
+    _name_re = re.compile(r'''name=["']([^"']*)["']''')
+    _value_re = re.compile(r'''value=["']([^"']*)["']''')
+
+    expires_time = 3600 * 24
+
+    options = PluginOptions({
+        'username': None,
+        'password': None,
+        'purge_credentials': None
+    })
+
+    def __init__(self, url):
+        super(ABweb, self).__init__(url)
+        self._session_attributes = Cache(filename='plugin-cache.json', key_prefix='abweb:attributes')
+        self._authed = self._session_attributes.get('ASP.NET_SessionId') and self._session_attributes.get('.abportail1')
+        self._expires = self._session_attributes.get('expires', time.time() + self.expires_time)
+
+    @classmethod
+    def can_handle_url(cls, url):
+        return cls._url_re.match(url) is not None
+
+    def set_expires_time_cache(self):
+        expires = time.time() + self.expires_time
+        self._session_attributes.set('expires', expires, expires=self.expires_time)
+
+    def get_iframe_url(self):
+        self.logger.debug('search for an iframe')
+        res = http.get(self.url)
+        m = self._iframe_re.search(res.text)
+        if not m:
+            raise PluginError('No iframe found.')
+
+        iframe_url = m.group('url')
+        iframe_url = update_scheme('http://', iframe_url)
+        self.logger.debug('IFRAME URL={0}'.format(iframe_url))
+        return iframe_url
+
+    def get_hls_url(self, iframe_url):
+        self.logger.debug('search for hls url')
+        res = http.get(iframe_url)
+        m = self._hls_re.search(res.text)
+        if not m:
+            raise PluginError('No playlist found.')
+
+        return m and m.group('url')
+
+    def _login(self, username, password):
+        '''login and update cached cookies'''
+        self.logger.debug('login ...')
+
+        res = http.get(self.login_url)
+        input_list = self._input_re.findall(res.text)
+        if not input_list:
+            raise PluginError('Missing input data on login website.')
+
+        data = {}
+        for _input_data in input_list:
+            try:
+                _input_name = self._name_re.search(_input_data).group(1)
+            except AttributeError:
+                continue
+
+            try:
+                _input_value = self._value_re.search(_input_data).group(1)
+            except AttributeError:
+                _input_value = ''
+
+            data[_input_name] = _input_value
+
+        login_data = {
+            'ctl00$Login1$UserName': username,
+            'ctl00$Login1$Password': password,
+            'ctl00$Login1$LoginButton.x': '0',
+            'ctl00$Login1$LoginButton.y': '0'
+        }
+        data.update(login_data)
+
+        res = http.post(self.login_url, data=data)
+
+        for cookie in http.cookies:
+            self._session_attributes.set(cookie.name, cookie.value, expires=3600 * 24)
+
+        if self._session_attributes.get('ASP.NET_SessionId') and self._session_attributes.get('.abportail1'):
+            self.logger.debug('New session data')
+            self.set_expires_time_cache()
+            return True
+        else:
+            self.logger.error('Failed to login, check your username/password')
+            return False
+
+    def _get_streams(self):
+        http.headers.update({'User-Agent': useragents.CHROME,
+                             'Referer': 'http://www.abweb.com/BIS-TV-Online/bistvo-tele-universal.aspx'})
+
+        login_username = self.get_option('username')
+        login_password = self.get_option('password')
+
+        if self.options.get('purge_credentials'):
+            self._session_attributes.set('ASP.NET_SessionId', None, expires=0)
+            self._session_attributes.set('.abportail1', None, expires=0)
+            self._authed = False
+            self.logger.info('All credentials were successfully removed.')
+
+        if not self._authed and not (login_username and login_password):
+            self.logger.error('A login for ABweb is required, use --abweb-username USERNAME --abweb-password PASSWORD')
+            return
+
+        if self._authed:
+            if self._expires < time.time():
+                self.logger.debug('get new cached cookies')
+                # login after 24h
+                self.set_expires_time_cache()
+                self._authed = False
+            else:
+                self.logger.info('Attempting to authenticate using cached cookies')
+                http.cookies.set('ASP.NET_SessionId', self._session_attributes.get('ASP.NET_SessionId'))
+                http.cookies.set('.abportail1', self._session_attributes.get('.abportail1'))
+
+        if not self._authed and not self._login(login_username, login_password):
+            return
+
+        iframe_url = self.get_iframe_url()
+        http.headers.update({'Referer': iframe_url})
+
+        hls_url = self.get_hls_url(iframe_url)
+        hls_url = update_scheme(self.url, hls_url)
+
+        self.logger.debug('URL={0}'.format(hls_url))
+        variant = HLSStream.parse_variant_playlist(self.session, hls_url)
+        if variant:
+            for q, s in variant.items():
+                yield q, s
+        else:
+            yield 'live', HLSStream(self.session, hls_url)
+
+
+__plugin__ = ABweb
diff --git a/src/streamlink/plugins/app17.py b/src/streamlink/plugins/app17.py
index d487a149..3b3373d6 100644
--- a/src/streamlink/plugins/app17.py
+++ b/src/streamlink/plugins/app17.py
@@ -46,7 +46,7 @@ class App17(Plugin):
             for stream in HLSStream.parse_variant_playlist(self.session, url).items():
                 yield stream
         else:
-            url = http_url.replace(".flv", ".m3u8")
+            url = http_url.replace("live-hdl", "live-hls").replace(".flv", ".m3u8")
             yield "live", HLSStream(self.session, url)
 
 
diff --git a/src/streamlink/plugins/brittv.py b/src/streamlink/plugins/brittv.py
index 89ee2e80..e4ccb3fc 100644
--- a/src/streamlink/plugins/brittv.py
+++ b/src/streamlink/plugins/brittv.py
@@ -17,15 +17,16 @@ class BritTV(Plugin):
         return cls.url_re.match(url) is not None
 
     def _get_streams(self):
-        res = http.get(self.url, headers={"User-Agent": useragents.CHROME})
+        http.headers.update({"User-Agent": useragents.CHROME})
+        res = http.get(self.url)
         m = self.js_re.search(res.text)
         if m:
             self.logger.debug("Found js key: {0}", m.group(1))
             js_url = m.group(0)
-            res = http.get(urljoin(self.url, js_url))
+            res = http.get(urljoin(self.url, js_url), headers={"Referer": self.url})
 
             self.logger.debug("Looking for stream URL...")
-            for url in self.player_re.findall(res.text):
+            for _, url in self.player_re.findall(res.text):
                 if "adblock" not in url:
                     yield "live", HLSStream(self.session, url)
 
diff --git a/src/streamlink/plugins/itvplayer.py b/src/streamlink/plugins/itvplayer.py
index 19c97bc6..90797db5 100644
--- a/src/streamlink/plugins/itvplayer.py
+++ b/src/streamlink/plugins/itvplayer.py
@@ -38,7 +38,7 @@ class ITVPlayer(Plugin):
     def production_id(self):
         if self._stream not in CHANNEL_MAP:
             res = http.get(self.url, verify=False)
-            production_id_match = re.findall(r"&productionId=(.*?)['&\"]", res.text, flags=re.DOTALL)
+            production_id_match = re.findall(r'data-video-production-id="(.+?)"', res.text)
             if production_id_match:
                 return unquote(production_id_match[0])
             else:
@@ -53,14 +53,14 @@ class ITVPlayer(Plugin):
 
         headers = {'Content-Length': '{0:d}'.format(len(soap_message)),
                    'Content-Type': 'text/xml; charset=utf-8',
-                   'Host': 'mercury.itv.com',
+                   'Host': 'secure-mercury.itv.com',
                    'Origin': 'http://www.itv.com',
                    'Referer': 'http://www.itv.com/Mercury/Mercury_VideoPlayer.swf?v=null',
                    'SOAPAction': "http://tempuri.org/PlaylistService/GetPlaylist",
                    'User-Agent': ITV_PLAYER_USER_AGENT,
                    "X-Requested-With": "ShockwaveFlash/16.0.0.305"}
 
-        res = http.post("http://mercury.itv.com/PlaylistService.svc?wsdl",
+        res = http.post("https://secure-mercury.itv.com/PlaylistService.svc?wsdl",
                         headers=headers,
                         data=soap_message)
 
diff --git a/src/streamlink_cli/argparser.py b/src/streamlink_cli/argparser.py
index 5ef90c00..31b0b6ee 100644
--- a/src/streamlink_cli/argparser.py
+++ b/src/streamlink_cli/argparser.py
@@ -1403,6 +1403,28 @@ plugin.add_argument(
     A pixiv.net account password to use with --pixiv-username
     """
 )
+plugin.add_argument(
+    "--abweb-username",
+    metavar="USERNAME",
+    help="""
+    The username associated with your ABweb account, required to access any ABweb stream.
+    """
+)
+plugin.add_argument(
+    "--abweb-password",
+    metavar="PASSWORD",
+    help="""
+    A ABweb account password to use with --abweb-username.
+    """
+)
+plugin.add_argument(
+    "--abweb-purge-credentials",
+    action="store_true",
+    help="""
+    Purge cached ABweb credentials to initiate a new session
+    and reauthenticate.
+    """
+)
 
 # Deprecated options
 stream.add_argument(
diff --git a/src/streamlink_cli/main.py b/src/streamlink_cli/main.py
index 0bcec76c..0ef080e9 100644
--- a/src/streamlink_cli/main.py
+++ b/src/streamlink_cli/main.py
@@ -990,6 +990,19 @@ def setup_plugin_options():
     if pixiv_password:
         streamlink.set_plugin_option("pixiv", "password", pixiv_password)
 
+    if args.abweb_username:
+        streamlink.set_plugin_option("abweb", "username", args.abweb_username)
+    if args.abweb_username and not args.abweb_password:
+        abweb_password = console.askpass("Enter ABweb password: ")
+    else:
+        abweb_password = args.abweb_password
+    if abweb_password:
+        streamlink.set_plugin_option("abweb", "password", abweb_password)
+
+    if args.abweb_purge_credentials:
+        streamlink.set_plugin_option("abweb", "purge_credentials",
+                                     args.abweb_purge_credentials)
+
 
 def check_root():
     if hasattr(os, "getuid"):


Test patch:
diff --git a/tests/test_plugin_abweb.py b/tests/test_plugin_abweb.py
new file mode 100644
index 00000000..3bab6f57
--- /dev/null
+++ b/tests/test_plugin_abweb.py
@@ -0,0 +1,20 @@
+import unittest
+
+from streamlink.plugins.abweb import ABweb
+
+
+class TestPluginABweb(unittest.TestCase):
+    def test_can_handle_url(self):
+        should_match = [
+            'http://www.abweb.com/bis-tv-online/bistvo-tele-universal.aspx?chn=ab1',
+            'http://www.abweb.com/BIS-TV-Online/bistvo-tele-universal.aspx?chn=ab1',
+            'http://www.abweb.com/BIS-TV-Online/bistvo-tele-universal.aspx?chn=luckyjack',
+        ]
+        for url in should_match:
+            self.assertTrue(ABweb.can_handle_url(url))
+
+        should_not_match = [
+            'http://www.abweb.com',
+        ]
+        for url in should_not_match:
+            self.assertFalse(ABweb.can_handle_url(url))