Repository: streamlink/streamlink

Issue:
BBC iplayer keyerror: goto
### Plugin Issue

- [x] This is a plugin issue and I have read the contribution guidelines.

### Description

When trying to connect to bbc iplayer, live stream of bbc channel, it crashes with KeyError: 'goto'


### Reproduction steps / Explicit stream URLs to test

1. i updated streamlink to latest from github: pip-3.6 install --user git+https://github.com/streamlink/streamlink.git
2. i ran the following cmd: streamlink --loglevel debug --http-no-ssl-verify --bbciplayer-username "username" --bbciplayer-password password --player-external-http --player-external-http-port 3280 https://www.bbc.co.uk/iplayer/live/bbctwo best


### Log output

```
[cli][debug] OS:         FreeBSD-12.0-RELEASE-p3-amd64-64bit-ELF
[cli][debug] Python:     3.6.8
[cli][debug] Streamlink: 1.2.0+12.g1fb9391
[cli][debug] Requests(2.21.0), Socks(1.6.7), Websocket(0.50.0)
[cli][info] Found matching plugin bbciplayer for URL https://www.bbc.co.uk/iplayer/live/bbctwo
[cli][debug] Plugin specific arguments:
[cli][debug]  --bbciplayer-username=***@*** (username)
[cli][debug]  --bbciplayer-password=******** (password)
[plugin.bbciplayer][info] A TV License is required to watch BBC iPlayer streams, see the BBC website for more information: https://www.bbc.co.uk/iplayer/help/tvlicence
Traceback (most recent call last):
  File "/home/username/bin/streamlink", line 11, in <module>
    load_entry_point('streamlink==1.2.0+12.g1fb9391', 'console_scripts', 'streamlink')()
  File "/home/username/.local/lib/python3.6/site-packages/streamlink_cli/main.py", line 1038, in main
    handle_url()
  File "/home/username/.local/lib/python3.6/site-packages/streamlink_cli/main.py", line 578, in handle_url
    streams = fetch_streams(plugin)
  File "/home/username/.local/lib/python3.6/site-packages/streamlink_cli/main.py", line 458, in fetch_streams
    sorting_excludes=args.stream_sorting_excludes)
  File "/home/username/.local/lib/python3.6/site-packages/streamlink/plugin/plugin.py", line 323, in streams
    ostreams = list(ostreams)
  File "/home/username/.local/lib/python3.6/site-packages/streamlink/plugins/bbciplayer.py", line 225, in _get_streams
    if not self.login(self.url):
  File "/home/username/.local/lib/python3.6/site-packages/streamlink/plugins/bbciplayer.py", line 199, in login
    http_nonce = self._extract_nonce(session_res)
  File "/home/username/.local/lib/python3.6/site-packages/streamlink/plugins/bbciplayer.py", line 112, in _extract_nonce
    final_url = urlparse(last_redirect_query['goto'])
KeyError: 'goto'

```


Discussion:
Yup, same here too.  I also updated to the latest git release via pip.
```
streamlink --config /home/mythtv-iptv-setup/conf/streamlinkrc -O bbc.co.uk/iplayer/live/bbcone 396p > /tmp/test.ts
[cli][info] streamlink is running as root! Be careful!
[cli][info] Found matching plugin bbciplayer for URL bbc.co.uk/iplayer/live/bbcone
[plugin.bbciplayer][info] A TV License is required to watch BBC iPlayer streams, see the BBC website for more information: https://www.bbc.co.uk/iplayer/help/tvlicence
Traceback (most recent call last):
  File "/home/mythtv-iptv-setup/utils/streamlink/bin/streamlink", line 10, in <module>
    sys.exit(main())
  File "/home/mythtv-iptv-setup/utils/streamlink/local/lib/python2.7/site-packages/streamlink_cli/main.py", line 1038, in main
    handle_url()
  File "/home/mythtv-iptv-setup/utils/streamlink/local/lib/python2.7/site-packages/streamlink_cli/main.py", line 578, in handle_url
    streams = fetch_streams(plugin)
  File "/home/mythtv-iptv-setup/utils/streamlink/local/lib/python2.7/site-packages/streamlink_cli/main.py", line 458, in fetch_streams
    sorting_excludes=args.stream_sorting_excludes)
  File "/home/mythtv-iptv-setup/utils/streamlink/local/lib/python2.7/site-packages/streamlink/plugin/plugin.py", line 323, in streams
    ostreams = list(ostreams)
  File "/home/mythtv-iptv-setup/utils/streamlink/local/lib/python2.7/site-packages/streamlink/plugins/bbciplayer.py", line 225, in _get_streams
    if not self.login(self.url):
  File "/home/mythtv-iptv-setup/utils/streamlink/local/lib/python2.7/site-packages/streamlink/plugins/bbciplayer.py", line 199, in login
    http_nonce = self._extract_nonce(session_res)
  File "/home/mythtv-iptv-setup/utils/streamlink/local/lib/python2.7/site-packages/streamlink/plugins/bbciplayer.py", line 112, in _extract_nonce
    final_url = urlparse(last_redirect_query['goto'])
KeyError: 'goto'
```
Appears to be an issue with the login flow. 
streamlink -V
streamlink 1.2.0+12.g1fb9391
OS: Linux Mint 19.2

streamlink http://www.bbc.co.uk/iplayer/live/bbcone 540p --bbciplayer-username xxxxx --bbciplayer-password xxxxx

[cli][info] Found matching plugin bbciplayer for URL http://www.bbc.co.uk/iplayer/live/bbcone
[plugin.bbciplayer][info] A TV License is required to watch BBC iPlayer streams, see the BBC website for more information: https://www.bbc.co.uk/iplayer/help/tvlicence
Traceback (most recent call last):
  File "/usr/local/bin/streamlink", line 11, in <module>
    load_entry_point('streamlink==1.2.0+12.g1fb9391', 'console_scripts', 'streamlink')()
  File "/usr/local/lib/python2.7/dist-packages/streamlink-1.2.0+12.g1fb9391-py2.7.egg/streamlink_cli/main.py", line 1038, in main
    handle_url()
  File "/usr/local/lib/python2.7/dist-packages/streamlink-1.2.0+12.g1fb9391-py2.7.egg/streamlink_cli/main.py", line 578, in handle_url
    streams = fetch_streams(plugin)
  File "/usr/local/lib/python2.7/dist-packages/streamlink-1.2.0+12.g1fb9391-py2.7.egg/streamlink_cli/main.py", line 458, in fetch_streams
    sorting_excludes=args.stream_sorting_excludes)
  File "/usr/local/lib/python2.7/dist-packages/streamlink-1.2.0+12.g1fb9391-py2.7.egg/streamlink/plugin/plugin.py", line 323, in streams
    ostreams = list(ostreams)
  File "/usr/local/lib/python2.7/dist-packages/streamlink-1.2.0+12.g1fb9391-py2.7.egg/streamlink/plugins/bbciplayer.py", line 225, in _get_streams
    if not self.login(self.url):
  File "/usr/local/lib/python2.7/dist-packages/streamlink-1.2.0+12.g1fb9391-py2.7.egg/streamlink/plugins/bbciplayer.py", line 199, in login
    http_nonce = self._extract_nonce(session_res)
  File "/usr/local/lib/python2.7/dist-packages/streamlink-1.2.0+12.g1fb9391-py2.7.egg/streamlink/plugins/bbciplayer.py", line 112, in _extract_nonce
    final_url = urlparse(last_redirect_query['goto'])
KeyError: 'goto'



The attached diff for bbciplayer.py seems to fix it.  I'm not suggesting it's the ideal solution, but it does work for me.

[bbciplayer.py.diff.txt](https://github.com/streamlink/streamlink/files/3569528/bbciplayer.py.diff.txt)

Comment edited to update supplied diff with minor improvements.
Thanks! The patch seems to have fix bbciplayer.py 
My distro (UbuntuStusio 18.04 LTS) has Streamlink 1.1.1 which exhibits the same problem, starting last week. I tried the patch, but it doesn't seem to work for me.
```
streamlink www.bbc.co.uk/iplayer/live/bbcnews 540p --bbciplayer-username xxxxxxxx --bbciplayer-password xxxxxxxx

[cli][info] Found matching plugin bbciplayer for URL www.bbc.co.uk/iplayer/live/bbcnews
[plugin.bbciplayer][info] A TV License is required to watch BBC iPlayer streams, see the BBC website for more information: https://www.bbc.co.uk/iplayer/help/tvlicence
Traceback (most recent call last):
  File "/usr/bin/streamlink", line 11, in <module>
    load_entry_point('streamlink==1.1.1', 'console_scripts', 'streamlink')()
  File "/usr/share/streamlink/streamlink_cli/main.py", line 1033, in main
    handle_url()
  File "/usr/share/streamlink/streamlink_cli/main.py", line 577, in handle_url
    streams = fetch_streams(plugin)
  File "/usr/share/streamlink/streamlink_cli/main.py", line 457, in fetch_streams
    sorting_excludes=args.stream_sorting_excludes)
  File "/usr/lib/python3/dist-packages/streamlink/plugin/plugin.py", line 323, in streams
    ostreams = list(ostreams)
  File "/usr/lib/python3/dist-packages/streamlink/plugins/bbciplayer.py", line 234, in _get_streams
    if not self.login(self.url):
  File "/usr/lib/python3/dist-packages/streamlink/plugins/bbciplayer.py", line 208, in login
    http_nonce = self._extract_nonce(session_res)
  File "/usr/lib/python3/dist-packages/streamlink/plugins/bbciplayer.py", line 116, in _extract_nonce
    final_url = urlparse(last_redirect_query['goto'])
KeyError: 'goto'
```
I would say uninstall your Streamlink 1.1.1, and then install the latest Streamlink 1.2.0, using git clone to get the 1.2.0 version, then using git pull to get the latest and greatest from time to time.

I hope the bbciplayer.py patch will be added a.s.a.p., since many could get dangerous withdrawal if they can't watch their BBC iPlayer. 

The patch worked for me, using Streamlink 1.2.0 (using git clone, then installing 'python setup.py install').   
> Sorry if stupid question - how do I apply this patch in Windows?

You could get a windows patch binary, from here for example:

http://gnuwin32.sourceforge.net/packages/patch.htm

It's probably just as quick to open bbciplayer.py in notepad and then manually change the _exrtract_nonce method to look like below:
```
    def _extract_nonce(cls, http_result):
        """
        Given an HTTP response from the sessino endpoint, extract the nonce, so we can "sign" requests with it.
        We don't really sign the requests in the traditional sense of a nonce, we just incude them in the auth requests.

        :param http_result: HTTP response from the bbc session endpoint.
        :type http_result: requests.Response
        :return: nonce to "sign" url requests with
        :rtype: string
        """

        # Extract the redirect URL from the last call
        last_redirect_url = http_result.history[-1].headers['Location']

        # Extract the nonce
        nonce = re.match(r'.*nonce=(.*?)&.*', last_redirect_url)

        # Return the nonce we can use for future queries
        return nonce.group(1)
```

Resolving patch:
diff --git a/src/streamlink/plugins/bbciplayer.py b/src/streamlink/plugins/bbciplayer.py
--- a/src/streamlink/plugins/bbciplayer.py
+++ b/src/streamlink/plugins/bbciplayer.py
@@ -31,8 +31,7 @@ class BBCiPlayer(Plugin):
     """, re.VERBOSE)
     mediator_re = re.compile(
         r'window\.__IPLAYER_REDUX_STATE__\s*=\s*({.*?});', re.DOTALL)
-    tvip_re = re.compile(r'channel"\s*:\s*{\s*"id"\s*:\s*"(\w+?)"')
-    tvip_master_re = re.compile(r'event_master_brand=(\w+?)&')
+    state_re = re.compile(r'window.__IPLAYER_REDUX_STATE__\s*=\s*({.*});')
     account_locals_re = re.compile(r'window.bbcAccount.locals\s*=\s*({.*?});')
     hash = base64.b64decode(b"N2RmZjc2NzFkMGM2OTdmZWRiMWQ5MDVkOWExMjE3MTk5MzhiOTJiZg==")
     api_url = "https://open.live.bbc.co.uk/mediaselector/6/select/version/2.0/mediaset/{platform}/vpid/{vpid}/format/json/atk/{vpid_hash}/asn/1/"
@@ -96,8 +95,8 @@ def _hash_vpid(cls, vpid):
     @classmethod
     def _extract_nonce(cls, http_result):
         """
-        Given an HTTP response from the sessino endpoint, extract the nonce, so we can "sign" requests with it.
-        We don't really sign the requests in the traditional sense of a nonce, we just incude them in the auth requests.
+        Given an HTTP response from the session endpoint, extract the nonce, so we can "sign" requests with it.
+        We don't really sign the requests in the traditional sense of a nonce, we just include them in the auth requests.
 
         :param http_result: HTTP response from the bbc session endpoint.
         :type http_result: requests.Response
@@ -105,16 +104,9 @@ def _extract_nonce(cls, http_result):
         :rtype: string
         """
 
-        # Extract the redirect URL from the last call
-        last_redirect_url = urlparse(http_result.history[-1].request.url)
-        last_redirect_query = dict(parse_qsl(last_redirect_url.query))
-        # Extract the nonce from the query string in the redirect URL
-        final_url = urlparse(last_redirect_query['goto'])
-        goto_url = dict(parse_qsl(final_url.query))
-        goto_url_query = parse_json(goto_url['state'])
-
-        # Return the nonce we can use for future queries
-        return goto_url_query['nonce']
+        p = urlparse(http_result.url)
+        d = dict(parse_qsl(p.query))
+        return d.get("nonce")
 
     def find_vpid(self, url, res=None):
         """
@@ -135,13 +127,15 @@ def find_vpid(self, url, res=None):
         return vpid
 
     def find_tvip(self, url, master=False):
-        log.debug("Looking for {0} tvip on {1}", "master" if master else "", url)
+        log.debug("Looking for {0} tvip on {1}".format("master" if master else "", url))
         res = self.session.http.get(url)
-        if master:
-            m = self.tvip_master_re.search(res.text)
-        else:
-            m = self.tvip_re.search(res.text)
-        return m and m.group(1)
+        m = self.state_re.search(res.text)
+        data = m and parse_json(m.group(1))
+        if data:
+            channel = data.get("channel")
+            if master:
+                return channel.get("masterBrand")
+            return channel.get("id")
 
     def mediaselector(self, vpid):
         urls = defaultdict(set)
@@ -183,6 +177,7 @@ def login(self, ptrt_url):
         :return: Whether authentication was successful
         :rtype: bool
         """
+
         def auth_check(res):
             return ptrt_url in ([h.url for h in res.history] + [res.url])
 


Test patch:
diff --git a/tests/plugins/test_bbciplayer.py b/tests/plugins/test_bbciplayer.py
--- a/tests/plugins/test_bbciplayer.py
+++ b/tests/plugins/test_bbciplayer.py
@@ -27,20 +27,8 @@ def test_vpid_hash(self):
 
     def test_extract_nonce(self):
         mock_nonce = "mock-nonce-nse"
-
-        last_response = Response()
-        last_response.request = Request('GET', "http://example.com/?" + urlencode(dict(
-            goto="http://example.com/?" + urlencode(dict(
-                state=json.dumps(dict(nonce=mock_nonce))
-            ))
-        )))
-
         mock_response = Response()
-        mock_response.history = [
-            Response(),  # Add some extra dummy responses in to make sure we always get the last
-            Response(),
-            last_response
-        ]
+        mock_response.url = "http://example.com/?" + urlencode(dict(nonce=mock_nonce))
 
         self.assertEqual(BBCiPlayer._extract_nonce(mock_response), mock_nonce)