Repository: streamlink/streamlink

Issue:
Bloomberg plugin stopped working
## Plugin Issue

- [x] This is a plugin issue and I have read the contribution guidelines.

### Description

Bloomberg plugin stopped working May 22. Now it displays "Failed to parse playlist: Missing #EXTM3U header" error message.

### Reproduction steps / Explicit stream URLs to test

streamlink -l debug https://www.bloomberg.com/live/us

### Log output

[cli][info] streamlink is running as root! Be careful!
[cli][debug] OS:         Linux-4.9.0-9-amd64-x86_64-with-debian-9.9
[cli][debug] Python:     2.7.13
[cli][debug] Streamlink: 1.1.1
[cli][debug] Requests(2.22.0), Socks(1.7.0), Websocket(0.56.0)
[cli][info] Found matching plugin bloomberg for URL https://www.bloomberg.com/live/us
[utils.l10n][debug] Language code: en_US
error: Failed to parse playlist: Missing #EXTM3U header


Discussion:
I can reproduce this. My browser is requesting https://d1g4mbj10o50ni.cloudfront.net/btv/desktop/us_live.m3u8 from https://www.bloomberg.com/live/us which still works. 

It would be useful to be able to get _more_ debugging information out of the plugin - what is it trying to download, what playlist does it think it's getting that doesn't have #EXTM3U

Update: pretty sure this is triggering an "are you a robot" check

```
$ streamlink -l debug https://www.bloomberg.com/live/us
[cli][debug] OS:         Linux-5.0.0-13-generic-x86_64-with-Ubuntu-19.04-disco
[cli][debug] Python:     3.7.3
[cli][debug] Streamlink: 1.1.1+27.gad0bee8.dirty
[cli][debug] Requests(2.21.0), Socks(1.6.7), Websocket(0.53.0)
[cli][info] Found matching plugin bloomberg for URL https://www.bloomberg.com/live/us
[plugin.bloomberg][debug] live_player_url is https://cdn.gotraffic.net/projector/v0.11.147/app.js
[plugin.bloomberg][debug] parsing m3u8 url at https://www.bloomberg.com/media-manifest/streams/phoenix-us.m3u8
[utils.l10n][debug] Language code: en_AU
[stream.hls_playlist][debug] First line of playlist was <!doctype html>
error: Failed to parse playlist: Missing #EXTM3U header
```

whereas if we chose https://liveproduseast.global.ssl.fastly.net/btv/desktop/us_live.m3u8 (second matching URL) I think it'd work.

This can probably be fixed by adding a User-Agent HTTP header to the HTTP requests. I haven't tested it though...
https://github.com/streamlink/streamlink/blob/1.1.1/src/streamlink/plugins/bloomberg.py#L91-L98
https://github.com/streamlink/streamlink/blob/1.1.1/src/streamlink/plugins/bloomberg.py#L110-L117
Example here:
https://github.com/streamlink/streamlink/blob/1.1.1/src/streamlink/plugins/welt.py#L74-L75

You don't need to add more debug logging, especially log calls which include HTTP response data, as you can simply use python debug tools to see what exactly is going on in detail while fixing/developing a plugin. Including response data can lead to other issues or could potentially be abused.
I understand the bit about not including response data, but I think including *some* debug logging is OK - I'm not using print(), it's only activated when -l debug is specified. The other thing is that the plugin finds multiple matching URLs but bails out after trying only one - perhaps a better behaviour would be to failover to another URL?
`self.session.http.headers.update({"User-Agent": useragents.CHROME})` at the start of _get_live_streams seems to work, but I can't work out why I can't get an `__init__(self)` method to work.
I added manually definition of user agent in beginning of _get_live_streams and during two days Bloomberg plugin works well.

Resolving patch:
diff --git a/src/streamlink/plugins/bloomberg.py b/src/streamlink/plugins/bloomberg.py
--- a/src/streamlink/plugins/bloomberg.py
+++ b/src/streamlink/plugins/bloomberg.py
@@ -1,11 +1,14 @@
-from functools import partial
+import logging
 import re
+from functools import partial
 
 from streamlink.plugin import Plugin
-from streamlink.plugin.api import validate
+from streamlink.plugin.api import validate, useragents
 from streamlink.stream import HDSStream, HLSStream, HTTPStream
 from streamlink.utils import parse_json, update_scheme
 
+log = logging.getLogger(__name__)
+
 
 class Bloomberg(Plugin):
     VOD_API_URL = 'https://www.bloomberg.com/api/embed?id={0}'
@@ -21,23 +24,26 @@ class Bloomberg(Plugin):
     }
 
     _url_re = re.compile(r'''
-        https?://www\.bloomberg\.com/(
-            news/videos/[^/]+/[^/]+ |
-            (?P<channel>live/(?:stream|emea|asia_stream|europe|us|asia)|audio)/?
+        https?://(?:www\.)?bloomberg\.com/
+        (?:
+            news/videos/[^/]+/[^/]+|
+            live/(?P<channel>.+)/?
         )
 ''', re.VERBOSE)
     _live_player_re = re.compile(r'{APP_BUNDLE:"(?P<live_player_url>.+?/app.js)"')
     _js_to_json_re = partial(re.compile(r'(\w+):(["\']|\d?\.?\d+,|true|false|\[|{)').sub, r'"\1":\2')
     _video_id_re = re.compile(r'data-bmmr-id=\\"(?P<video_id>.+?)\\"')
     _mp4_bitrate_re = re.compile(r'.*_(?P<bitrate>[0-9]+)\.mp4')
+    _preload_state_re = re.compile(r'window.__PRELOADED_STATE__\s*=\s*({.*});', re.DOTALL)
+    _live_stream_info_re = re.compile(r'12:.*t.exports=({.*})},{}],13:', re.DOTALL)
 
     _live_streams_schema = validate.Schema(
         validate.transform(_js_to_json_re),
         validate.transform(lambda x: x.replace(':.', ':0.')),
         validate.transform(parse_json),
-        validate.Schema(
-            {
-                'cdns': validate.all(
+        validate.Schema({
+            validate.text: {
+                validate.optional('cdns'): validate.all(
                     [
                         validate.Schema(
                             {
@@ -54,11 +60,21 @@ class Bloomberg(Plugin):
                     ],
                     validate.transform(lambda x: [i for y in x for i in y])
                 )
-            },
-            validate.get('cdns')
+            }
+        },
         )
     )
 
+    _channel_list_schema = validate.Schema(
+        validate.transform(parse_json),
+        {"live": {"channels": {"byChannelId": {
+            validate.text: validate.all({"liveId": validate.text}, validate.get("liveId"))
+        }}}},
+        validate.get("live"),
+        validate.get("channels"),
+        validate.get("byChannelId"),
+    )
+
     _vod_api_schema = validate.Schema(
         {
             'secureStreams': validate.all([
@@ -87,23 +103,39 @@ def _get_live_streams(self):
         match = self._url_re.match(self.url)
         channel = match.group('channel')
 
-        # Retrieve live player URL
-        res = self.session.http.get(self.PLAYER_URL)
-        match = self._live_player_re.search(res.text)
-        if match is None:
-            return []
-        live_player_url = update_scheme(self.url, match.group('live_player_url'))
-
-        # Extract streams from the live player page
-        res = self.session.http.get(live_player_url)
-        stream_datas = re.findall(r'{0}(?:_MINI)?:({{.+?}}]}}]}})'.format(self.CHANNEL_MAP[channel]), res.text)
-        streams = []
-        for s in stream_datas:
-            for u in self._live_streams_schema.validate(s):
-                if u not in streams:
-                    streams.append(u)
-
-        return streams
+        res = self.session.http.get(self.url, headers={
+            "authority": "www.bloomberg.com",
+            "upgrade-insecure-requests": "1",
+            "dnt": "1",
+            "accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3"
+        })
+        if "Are you a robot?" in res.text:
+            log.error("Are you a robot?")
+        match = self._preload_state_re.search(res.text)
+        if match:
+            live_ids = self._channel_list_schema.validate(match.group(1))
+            live_id = live_ids.get(channel)
+            if live_id:
+                log.debug("Found liveId = {0}".format(live_id))
+                # Retrieve live player URL
+                res = self.session.http.get(self.PLAYER_URL)
+                match = self._live_player_re.search(res.text)
+                if match is None:
+                    return []
+                live_player_url = update_scheme(self.url, match.group('live_player_url'))
+
+                # Extract streams from the live player page
+                log.debug("Player URL: {0}".format(live_player_url))
+                res = self.session.http.get(live_player_url)
+                match = self._live_stream_info_re.search(res.text)
+                if match:
+                    stream_info = self._live_streams_schema.validate(match.group(1))
+                    data = stream_info.get(live_id, {})
+                    return data.get('cdns', [])
+            else:
+                log.error("Could not find liveId for channel '{0}'".format(channel))
+
+        return []
 
     def _get_vod_streams(self):
         # Retrieve URL page and search for video ID
@@ -118,6 +150,7 @@ def _get_vod_streams(self):
         return streams
 
     def _get_streams(self):
+        self.session.http.headers.update({"User-Agent": useragents.CHROME})
         if '/news/videos/' in self.url:
             # VOD
             streams = self._get_vod_streams()
@@ -126,6 +159,7 @@ def _get_streams(self):
             streams = self._get_live_streams()
 
         for video_url in streams:
+            log.debug("Found stream: {0}".format(video_url))
             if '.f4m' in video_url:
                 for stream in HDSStream.parse_manifest(self.session, video_url).items():
                     yield stream


Test patch: