Repository: streamlink/streamlink

Issue:
How to Stream Akamai (?) Video from Senate Foreign Relations Committee Website
## Issue

<!-- Replace [ ] with [x] in order to check the box -->
- [x] This is not a bug report, feature request, or plugin issue/request.
- [x] I have read the contribution guidelines.

### Description

I am attempting to load archived video and live video streams of the US Senate Foreign Relations Committee ([Example](https://www.foreign.senate.gov/hearings/business-meeting-082218)) in VLC via streamlink. It appears their video streaming is done via Akamai and AMP v4.90.9 from right-clicking the player embedded in provided examples.

### Expected / Actual behavior

Ideally, pointing streamlink to the hearing page containing an embedded player would automagically identify available streams and open the best quality stream in VLC:

`streamlink "https://www.foreign.senate.gov/hearings/business-meeting-082218" best`   

What actually happens is that streamlink fails with an error `error: No plugin can handle URL: https://www.foreign.senate.gov/hearings/business-meeting-082218`

### Reproduction steps / Explicit stream URLs to test

I have discovered that the URL for their embedded player gets a little closer ([Example](http://www.senate.gov/isvp/?comm=foreign&type=live&filename=foreign082218)), but streamlink still doesn't find any streams at this URL. That is, until I replace https:// with *akamaihd://*:

`streamlink "akamaihd://www.senate.gov/isvp/?comm=foreign&type=live&filename=foreign082218"`   

Results in:

```[cli][info] Found matching plugin akamaihd for URL akamaihd://www.senate.gov/isvp/?comm=foreign&type=live&filename=foreign082218
Available streams: live (worst, best)
```

When I specify "best" and run the command again, the result is:

```[cli][info] Found matching plugin akamaihd for URL akamaihd://www.senate.gov/isvp/?comm=foreign&type=live&filename=foreign082218
[cli][info] Available streams: live (worst, best)
[cli][info] Opening stream: live (akamaihd)
[cli][error] Try 1/1: Could not open stream <AkamaiHDStream('http://www.senate.gov/isvp/?comm=foreign&type=live&filename=foreign082218', swf=None)> (Could not open stream: Unable to open URL: http://www.senate.gov/isvp/ (503 Server Error: Service Unavailable for url: http://www.senate.gov/isvp/?fp=LNX+11%2C1%2C102%2C63&r=PPJRC&g=VFVQOIITOQRF&v=2.5.8))
error: Could not open stream <AkamaiHDStream('http://www.senate.gov/isvp/?comm=foreign&type=live&filename=foreign082218', swf=None)>, tried 1 times, exiting
```
### Logs

```
[cli][debug] OS:         macOS 10.13.6
[cli][debug] Python:     2.7.15
[cli][debug] Streamlink: 0.14.2
[cli][debug] Requests(2.19.1), Socks(1.6.7), Websocket(0.48.0)
[cli][info] Found matching plugin akamaihd for URL akamaihd://www.senate.gov/isvp/?comm=foreign&type=live&filename=foreign082218
[plugin.akamaihd][debug] URL=http://www.senate.gov/isvp/?comm=foreign&type=live&filename=foreign082218; params={}
[cli][info] Available streams: live (worst, best)
[cli][info] Opening stream: live (akamaihd)
[stream.akamaihd][debug] Opening host=http://www.senate.gov streamname=isvp/
[cli][error] Try 1/1: Could not open stream <AkamaiHDStream('http://www.senate.gov/isvp/?comm=foreign&type=live&filename=foreign082218', swf=None)> (Could not open stream: Unable to open URL: http://www.senate.gov/isvp/ (503 Server Error: Service Unavailable for url: http://www.senate.gov/isvp/?fp=LNX+11%2C1%2C102%2C63&r=VREJP&g=JOHWJGPLHFNA&v=2.5.8))
error: Could not open stream <AkamaiHDStream('http://www.senate.gov/isvp/?comm=foreign&type=live&filename=foreign082218', swf=None)>, tried 1 times, exiting
```

### Additional comments, screenshots, etc.

It appears the Senate Foreign Relations Committee uses the date of the hearing as the main reference ID. Since the committee only meets once a day, there's only ever one stream on any given day.

There's possibly useful information in the source of each player page. In the source code for the example link I've used above, there's this snippet:

```
<a id="watch-live-now" class="small btn" href="javascript:openVideoWin('/hearings/watch?hearingid=AA0FF459-5056-A066-60A0-8B11F704E85E');">Open New Window</a>
```

Also, from the source code the video player code can have a lot of parameters specified:

```
https://www.senate.gov/isvp/?comm=foreign&type=arch&stt=21:50&filename=foreign082218&auto_play=false&wmode=transparent&poster=https%3A%2F%2Fwww%2Eforeign%2Esenate%2Egov%2Fthemes%2Fforeign%2Fimages%2Fvideo%2Dposter%2Dflash%2Dfit%2Epng
```

[Love Streamlink? Please consider supporting our collective. Thanks!](https://opencollective.com/streamlink/donate)


Discussion:
You can watch this directly, stream url appears to be static.

https://foreign-f.akamaihd.net/i/foreign082218_1@76451/master.m3u8?

```
[cli][info] Found matching plugin hls for URL https://foreign-f.akamaihd.net/i/foreign082218_1@76451/master.m3u8?" best
Available streams: 360p_alt (worst), 360p (best)
```

```
C:\>youtube-dl.exe -v --hls-prefer-native --no-check-certificate --no-overwrites "https://foreign-f.akamaihd.net/i/foreign082218_1@76451/master.m3u8?"
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['-v', '--hls-prefer-native', '--no-check-certificate', '--no-overwrites', 'https://foreign-f.akamaihd.net/i/foreign082218_1@76451/master.m3u8?']
[debug] Encodings: locale cp1252, fs mbcs, out cp850, pref cp1252
[debug] youtube-dl version 2018.08.28
[debug] Python version 3.4.4 (CPython) - Windows-10-10.0.17134
[debug] exe versions: ffmpeg n4.0.2-ffmpeg-windows-build-helpers, ffprobe n4.0.2-ffmpeg-windows-build-helpers, rtmpdump 2.4
[debug] Proxy map: {}
[generic] master: Requesting header
WARNING: Could not send HEAD request to https://foreign-f.akamaihd.net/i/foreign082218_1@76451/master.m3u8?: HTTP Error 405: Method Not Allowed
[generic] master: Downloading webpage
[generic] master: Downloading m3u8 information
[debug] Default format spec: bestvideo+bestaudio/best
[debug] Invoking downloader on 'https://foreign-f.akamaihd.net/i/foreign082218_1@76451/index_356_av-b.m3u8?sd=10&rebase=on'
[hlsnative] Downloading m3u8 manifest
[hlsnative] Total fragments: 734
[download] Destination: master-master.mp4
[download]   2.5% of ~300.16MiB at  1.77MiB/s ETA 04:51    [download] Got server HTTP error: HTTP Error 404: Not Found. Retrying fragment 19 (attempt 1 of 10)...
[download] Got server HTTP error: HTTP Error 404: Not Found. Retrying fragment 19 (attempt 2 of 10)...
[download] Got server HTTP error: HTTP Error 404: Not Found. Retrying fragment 19 (attempt 3 of 10)...
[download] Got server HTTP error: HTTP Error 404: Not Found. Retrying fragment 19 (attempt 4 of 10)...
[download] Got server HTTP error: HTTP Error 404: Not Found. Retrying fragment 19 (attempt 5 of 10)...
[download] Got server HTTP error: HTTP Error 404: Not Found. Retrying fragment 19 (attempt 6 of 10)...
[download] Got server HTTP error: HTTP Error 404: Not Found. Retrying fragment 19 (attempt 7 of 10)...
[download] Got server HTTP error: HTTP Error 404: Not Found. Retrying fragment 19 (attempt 8 of 10)...
[download] Got server HTTP error: HTTP Error 404: Not Found. Retrying fragment 19 (attempt 9 of 10)...
[download] Got server HTTP error: HTTP Error 404: Not Found. Retrying fragment 19 (attempt 10 of 10)...
[download] Skipping fragment 19...
[download]   7.5% of ~300.27MiB at  1.59MiB/s ETA 05:02
```

<a href='https://postimg.cc/image/ehaqjlf5t/' target='_blank'><img src='https://s8.postimg.cc/ehaqjlf5t/Screen_Shot.jpg' border='0' alt='Screen_Shot'/></a>


This is helpful and works—could you explain how you arrived at this being the static URL? I want to reverse engineer this into an Automator action if possible—to convert a URL of pages with the embedded player into the appropriate stream URL to load in VLC via streamlink. Specifically, where does "76451" come from in your working URL? What function does that serve in loading the stream? And the _01 after the date of the hearing? Why does the URL end with master.m3u8?
I played the video in the morning and later in the afternoon, url was the same so it must be static.
To get stream url run firefox and install extension HLS stream detector,

https://addons.mozilla.org/en-GB/firefox/addon/hls-stream-detector/

Open main url, start video in embeded player and stream detector will politely pop up with stream url.
This is as simple as you can get. As you said it works, so I didn't look more into that.

Resolving patch:
diff --git a/src/streamlink/plugins/senategov.py b/src/streamlink/plugins/senategov.py
new file mode 100644
--- /dev/null
+++ b/src/streamlink/plugins/senategov.py
@@ -0,0 +1,113 @@
+import re
+import logging
+
+from streamlink.plugin import Plugin
+from streamlink.plugin.api import useragents
+from streamlink.plugin.api.utils import itertags
+from streamlink.stream import HLSStream
+from streamlink.utils import parse_json
+from streamlink.compat import urlparse, parse_qsl
+from streamlink.utils.times import hours_minutes_seconds
+
+log = logging.getLogger(__name__)
+
+
+class SenateGov(Plugin):
+    url_re = re.compile(r"""https?://(?:.+\.)?senate\.gov/(isvp)?""")
+    streaminfo_re = re.compile(r"""var\s+streamInfo\s+=\s+new\s+Array\s*\(\s*(\[.*\])\);""")
+    stt_re = re.compile(r"""^(?:(?P<hours>\d+):)?(?P<minutes>\d+):(?P<seconds>\d+)$""")
+    url_lookup = {
+        "ag": ["76440", "https://ag-f.akamaihd.net"],
+        "aging": ["76442", "https://aging-f.akamaihd.net"],
+        "approps": ["76441", "https://approps-f.akamaihd.net"],
+        "armed": ["76445", "https://armed-f.akamaihd.net"],
+        "banking": ["76446", "https://banking-f.akamaihd.net"],
+        "budget": ["76447", "https://budget-f.akamaihd.net"],
+        "cecc": ["76486", "https://srs-f.akamaihd.net"],
+        "commerce": ["80177", "https://commerce1-f.akamaihd.net"],
+        "csce": ["75229", "https://srs-f.akamaihd.net"],
+        "dpc": ["76590", "https://dpc-f.akamaihd.net"],
+        "energy": ["76448", "https://energy-f.akamaihd.net"],
+        "epw": ["76478", "https://epw-f.akamaihd.net"],
+        "ethics": ["76449", "https://ethics-f.akamaihd.net"],
+        "finance": ["76450", "https://finance-f.akamaihd.net"],
+        "foreign": ["76451", "https://foreign-f.akamaihd.net"],
+        "govtaff": ["76453", "https://govtaff-f.akamaihd.net"],
+        "help": ["76452", "https://help-f.akamaihd.net"],
+        "indian": ["76455", "https://indian-f.akamaihd.net"],
+        "intel": ["76456", "https://intel-f.akamaihd.net"],
+        "intlnarc": ["76457", "https://intlnarc-f.akamaihd.net"],
+        "jccic": ["85180", "https://jccic-f.akamaihd.net"],
+        "jec": ["76458", "https://jec-f.akamaihd.net"],
+        "judiciary": ["76459", "https://judiciary-f.akamaihd.net"],
+        "rpc": ["76591", "https://rpc-f.akamaihd.net"],
+        "rules": ["76460", "https://rules-f.akamaihd.net"],
+        "saa": ["76489", "https://srs-f.akamaihd.net"],
+        "smbiz": ["76461", "https://smbiz-f.akamaihd.net"],
+        "srs": ["75229", "https://srs-f.akamaihd.net"],
+        "uscc": ["76487", "https://srs-f.akamaihd.net"],
+        "vetaff": ["76462", "https://vetaff-f.akamaihd.net"],
+    }
+
+    hls_url = "{base}/i/{filename}_1@{number}/master.m3u8?"
+    hlsarch_url = "https://ussenate-f.akamaihd.net/i/{filename}.mp4/master.m3u8"
+
+    @classmethod
+    def can_handle_url(cls, url):
+        return cls.url_re.match(url) is not None
+
+    def _isvp_to_m3u8(self, url):
+        qs = dict(parse_qsl(urlparse(url).query))
+        if "comm" not in qs:
+            log.error("Missing `comm` value")
+        if "filename" not in qs:
+            log.error("Missing `filename` value")
+
+        d = self.url_lookup.get(qs['comm'])
+        if d:
+            snumber, baseurl = d
+            stream_url = self.hls_url.format(filename=qs['filename'], number=snumber, base=baseurl)
+        else:
+            stream_url = self.hlsarch_url.format(filename=qs['filename'])
+
+        return stream_url, self.parse_stt(qs.get('stt', 0))
+
+    def _get_streams(self):
+        self.session.http.headers.update({
+            "User-Agent": useragents.CHROME,
+        })
+        m = self.url_re.match(self.url)
+        if m and not m.group(1):
+            log.debug("Searching for ISVP URL")
+            isvp_url = self._get_isvp_url()
+        else:
+            isvp_url = self.url
+
+        if not isvp_url:
+            log.error("Could not find the ISVP URL")
+            return
+        else:
+            log.debug("ISVP URL: {0}".format(isvp_url))
+
+        stream_url, start_offset = self._isvp_to_m3u8(isvp_url)
+        log.debug("Start offset is: {0}s".format(start_offset))
+        return HLSStream.parse_variant_playlist(self.session, stream_url, start_offset=start_offset)
+
+    def _get_isvp_url(self):
+        res = self.session.http.get(self.url)
+        for iframe in itertags(res.text, 'iframe'):
+            m = self.url_re.match(iframe.attributes.get('src'))
+            return m and m.group(1) is not None and iframe.attributes.get('src')
+
+    @classmethod
+    def parse_stt(cls, param):
+        m = cls.stt_re.match(param)
+        if m:
+            return int(m.group('hours') or 0) * 3600 + \
+                   int(m.group('minutes')) * 60 + \
+                   int(m.group('seconds'))
+        else:
+            return 0
+
+
+__plugin__ = SenateGov


Test patch:
diff --git a/tests/plugins/test_senategov.py b/tests/plugins/test_senategov.py
new file mode 100644
--- /dev/null
+++ b/tests/plugins/test_senategov.py
@@ -0,0 +1,18 @@
+from streamlink.plugins.senategov import SenateGov
+import unittest
+
+
+class TestPluginSenateGov(unittest.TestCase):
+    def test_can_handle_url(self):
+        # should match
+        self.assertTrue(SenateGov.can_handle_url("https://www.foreign.senate.gov/hearings/business-meeting-082218"))
+        self.assertTrue(SenateGov.can_handle_url("https://www.senate.gov/isvp/?comm=foreign&type=arch&stt=21:50&filename=foreign082218&auto_play=false&wmode=transparent&poster=https%3A%2F%2Fwww%2Eforeign%2Esenate%2Egov%2Fthemes%2Fforeign%2Fimages%2Fvideo-poster-flash-fit%2Epng"))
+
+        # shouldn't match
+        self.assertFalse(SenateGov.can_handle_url("http://www.tvcatchup.com/"))
+        self.assertFalse(SenateGov.can_handle_url("http://www.youtube.com/"))
+
+    def test_stt_parse(self):
+        self.assertEqual(600, SenateGov.parse_stt("10:00"))
+        self.assertEqual(3600, SenateGov.parse_stt("01:00:00"))
+        self.assertEqual(70, SenateGov.parse_stt("1:10"))