Repository: streamlink/streamlink

Issue:
Mytf1.fr : TF 1, TMC, NT 1, HD 1, LCI
http://www.tf1.fr/direct
Channels on TF 1 site require only a free login and French proxy. They use HDS and HLS. 
LCi doesn't require login just proxy : http://www.lci.fr/direct/

But it would be of course better if you can bypass login. 


Discussion:
Can take a look, do you know any good proxy for lci? I can use unlocator for tf1 but I don't know any good http proxies for France. 
You can try here : 
http://spys.ru/free-proxy-list/FR/
http://www.freeproxylists.net/?c=FR&pt=&pr=&a%5B%5D=0&a%5B%5D=1&a%5B%5D=2&u=0
https://www.proxynova.com/proxy-server-list/country-fr/


@karlo2105 wrote: 

> LCi doesn't require login, just proxy: 
> http://www.lci.fr/direct/

@beardypig wrote: 

> do you know any good proxy for lci?

I am not inside France, but I find accessing the live (direct) stream from Greece 
does not need a French proxy; the "HDS Link detector" Firefox addon reveals 
the following HDS manifest:

`http://lcilivhlshdslive-lh.akamaihd.net/z/lci_1@301585/manifest.f4m?hdnea=st=1483822658~exp=1483824458~acl=/*~hmac=1bd7f429b37a778f80a7654e2f2db1d7a813fe5c0d050acb5279451fbd108c82&hdcore=2.11.3&g=MHIRXZLQFQZI` 

which, of course, has a limited lifespan; it contains five quality variants, with bitrates: 
240k, 496k, 896k, 1296k & 2596k

@beardypig 
I've been meaning to ask, you, of course, are not obliged in any way to reply, 
but what is your physical location?

Regards

I think through HLS, they provide 720p stream. 
@karlo2105 wrote: 

> I think through HLS, they provide 720p stream

The 2596k AdobeHDS stream is **720p**, too. 
But I have stumbled upon a problem; 
AdobeHDS.php would download streams 
from the manifest fine, 
but streamlink fails to, citing: 
`error: This manifest requires the 'pvswf' parameter to verify the SWF`
```
streamlink "hds://http://lcilivhlshdslive-lh.akamaihd.net/z/lci_1@301585/manifest.f4m?hdnea=st=1483827331~exp=1483829131~acl=/*~hmac=d0f781049713946d9ce00371c599a0abec582cb6b86ab0466971958d408aeb0d&hdcore=2.11.3&g=VXPKOBLDFCWY pvswf=http://www.wat.tv/images/v70/PlayerWat.swf?rev=04.00.861" best
```
Works for me :)
@beardypig 

Thanks! 
Where exactly in streamlink's documentation is the `pvswf parameter` detailed? 
I've searched, but to no avail :-(
BTW, this HDS parameter reminds me of the `--swfUrl` one used in RTMP streams...
@Vangelis66 it's not really documented, there is a section about passing extra parameters to streams here https://streamlink.github.io/cli.html#playing-built-in-streaming-protocols-directly. There is an example for rtmp but it works for everything I think. You have to know the `pvswf` parameter though... 
LCI is only sometimes geo locked, it should be quite straightforward to make a plugin :-)
@Vangelis66 
HDS is successor of RTMP. 

Source for HDS : 
http://www.wat.tv/get/androidliveconnect/591997?getURL=1
http://www.wat.tv/get/androidlivelci/591997?getURL=1
http://www.wat.tv/get/androidlivent1/591997?getURL=1
http://www.wat.tv/get/androidlivehd1/591997?getURL=1
http://www.wat.tv/get/androidlivetmc/591997?getURL=1
@karlo2105 wrote: 

 > Source for HDS :
> (snip)
> http://www.wat.tv/get/androidlivelci/591997?getURL=1

Hello Karlo :smile:
I have only tested LCI on a desktop browser, the manifest URIs sniffed either via 
Firefox's web console or (more easily) via "HDS Link Detector" addon are of the form 

`http://lcilivhlshdslive-lh.akamaihd.net/z/lci_1@301585/manifest.f4m?hdnea=st=1483843039~exp=1483844839~acl=/*~hmac=2afc34bd7b2a0c8c1782d4f2b48eb2732f0e99a3cb8dd7701e9e2102f727daf6&hdcore=2.11.3&g=TTSHPFEMNPXT`

Those are very short-lived (less than 30mins), but for as long as they are valid they are 
downloadable either via AdobeHDS.php script or via livestreamer/streamlink (the pvswf value 
must be supplied as per previous post in this thread). 

I don't have an android device to test, but opening 
http://www.wat.tv/get/androidlivelci/591997?getURL=1
in a desktop browser tab yields a manifest of this form: 

`http://lcilivhlshdslive-lh.akamaihd.net/z/lci_1@301585/manifest.f4m?hdnea=st=1483844264~exp=1483846064~acl=/*~hmac=e5b561cc06a10cfde5fffe135f6e16c225ce34fe6338c692cd4aa7b87c3725cc`

These ALWAYS get an **Access Denied** response when 
a download is attempted well within the manifest's validity; 
probably because the manifests generated by the link you 
posted are missing the final part of the form: 
`&hdcore=2.11.3&g=TTSHPFEMNPXT`

But if you modify a bit the manifest URI generated by your link, 
you get a valid (and downloadable) AppleHLS master playlist: 

http://www.wat.tv/get/androidlivelci/591997?getURL=1 => 

`http://lcilivhlshdslive-lh.akamaihd.net/z/lci_1@301585/manifest.f4m?hdnea=st=1483845377~exp=1483847177~acl=/*~hmac=095e53624d9d3cc9d07d9cceaf92379724fbcab66c7fab75bf76c81d370c2475` (403 Access Denied) 
If modified to
`http://lcilivhlshdslive-lh.akamaihd.net/i/lci_1@301585/master.m3u8?hdnea=st=1483845377~exp=1483847177~acl=/*~hmac=095e53624d9d3cc9d07d9cceaf92379724fbcab66c7fab75bf76c81d370c2475` => Valid playlist
```
#EXTM3U
#EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=240000,RESOLUTION=416x234,CODECS="avc1.66.30, mp4a.40.2"
http://lcilivhlshdslive-lh.akamaihd.net/i/lci_1@301585/index_240_av-p.m3u8?sd=10&rebase=on
#EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=264000,RESOLUTION=416x234,CODECS="avc1.66.30, mp4a.40.2"
http://lcilivhlshdslive-lh.akamaihd.net/i/lci_1@301585/index_240_av-b.m3u8?sd=10&rebase=on
#EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=496000,RESOLUTION=416x234,CODECS="avc1.66.30, mp4a.40.2"
http://lcilivhlshdslive-lh.akamaihd.net/i/lci_1@301585/index_496_av-p.m3u8?sd=10&rebase=on
#EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=464000,RESOLUTION=416x234,CODECS="avc1.66.30, mp4a.40.2"
http://lcilivhlshdslive-lh.akamaihd.net/i/lci_1@301585/index_496_av-b.m3u8?sd=10&rebase=on
#EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=896000,RESOLUTION=640x360,CODECS="avc1.77.30, mp4a.40.2"
http://lcilivhlshdslive-lh.akamaihd.net/i/lci_1@301585/index_896_av-p.m3u8?sd=10&rebase=on
#EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=896000,RESOLUTION=640x360,CODECS="avc1.66.30, mp4a.40.2"
http://lcilivhlshdslive-lh.akamaihd.net/i/lci_1@301585/index_896_av-b.m3u8?sd=10&rebase=on
#EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=1296000,RESOLUTION=1080x576,CODECS="avc1.77.30, mp4a.40.2"
http://lcilivhlshdslive-lh.akamaihd.net/i/lci_1@301585/index_1296_av-p.m3u8?sd=10&rebase=on
#EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=1296000,RESOLUTION=1024x576,CODECS="avc1.77.30, mp4a.40.2"
http://lcilivhlshdslive-lh.akamaihd.net/i/lci_1@301585/index_1296_av-b.m3u8?sd=10&rebase=on
#EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=2595000,RESOLUTION=1280x720,CODECS="avc1.77.30, mp4a.40.2"
http://lcilivhlshdslive-lh.akamaihd.net/i/lci_1@301585/index_2596_av-p.m3u8?sd=10&rebase=on
#EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=2595000,RESOLUTION=1280x720,CODECS="avc1.77.30, mp4a.40.2"
http://lcilivhlshdslive-lh.akamaihd.net/i/lci_1@301585/index_2596_av-b.m3u8?sd=10&rebase=on
```
As you can see, for each resolution there exist two (or more) streams: 
234p: 240k, 264k, 464k, 496k
360p: 896k-1, 896k-2
576p: 1296k-1, 1296k-2
720p: 2595k-1, 2595k-2
so this could present a small problem in creating the plugin... 

But I am sure @beardypig knows already all this info you and I are posting, 
probably feeling a bit amused at us newbies trying to figure out how things work... 😺 

All have a nice Sunday (deep-freeze over here at -8 Celsius)
Thanks for plugins. 
Some HDS streams are not working with proxy. 

TMC
```
livestreamer "http://www.tf1.fr/nt1/direct" 840k_hds
[cli][info] Found matching plugin tf1 for URL http://www.tf1.fr/tmc/direct
[cli][info] Available streams: 440k_hds, 840k_hds, 64k (worst), 440k, 640k, 840k, 1240k (best)
[cli][info] Opening stream: 840k_hds (hds)
[stream.hds][error] Failed to open fragment 1-23649054: Unable to open URL: http://tmchdslive-f.akamaihd.net/z/live_1@72268/840_bf2d3ed44cf2b891-p_Seg1-Frag23649054 (403 Client Error: Forbidden)
```

NT 1
```
livestreamer "http://www.tf1.fr/nt1/direct" 840k_hds
[cli][info] Found matching plugin tf1 for URL http://www.tf1.fr/nt1/direct
[cli][info] Available streams: 440k_hds, 840k_hds, 64k (worst), 440k, 640k, 840k, 1240k (best)
[cli][info] Opening stream: 840k_hds (hds)
[stream.hds][error] Failed to open fragment 1-23649136: Unable to open URL: http
://nt1hdslive-f.akamaihd.net/z/live_1@72464/840_487251dc402e69b8-p_Seg1-Frag2364
9136 (403 Client Error: Forbidden)
```

HD 1 

```
livestreamer "http://www.tf1.fr/hd1/direct" best
[cli][info] Found matching plugin tf1 for URL http://www.tf1.fr/hd1/direct
error: Unable to open URL: http://hd1livhdsandliv-lh.akamaihd.net/z/andlive_1@90
743/manifest.f4m?hdnea=st=1483915819~exp=1483917619~acl=/*~hmac=3c8731afcfae9226
e48bffa438b86cec873bb554af367c12e33d811b3fb85108 (404 Client Error: Not Found)


```
Do the stream work in a browser with the proxy? 
Yes it does.
Take a look. 

http://www.wat.tv/images/v70/PlayerLite.swf?videoId=L_TMC
`http://tmclivhdsweblive-lh.akamaihd.net/z/live_1@91207/manifest.f4m?hdnea=st=1483918400~exp=1483920200~acl=/*~hmac=228208d07495af8c38d7e84f2d1cb744a2ed3909ffabd1c0cf169280c2106216&`hdcore=2.11.3&g=PVHIWKMOGQRH
http://www.wat.tv/images/v70/PlayerLite.swf?videoId=L_NT1
`http://nt1livhdsweb-lh.akamaihd.net/z/live_1@90590/manifest.f4m?hdnea=st=1483918281~exp=1483920081~acl=/*~hmac=b7944375268a2d6e717d93e3b8788fed7e41d26b4948887e60be039210d40d9f&hdcore=2.11.3&g=VVQMAMHFJBVH&hdcore=3.1.0`
http://www.wat.tv/images/v70/PlayerLite.swf?videoId=L_HD1
`http://hd1livhdsweblive-lh.akamaihd.net/z/weblive_1@90467/manifest.f4m?hdnea=st=1483918423~exp=1483920223~acl=/*~hmac=f8e5823edc3e82247a523c35921a1a5809815061bc6072bc0edb8e7c10807007&hdcore=2.11.3&g=WOKQQUMWRNDO`

I guess you miss that part "&hdcore=2.11.3&g=WOKQQUMWRNDO"
You can also add that parameter on TF 1 and LCI in case they add it to their HDS address.
Looks like there is some extra verification some of the time for some of the streams. Need to work out how the `g` argument is calculated...

Resolving patch:
diff --git a/src/streamlink/plugins/tf1.py b/src/streamlink/plugins/tf1.py
--- a/src/streamlink/plugins/tf1.py
+++ b/src/streamlink/plugins/tf1.py
@@ -11,9 +11,9 @@ class TF1(Plugin):
     url_re = re.compile(r"https?://(?:www\.)?(?:tf1\.fr/(\w+)/direct|(lci).fr/direct)/?")
     embed_url = "http://www.wat.tv/embedframe/live{0}"
     embed_re = re.compile(r"urlLive.*?:.*?\"(http.*?)\"", re.MULTILINE)
-    api_url = "http://www.wat.tv/get/androidlive{0}/591997"
-    swf_url = "http://www.wat.tv/images/v70/PlayerWat.swf?rev=04.00.861"
-    hds_channel_remap = {"tf1": "connect"}
+    api_url = "http://www.wat.tv/get/{0}/591997"
+    swf_url = "http://www.wat.tv/images/v70/PlayerLite.swf"
+    hds_channel_remap = {"tf1": "androidliveconnect", "lci": "androidlivelci"}
     hls_channel_remap = {"lci": "LCI", "tf1": "V4"}
 
     @classmethod
@@ -21,13 +21,15 @@ def can_handle_url(cls, url):
         return cls.url_re.match(url) is not None
 
     def _get_hds_streams(self, channel):
-        channel = self.hds_channel_remap.get(channel, channel)
+        channel = self.hds_channel_remap.get(channel, "{0}live".format(channel))
         manifest_url = http.get(self.api_url.format(channel),
-                                params={"getURL": 1}).text
+                                params={"getURL": 1},
+                                headers={"User-Agent": useragents.FIREFOX}).text
 
         for s in HDSStream.parse_manifest(self.session,
                                           manifest_url,
-                                          pvswf=self.swf_url).items():
+                                          pvswf=self.swf_url,
+                                          headers={"User-Agent": useragents.FIREFOX}).items():
             yield s
 
     def _get_hls_streams(self, channel):
@@ -41,8 +43,11 @@ def _get_hls_streams(self, channel):
         if m:
             hls_stream_url = m.group(1)
 
-            for s in HLSStream.parse_variant_playlist(self.session, hls_stream_url).items():
-                yield s
+            try:
+                for s in HLSStream.parse_variant_playlist(self.session, hls_stream_url).items():
+                    yield s
+            except:
+                self.logger.error("Failed to load the HLS playlist for {0}", channel)
 
     def _get_streams(self):
         m = self.url_re.match(self.url)
diff --git a/src/streamlink/stream/hds.py b/src/streamlink/stream/hds.py
--- a/src/streamlink/stream/hds.py
+++ b/src/streamlink/stream/hds.py
@@ -2,8 +2,10 @@
 
 import base64
 import hmac
+import random
 import re
 import os.path
+import string
 
 from binascii import unhexlify
 from collections import namedtuple
@@ -68,11 +70,15 @@ def fetch(self, fragment, retries=None):
             return
 
         try:
+            request_params = self.stream.request_params.copy()
+            params = request_params.pop("params", {})
+            params.pop("g", None)
             return self.session.http.get(fragment.url,
                                          stream=True,
                                          timeout=self.timeout,
                                          exception=StreamError,
-                                         **self.stream.request_params)
+                                         params=params,
+                                         **request_params)
         except StreamError as err:
             self.logger.error("Failed to open fragment {0}-{1}: {2}",
                               fragment.segment, fragment.fragment, err)
@@ -442,6 +448,7 @@ def parse_manifest(cls, session, url, timeout=60, pvswf=None, is_akamai=False,
 
         if "akamaihd" in url or is_akamai:
             request_params["params"]["hdcore"] = HDCORE_VERSION
+            request_params["params"]["g"] = cls.cache_buster_string(12)
 
         res = session.http.get(url, exception=IOError, **request_params)
         manifest = session.http.xml(res, "manifest XML", ignore_ns=True,
@@ -586,3 +593,7 @@ def _pv_params(cls, session, pvswf, pv, **request_params):
         params.extend(parse_qsl(hdntl, keep_blank_values=True))
 
         return params
+
+    @staticmethod
+    def cache_buster_string(length):
+        return "".join([random.choice(string.ascii_uppercase) for i in range(length)])


Test patch: