Repository: streamlink/streamlink

Issue:
plugins.youtube Receiving "Unable to open URL: https://youtube.com/get_video_info" when trying to do --retry-streams after a few retry attempts.
<!--
Thanks for reporting a plugin issue!
USE THE TEMPLATE. Otherwise your plugin issue may be rejected.

First, see the contribution guidelines:
https://github.com/streamlink/streamlink/blob/master/CONTRIBUTING.md#contributing-to-streamlink

Plugin issues describe broken functionality within a plugin's code base, eg. the streaming site has made breaking changes, streams don't get resolved correctly, or authentication has stopped working, etc.

Also check the list of open and closed plugin issues:
https://github.com/streamlink/streamlink/issues?q=is%3Aissue+label%3A%22plugin+issue%22

Please see the text preview to avoid unnecessary formatting errors.
-->


## Plugin Issue

<!-- Replace the space character between the square brackets with an x in order to check the boxes -->
- [x] This is a plugin issue and I have read the contribution guidelines.
- [x] I am using the latest development version from the master branch.


### Description

<!-- Explain the plugin issue as thoroughly as you can. -->
When trying to use the --retry-streams options on a YouTube livestream that has not begun yet, shortly after running it with either 1-20 seconds between when it checks, it will start outputting

`[cli][error] Unable to open URL: https://youtube.com/get_video_info (404 Client Error: Not Found for url: https://www.youtube.com/get_video_info?video_id=yZ7CWjuMCe8&eurl=https%3A%2F%2Fyoutube.googleapis.com%2Fv%2FyZ7CWjuMCe8)`

It used to not do this at all until recently about 2-3 days ago on streamlink 2.0.0 and when it did, I tried updating to the current development build but it is still giving me the same thing.

So I am unable to setup streamlink to record for a stream that I wouldn't be around for since it would give that error even after the stream would start.

### Reproduction steps / Explicit stream URLs to test

<!-- How can we reproduce this? Please note the exact steps below using the list format supplied. If you need more steps please add them. -->

1. input `streamlink --retry-streams <#> -o <name.ts> <link of a stream that is planned to go live later>`


### Log output

<!--
DEBUG LOG OUTPUT IS REQUIRED for a plugin issue!
INCLUDE THE ENTIRE COMMAND LINE and make sure to **remove usernames and passwords**

Use the `--loglevel debug` parameter and avoid using parameters which suppress log output.
Debug log includes important details about your platform. Don't remove it.
https://streamlink.github.io/latest/cli.html#cmdoption-loglevel

You can copy the output to https://gist.github.com/ or paste it below.

Don't post screenshots of the log output and instead copy the text from your terminal application.
-->

```
[cli][debug] OS:         Windows 10
[cli][debug] Python:     3.8.7
[cli][debug] Streamlink: 2.1.1+14.g2884485
[cli][debug] Requests(2.25.1), Socks(1.7.1), Websocket(0.59.0)
[cli][debug] Arguments:
[cli][debug]  --loglevel=debug
[cli][debug]  --rtmpdump=C:\Program Files (x86)\Streamlink\rtmpdump\rtmpdump.exe
[cli][debug]  --ffmpeg-ffmpeg=C:\Program Files (x86)\Streamlink\ffmpeg\ffmpeg.exe
```


Discussion:
> This appears to be a Youtube issue, you can get a 404 on get_video_info if you try to access it too quickly on any URL, not just planned streams or live streams. It appears to be fine if you wait 1 second between requests to get_video_info though.

I've tried 2 seconds and 15 seconds and it still does that. It seems to not do it when I try for 30 seconds, but otherwise it doesn't seem to be possible to do anywhere from 1-15 or 20 seconds for the past 4 days.

So it's just YouTube's side that's causing it to not work with such short intervals?
Is there anything that can be done about it, or something that just can't be worked around?

Edit: I guess the llyyr individual retracted their message...?
I use 5 minutes with --retry-streams and I still get the error sometimes. I've also gotten it when the stream is already live and I've included --retry-streams in the command. This is the only request I make to youtube at that time, its not like I have a lot of streams waiting to go live when the error appears. It used to work fine before so I don't know what happened.
There is some kind of YouTube DDOS protection that kicks in sometimes. There is nothing you can do to stop it. It is on the YouTube server side. All you can do is wait a few days until your IP address is unblocked.  It will clear eventually on its own.  I have experienced this as well.

Sent from my iPhone

On May 16, 2021, at 1:07 PM, jenlunds ***@***.***> wrote:

﻿

I use 5 minutes with --retry-streams and I still get the error sometimes. I've also gotten it when the stream is already live and I've included --retry-streams in the command. This is the only request I make to youtube at that time, its not like I have a lot of streams waiting to go live when the error appears. It used to work fine before so I don't know what happened.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub<https://github.com/streamlink/streamlink/issues/3724#issuecomment-841845613>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AR6YJPNH4RBUDAJLDWYOPPLTN73UFANCNFSM444XOVKQ>.

It feels too inconsistent to be some kind of DDoS protection.
Seems more like some of their servers are a little screwy from some update that might have been pushed around a week ago. It's affected the RSS of someone I know as well, giving him a lot of 404s recently.

@xanek, does it happen for you 100% of the time, most of the time, occasionally, etc.?
I hit it surprisingly often with streamlink compared to one of my own scripts, and when I was testing with curl. More interesting is that if it gets hit, it continues to be hit for that running instance of streamlink. Stopping and immediately restarting will usually work, even if there's <2 seconds between the failed request of the old instance, and the request from the new instance. I imagine this has to do with requests keeping the connection open, and thus whatever server the youtube proxies forwarded you to continues to be used, so if it is a bad one, it just will not work. I've set `--retry-streams 60` for one test to see if that would be long enough for requests to make a completely new connection, but it seems not.

I've opted for pulling the player_response object from a `script` tag in the watch page as a fallback for my script, disgusting as that is. It's simple enough though, as the script tag that declares it is used solely for the declaration, so there little that needs to be stripped before using it. For now.
> @xanek, does it happen for you 100% of the time, most of the time, occasionally, etc.?

This has been happening ever since a few days ago.
It never happened prior to that and I've been using streamlink for more than half a year already.
This has the same characteristics of what I have experienced. Like I said, it’s definitely some kind of DDOS protection against this URL. Your IP is on a banned list. It will go away on its own after 3/4/5 days or something. There is nothing at all you can do about it as it’s on the server side.

You can tell it’s not a streamlink issue because when this is occurring, the youtube-dl application also exhibits the same issue.

Sent from my iPhone

On May 16, 2021, at 9:26 PM, xanek ***@***.***> wrote:

﻿

@xanek<https://github.com/xanek>, does it happen for you 100% of the time, most of the time, occasionally, etc.?

This has been happening ever since a few days ago.
It never happened prior to that and I've been using streamlink for more than half a year already.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub<https://github.com/streamlink/streamlink/issues/3724#issuecomment-841917445>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AR6YJPIIOZZJL7GHMAFS77TTOBWELANCNFSM444XOVKQ>.

> This has the same characteristics of what I have experienced. Like I said, it’s definitely some kind of DDOS protection against this URL. Your IP is on a banned list. It will go away on its own after 3/4/5 days or something. There is nothing at all you can do about it as it’s on the server side. You can tell it’s not a streamlink issue because when this is occurring, the youtube-dl application also exhibits the same issue.

Wouldn't it completely lock me out from using streamlink in general then?
I am able to still use it, just not able to have it retry constantly.
It is not necessarily 100% consistent…so yes this makes sense.

Sent from my iPhone

On May 16, 2021, at 9:31 PM, xanek ***@***.***> wrote:

﻿

This has the same characteristics of what I have experienced. Like I said, it’s definitely some kind of DDOS protection against this URL. Your IP is on a banned list. It will go away on its own after 3/4/5 days or something. There is nothing at all you can do about it as it’s on the server side. You can tell it’s not a streamlink issue because when this is occurring, the youtube-dl application also exhibits the same issue.

Wouldn't it completely lock me out from using streamlink in general then?
I am able to still use it, just not able to have it retry constantly.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub<https://github.com/streamlink/streamlink/issues/3724#issuecomment-841918799>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AR6YJPN3VJGERQIGCC3BETTTOBWYZANCNFSM444XOVKQ>.

> It is not necessarily 100% consistent

That would make for some pointless DDoS protection, I must say. Not to mention, at least in my case, it started happening after using streamlink for the first time in almost a week, and I was not exactly spamming downloads with other tools in that time either. 
You are correct it's not a streamlink issue though, at least it's not the cause. But if it ends up that this stays an issue for a long while, streamlink devs would probably want to try to figure out some way to mitigate it.
Well the logic of their DDOS protection may indeed be faulty, but unless you are assigned a static IP address from your ISP you have no idea what the person who has your IP before you did was doing.

I have had this happen to me on at least 3 occasions. There is to way to tell what triggered it and there is no way to get around it, except to wait for it to clear out. Trust me.

Sent from my iPhone

On May 16, 2021, at 9:52 PM, Kethsar ***@***.***> wrote:

﻿

It is not necessarily 100% consistent

That would make for some pointless DDoS protection, I must say. Not to mention, at least in my case, it started happening after using streamlink for the first time in almost a week, and I was not exactly spamming downloads with other tools in that time either.
You are correct it's not a streamlink issue though, at least it's not the cause. But if it ends up that this stays an issue for a long while, streamlink would probably want to try to figure out some way to mitigate it.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub<https://github.com/streamlink/streamlink/issues/3724#issuecomment-841924154>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AR6YJPM3IPIFBGLGGESVSBDTOBZG5ANCNFSM444XOVKQ>.

> It is not necessarily 100% consistent…so yes this makes sense.

Well I remember the last time I had something similar to where I couldn't even use streamlink at all because of being banned, that I had to use a VPN to get around it.

But even with a VPN right now it is giving me the same message.
is it really DDoS? i tried with youtube-dl and it parse different list of m3u8 contents for an ongoing livestream video.

it's no longer listing webm, mp4 or audio and video only formats.

![image](https://user-images.githubusercontent.com/35561383/118475579-a0deb900-b736-11eb-9be0-fd4a42d028e9.png)


That is a livestream you checked. Livestreams only have VP9 (webm) if they are using Low Latency mode. Ultra-Low Latency and Normal Latency do not have VP9 as an option. That stream is Normal latency.
Even if it were Low Latency mode, youtube-dl and I think streamlink both grab from the HLS stream playlists, which youtube only provides H264+AAC combined, and thus no audio-only or VP9.
Youtube has again changed the way the stream info is presented.  Even the (slightly older) VLC v3.0.9.2 now barfs on most youtube urls.
So would we have to wait for a new update with streamlink or is this something that can't be workaround?
Mpc-be also has issues with the built in parser for youtube... https://sourceforge.net/p/mpcbe/tickets/819/

Resolving patch:
diff --git a/src/streamlink/plugins/youtube.py b/src/streamlink/plugins/youtube.py
--- a/src/streamlink/plugins/youtube.py
+++ b/src/streamlink/plugins/youtube.py
@@ -286,7 +286,7 @@ def _get_stream_info(self, video_id):
         info_parsed = None
         for _params in (_params_1, _params_2, _params_3):
             count += 1
-            params = {"video_id": video_id}
+            params = {"video_id": video_id, "html5": "1"}
             params.update(_params)
 
             res = self.session.http.get(self._video_info_url, params=params)


Test patch: