Repository: yt-dlp/yt-dlp

Issue:
[Iwara] Unable to access some video without access token
### DO NOT REMOVE OR SKIP THE ISSUE TEMPLATE

- [X] I understand that I will be **blocked** if I *intentionally* remove or skip any mandatory\* field

### Checklist

- [X] I'm reporting a bug unrelated to a specific site
- [X] I've verified that I'm running yt-dlp version **2023.03.04** ([update instructions](https://github.com/yt-dlp/yt-dlp#update)) or later (specify commit)
- [X] I've checked that all provided URLs are playable in a browser with the same IP and same login details
- [X] I've checked that all URLs and arguments with special characters are [properly quoted or escaped](https://github.com/yt-dlp/yt-dlp/wiki/FAQ#video-url-contains-an-ampersand--and-im-getting-some-strange-output-1-2839-or-v-is-not-recognized-as-an-internal-or-external-command)
- [X] I've searched [known issues](https://github.com/yt-dlp/yt-dlp/issues/3766) and the [bugtracker](https://github.com/yt-dlp/yt-dlp/issues?q=) for similar issues **including closed ones**. DO NOT post duplicates
- [X] I've read the [guidelines for opening an issue](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#opening-an-issue)

### Provide a description that is worded well enough to be understood

Some video is not able to access without 'Authorization' in http request to [https://api.iwara.tv/video/](url)`<video id>`. In the browser, I can login to get this header but yt-dlp isn't able to send it because it's not in the cookie.

The Authorization header string is in the form of 'Bearer ' . concat(e) where e is the accessToken return from [https://api.iwara.tv/user/token](url) request.

~Request~
~https://api.iwara.tv/user/token~
~POST~
~Http header: {`Authorization: Bearer <user token>`}   --> user token was stored in localStorage and accessed by this localStorage.getItem('token')~

~Reponse~
~JSON:~
~accessToken: `<access token>`~

Request
```
https://api.iwara.tv/user/login
POST {
    "email": <username/email>,
    "password": <password>
}
```

Response 
```
JSON: {
    "token": <user token>
}
```

Request
```
https://api.iwara.tv/user/token
POST
Authorization: Bearer <user token>
```

Reponse
```
JSON: {
    "accessToken": <access token>
}
```

Once access token is gained, [https://api.iwara.tv/video/](url)`<video id>` can be request with this header
`Authorization: Bearer <access token>`

Request
```
https://api.iwara.tv/video/<id>
POST
Authorization: Bearer <access token>
```

Reponse
```
JSON: {
    <video info> --> otherwise "message":"errors.notFound" will be returned
}
```

~The problem is I don't know how to get the user token. It seems it was generated from user id and other params and ran through some function to create that that token. I assume this is similar to the X-Version header for download link request.~

The access token can be stored and reused for sub-sequence api requests for other videos.

Seem like they use this for private and login required videos.

List of some video that need this:
token: jlavxivexhegqyge
token: vqrz5sr6j5fegrzv4
404: neglocmyo5tpz7rl4
private: djevysgz9ik044wr
private: 0gqdecjqr1szqv20a

The first 2 will return 404 if Authorization header is not used.

### Provide verbose output that clearly demonstrates the problem

- [X] Run **your** yt-dlp command with **-vU** flag added (`yt-dlp -vU <your command line>`)
- [X] If using API, add `'verbose': True` to `YoutubeDL` params instead
- [X] Copy the WHOLE output (starting with `[debug] Command-line config`) and insert it below

### Complete Verbose Output

```shell
[debug] Command-line config: ['-vU', 'https://www.iwara.tv/video/jlavxivexhegqyge']
[debug] User config "C:\Users\Nobita\AppData\Roaming\yt-dlp\config.txt": ['--compat-options', 'filename', '--cookies-from-browser', 'firefox', '--retries', 'infinite', '--retry-sleep', '2', '--extractor-args', 'youtube:skip=dash']
[debug] Encodings: locale cp932, fs utf-8, pref cp932, out cp932 (No VT), error cp932 (No VT), screen cp932 (No VT)
[debug] yt-dlp version stable@2023.03.04 [392389b7d]
[debug] Lazy loading extractors is disabled
[debug] Compatibility options: filename
[debug] Python 3.8.5 (CPython AMD64 32bit) - Windows-7-6.1.7601-SP1 (OpenSSL 1.1.1g  21 Apr 2020)
[debug] exe versions: ffmpeg N-106355-ga24e6ca8bf-20220320 (setts), ffprobe N-106355-ga24e6ca8bf-20220320
[debug] Optional libraries: Cryptodome-3.17, brotli-1.0.9, certifi-2022.12.07, mutagen-1.46.0, sqlite3-2.6.0, websockets-10.4
[Cookies] Extracting cookies from firefox
[debug] Extracting cookies from: "C:\Users\Nobita\AppData\Roaming\Mozilla\Firefox\Profiles\jdltat92.dev-edition-default-1663626297735\cookies.sqlite"
[Cookies] Extracted 1766 cookies from firefox
[debug] Proxy map: {}
[debug] Loaded 1793 extractors
[debug] Fetching release info: https://api.github.com/repos/yt-dlp/yt-dlp/releases/latest
Available version: stable@2023.03.04, Current version: stable@2023.03.04
yt-dlp is up to date (stable@2023.03.04)
[iwara] Extracting URL: https://www.iwara.tv/video/jlavxivexhegqyge
[iwara] jlavxivexhegqyge: Downloading JSON metadata
ERROR: [iwara] jlavxivexhegqyge: Iwara says: errors.notFound; please report this issue on  https://github.com/yt-dlp/yt-dlp/issues?q= , filling out the appropriate issue template. Confirm you are on the latest version using  yt-dlp -U
  File "\\192.168.1.40\qemutemp\Temp\yt-dlp\yt_dlp\extractor\common.py", line 694, in extract
    ie_result = self._real_extract(url)
  File "\\192.168.1.40\qemutemp\Temp\yt-dlp\yt_dlp\extractor\iwara.py", line 85, in _real_extract
    raise ExtractorError(f'Iwara says: {errmsg}')


```

Edit



Discussion:
I had created a patch. This requires iwara account in netrc and need some clean up for checking account available or not.
@Lesmiscore can you take a look at this? I based it loosely on your update of abematv extractor because they're both similar in token usage.

Some video ids to test: (404 without access token)
blggmfno8ghl725bg
0nyzyh01eohzag5o5
b7rkks86qvu32jep

```diff --git a/yt_dlp/extractor/iwara.py b/yt_dlp/extractor/iwara.py
index ae2960af0..590777085 100644
--- a/yt_dlp/extractor/iwara.py
+++ b/yt_dlp/extractor/iwara.py
@@ -1,6 +1,7 @@
 import functools
 import urllib.parse
 import hashlib
+import json
 
 from .common import InfoExtractor
 from ..utils import (
@@ -12,8 +13,45 @@
     unified_timestamp,
 )
 
+class IwaraBaseIE(InfoExtractor):
+    _USERTOKEN = None
+    _MEDIATOKEN = None
+    _NETRC_MACHINE = "iwara"
 
-class IwaraIE(InfoExtractor):
+    def _get_user_token(self):
+        if self._USERTOKEN:
+            return self._USERTOKEN
+
+        username, password = self._get_login_info()
+        IwaraBaseIE._USERTOKEN = self._download_json(
+            f'https://api.iwara.tv/user/login', None, note='Logging in',
+            data=json.dumps({
+                'email': username,
+                'password': password
+            }).encode('utf-8'),
+            headers={
+                'Content-Type': 'application/json'
+            })['token']
+
+        self.cache.store(self._NETRC_MACHINE, username, IwaraBaseIE._USERTOKEN)
+
+        return self._USERTOKEN
+
+    def _get_media_token(self, invalidate=False):
+        if not invalidate and self._MEDIATOKEN:
+            return self._MEDIATOKEN
+
+        IwaraBaseIE._MEDIATOKEN = self._download_json(f'https://api.iwara.tv/user/token', None, note='Fetching media token',
+            data=b'', # Need to have some data here, even if it's empty
+            headers={
+                'Authorization': f'Bearer {self._get_user_token()}',
+                'Content-Type': 'application/json'
+            })['accessToken']
+
+        return self._MEDIATOKEN
+
+
+class IwaraIE(IwaraBaseIE):
     IE_NAME = 'iwara'
     _VALID_URL = r'https?://(?:www\.|ecchi\.)?iwara\.tv/videos?/(?P<id>[a-zA-Z0-9]+)'
     _TESTS = [{
@@ -76,7 +114,11 @@ def _extract_formats(self, video_id, fileurl):
 
     def _real_extract(self, url):
         video_id = self._match_id(url)
-        video_data = self._download_json(f'http://api.iwara.tv/video/{video_id}', video_id, expected_status=lambda x: True)
+        video_data = self._download_json(f'http://api.iwara.tv/video/{video_id}', video_id, expected_status=lambda x: True,
+            headers={
+                'Authorization': f'Bearer {self._get_media_token()}',
+                'Content-Type': 'application/json'
+            })
         errmsg = video_data.get('message')
         # at this point we can actually get uploaded user info, but do we need it?
         if errmsg == 'errors.privateVideo':
@@ -104,8 +146,17 @@ def _real_extract(self, url):
             'formats': list(self._extract_formats(video_id, video_data.get('fileUrl'))),
         }
 
+    def _perform_login(self, username, password):
+        if self.cache.load(self._NETRC_MACHINE, username) and self._get_media_token():
+            self.write_debug('Skipping logging in')
+            return
+#
+        IwaraBaseIE._USERTOKEN = self._get_user_token()
+        self._get_media_token(True)
+        self.cache.store(self._NETRC_MACHINE, username, IwaraBaseIE._USERTOKEN)
+
 
-class IwaraUserIE(InfoExtractor):
+class IwaraUserIE(IwaraBaseIE):
     _VALID_URL = r'https?://(?:www\.)?iwara\.tv/profile/(?P<id>[^/?#&]+)'
     IE_NAME = 'iwara:user'
     _PER_PAGE = 32
@@ -157,7 +208,7 @@ def _real_extract(self, url):
             playlist_id, traverse_obj(user_info, ('user', 'name')))
 
 
-class IwaraPlaylistIE(InfoExtractor):
+class IwaraPlaylistIE(IwaraBaseIE):
     # the ID is an UUID but I don't think it's necessary to write concrete regex
     _VALID_URL = r'https?://(?:www\.)?iwara\.tv/playlist/(?P<id>[0-9a-f-]+)'
     IE_NAME = 'iwara:playlist'
```
That looks good
You should open an PR with it

Resolving patch:
diff --git a/yt_dlp/extractor/iwara.py b/yt_dlp/extractor/iwara.py
--- a/yt_dlp/extractor/iwara.py
+++ b/yt_dlp/extractor/iwara.py
@@ -1,6 +1,7 @@
 import functools
 import urllib.parse
 import hashlib
+import json
 
 from .common import InfoExtractor
 from ..utils import (
@@ -14,7 +15,49 @@
 )
 
 
-class IwaraIE(InfoExtractor):
+# https://github.com/yt-dlp/yt-dlp/issues/6671
+class IwaraBaseIE(InfoExtractor):
+    _USERTOKEN = None
+    _MEDIATOKEN = None
+    _NETRC_MACHINE = 'iwara'
+
+    def _get_user_token(self, invalidate=False):
+        if not invalidate and self._USERTOKEN:
+            return self._USERTOKEN
+
+        username, password = self._get_login_info()
+        IwaraBaseIE._USERTOKEN = username and self.cache.load(self._NETRC_MACHINE, username)
+        if not IwaraBaseIE._USERTOKEN or invalidate:
+            IwaraBaseIE._USERTOKEN = self._download_json(
+                'https://api.iwara.tv/user/login', None, note='Logging in',
+                data=json.dumps({
+                    'email': username,
+                    'password': password
+                }).encode('utf-8'),
+                headers={
+                    'Content-Type': 'application/json'
+                })['token']
+
+            self.cache.store(self._NETRC_MACHINE, username, IwaraBaseIE._USERTOKEN)
+
+        return self._USERTOKEN
+
+    def _get_media_token(self, invalidate=False):
+        if not invalidate and self._MEDIATOKEN:
+            return self._MEDIATOKEN
+
+        IwaraBaseIE._MEDIATOKEN = self._download_json(
+            'https://api.iwara.tv/user/token', None, note='Fetching media token',
+            data=b'',  # Need to have some data here, even if it's empty
+            headers={
+                'Authorization': f'Bearer {self._get_user_token()}',
+                'Content-Type': 'application/json'
+            })['accessToken']
+
+        return self._MEDIATOKEN
+
+
+class IwaraIE(IwaraBaseIE):
     IE_NAME = 'iwara'
     _VALID_URL = r'https?://(?:www\.|ecchi\.)?iwara\.tv/videos?/(?P<id>[a-zA-Z0-9]+)'
     _TESTS = [{
@@ -56,6 +99,26 @@ class IwaraIE(InfoExtractor):
             'timestamp': 1678732213,
             'modified_timestamp': 1679110271,
         },
+    }, {
+        'url': 'https://iwara.tv/video/blggmfno8ghl725bg',
+        'info_dict': {
+            'id': 'blggmfno8ghl725bg',
+            'ext': 'mp4',
+            'age_limit': 18,
+            'title': 'お外でおしっこしちゃう猫耳ロリメイド',
+            'description': 'md5:0342ba9bf6db09edbbb28729657c3611',
+            'uploader': 'Fe_Kurosabi',
+            'uploader_id': 'fekurosabi',
+            'tags': [
+                'pee'
+            ],
+            'like_count': 192,
+            'view_count': 12119,
+            'comment_count': 0,
+            'timestamp': 1598880567,
+            'modified_timestamp': 1598908995,
+            'availability': 'needs_auth',
+        },
     }]
 
     def _extract_formats(self, video_id, fileurl):
@@ -79,12 +142,18 @@ def _extract_formats(self, video_id, fileurl):
 
     def _real_extract(self, url):
         video_id = self._match_id(url)
-        video_data = self._download_json(f'https://api.iwara.tv/video/{video_id}', video_id, expected_status=lambda x: True)
+        username, password = self._get_login_info()
+        headers = {
+            'Authorization': f'Bearer {self._get_media_token()}',
+        } if username and password else None
+        video_data = self._download_json(f'https://api.iwara.tv/video/{video_id}', video_id, expected_status=lambda x: True, headers=headers)
         errmsg = video_data.get('message')
         # at this point we can actually get uploaded user info, but do we need it?
         if errmsg == 'errors.privateVideo':
             self.raise_login_required('Private video. Login if you have permissions to watch')
-        elif errmsg:
+        elif errmsg == 'errors.notFound' and not username:
+            self.raise_login_required('Video may need login to view')
+        elif errmsg:  # None if success
             raise ExtractorError(f'Iwara says: {errmsg}')
 
         if not video_data.get('fileUrl'):
@@ -112,8 +181,17 @@ def _real_extract(self, url):
             'formats': list(self._extract_formats(video_id, video_data.get('fileUrl'))),
         }
 
+    def _perform_login(self, username, password):
+        if self.cache.load(self._NETRC_MACHINE, username) and self._get_media_token():
+            self.write_debug('Skipping logging in')
+            return
+
+        IwaraBaseIE._USERTOKEN = self._get_user_token(True)
+        self._get_media_token(True)
+        self.cache.store(self._NETRC_MACHINE, username, IwaraBaseIE._USERTOKEN)
+
 
-class IwaraUserIE(InfoExtractor):
+class IwaraUserIE(IwaraBaseIE):
     _VALID_URL = r'https?://(?:www\.)?iwara\.tv/profile/(?P<id>[^/?#&]+)'
     IE_NAME = 'iwara:user'
     _PER_PAGE = 32
@@ -165,7 +243,7 @@ def _real_extract(self, url):
             playlist_id, traverse_obj(user_info, ('user', 'name')))
 
 
-class IwaraPlaylistIE(InfoExtractor):
+class IwaraPlaylistIE(IwaraBaseIE):
     # the ID is an UUID but I don't think it's necessary to write concrete regex
     _VALID_URL = r'https?://(?:www\.)?iwara\.tv/playlist/(?P<id>[0-9a-f-]+)'
     IE_NAME = 'iwara:playlist'


Test patch: