Repository: Flexget/Flexget

Issue:
Plugin my_anime_list, URL JSON result maxed at 300 entries.


<!---
Before opening an issue, verify:

- Is this a feature request? Post it on https://feathub.com/Flexget/Flexget
- Is this an issue with webui? Make an issue over on https://github.com/Flexget/webui
- Did you recently upgrade? Look at the Change Log and Upgrade Actions to make sure that you don't need to make any changes to your config https://flexget.com/ChangeLog https://flexget.com/UpgradeActions
- Are you running FlexGet as a daemon? Stop it completely and then start it again https://flexget.com/CLI/daemon
- Did you search to see if the issue already exists? https://github.com/Flexget/Flexget/issues
- Did you fill out the issue template as completely as possible?

The issue template is here because it helps to ensure you submitted all the necessary information the first time, and allows us to more quickly review issues. Please fill it out correctly and do not ignore it, no matter how irrelevant you think it may be. Thanks in advance for your help with this!
--->

### Expected behaviour:
For the Plugin: my_anime_list to return the requested categories, preferably with less traffic overhead.

### Actual behaviour:
The URL JSON result returns the first 300 results of ALL categories in order of status id.
This is a limitation of the JSON url used and not a bug in the flexget plugin code.

### Steps to reproduce:
- Step 1: Have more than 300 listed anime on the site, in any given category.
- Step 2: Request a category that is sorted last in the JSON result, like "plan_to_watch".
- Step 3: If you have more than 300 anime listed as "Completed" or "Dropped", the JSON result would never load the "plan_to_watch" entries.

#### Config:
```yaml
my_anime_list:
  username: <removed>
  status:
    - watching
    - plan_to_watch
```
  
#### Log:
<details>
 <summary>(click to expand)</summary>

```
na
```
</details>

### Additional information:

The JSON source used in the plugin, only gives the first 300 results from MAL. (https://myanimelist.net/animelist/flexget/load.json)
This is not an issue if you use "watching" as they are sorted first in the JSON reply, but "plan_to_watch" that many use is sorted last, after categories like "completed" and "dropped".

Solutions:
It's possible to specify what result you want in the URL but only for a single status id.
If it merged multiple status specific JSON results, it would probably require less traffic than the current parameterless "load.JSON", as most users just want status "1" & "6".
Ex. Status 6 is "plan_to_watch" only: https://myanimelist.net/animelist/flexget/load.json?status=6

It's also possible to offset the result by a given amount, but would require merging large lists until an empty result is given "[]".
Ex. JSON results from 300->600: https://myanimelist.net/animelist/flexget/load.json?offset=300

Alternative solution:
From what i read, Load.JSON is an internal tool for loading lists and is not meant to be used as an API.
There is a new API, but it appears to require an authorization token to use and it's in beta:
API auth: https://myanimelist.net/apiconfig
API doc.: https://myanimelist.net/apiconfig/references/api/v2
<!---
In config and debug/crash logs, remember to redact any personal or sensitive information such as passwords, API keys, private URLs and so on.

Please verify that the following data is present before submitting your issue:

- Link to a paste service or paste above the relevant config (preferably full config, including templates if present). Please make sure the paste does not expire, if possible.
- Link to a paste service or paste above debug-level logs of the relevant task/s (use `flexget -L debug execute --tasks <Task_name>`).
- FlexGet version (use `flexget -V` to get it).
- Full Python version, for example `2.7.11` (use `python -V` to get it). 
- Installation method (pip, git install, etc).
- Whether or not you're running FlexGet as a daemon.
- OS and version.
- Attach crash log if one was generated, in addition to the debug-level log. It can be found in the directory with your config file.
--->



Discussion:
TBF, it's an input, API would be overkill since we're just reading. So much so that the plugin used to scrape the webpage before being rewritten to use the JSON 😅.

Good info with the parameters though, if there's no PR until later in the week I might give it a go :)

Resolving patch:
diff --git a/flexget/plugins/input/my_anime_list.py b/flexget/plugins/input/my_anime_list.py
--- a/flexget/plugins/input/my_anime_list.py
+++ b/flexget/plugins/input/my_anime_list.py
@@ -56,60 +56,65 @@ class MyAnimeList:
 
     @cached('my_anime_list', persist='2 hours')
     def on_task_input(self, task, config):
-        entries = []
-        selected_status = config['status']
-        selected_airing_status = config['airing_status']
-        selected_types = config['type']
-
+        selected_status = config.get('status', ['all'])
         if not isinstance(selected_status, list):
             selected_status = [selected_status]
 
+        selected_airing_status = config.get('airing_status', ['all'])
         if not isinstance(selected_airing_status, list):
             selected_airing_status = [selected_airing_status]
 
+        selected_types = config.get('type', ['all'])
         if not isinstance(selected_types, list):
             selected_types = [selected_types]
 
         selected_status = [STATUS[s] for s in selected_status]
-
         selected_airing_status = [AIRING_STATUS[s] for s in selected_airing_status]
-
-        try:
-            list_response = task.requests.get(
-                'https://myanimelist.net/animelist/' + config['username'] + '/load.json'
-            )
-        except RequestException as e:
-            raise plugin.PluginError('Error finding list on url: {url}'.format(url=e.request.url))
-
-        try:
-            list_json = list_response.json()
-        except ValueError:
-            raise plugin.PluginError('Invalid JSON response')
-
-        for anime in list_json:
-            has_selected_status = anime["status"] in selected_status or config['status'] == 'all'
-            has_selected_airing_status = (
-                anime["anime_airing_status"] in selected_airing_status
-                or config['airing_status'] == 'all'
-            )
-            has_selected_type = (
-                anime["anime_media_type_string"].lower() in selected_types
-                or config['type'] == 'all'
-            )
-            if has_selected_status and has_selected_type and has_selected_airing_status:
-                # MAL sometimes returns title as an integer :| GH #2901
-                anime["anime_title"] = str(anime["anime_title"])
-                entries.append(
-                    Entry(
-                        title=anime["anime_title"],
-                        url="https://myanimelist.net" + anime["anime_url"],
-                        mal_name=anime["anime_title"],
-                        mal_poster=anime["anime_image_path"],
-                        mal_type=anime["anime_media_type_string"],
-                        mal_tags=anime["tags"],
-                    )
+        list_json = []
+
+        for status in selected_status:
+            # JSON pages are limited to 300 entries
+            offset = 0
+            results = 300
+            while results == 300:
+                try:
+                    list_json += task.requests.get(
+                        f'https://myanimelist.net/animelist/{config.get("username")}/load.json',
+                        params={'status': status, 'offset': offset},
+                    ).json()
+                except RequestException as e:
+                    logger.error(f'Error finding list on url: {e.request.url}')
+                    break
+                except (ValueError, TypeError):
+                    logger.error('Invalid JSON response')
+                    break
+                results = len(list_json) or 1
+                offset += len(list_json)
+
+            for anime in list_json:
+                has_selected_status = (
+                    anime["status"] in selected_status or config['status'] == 'all'
+                )
+                has_selected_airing_status = (
+                    anime["anime_airing_status"] in selected_airing_status
+                    or config['airing_status'] == 'all'
+                )
+                has_selected_type = (
+                    anime["anime_media_type_string"].lower() in selected_types
+                    or config['type'] == 'all'
                 )
-        return entries
+                if has_selected_status and has_selected_type and has_selected_airing_status:
+                    # MAL sometimes returns title as an integer
+                    anime['anime_title'] = str(anime['anime_title'])
+                    entry = Entry()
+                    entry['title'] = anime['anime_title']
+                    entry['url'] = f'https://myanimelist.net{anime["anime_url"]}'
+                    entry['mal_name'] = anime['anime_title']
+                    entry['mal_poster'] = anime['anime_image_path']
+                    entry['mal_type'] = anime['anime_media_type_string']
+                    entry['mal_tags'] = anime['tags']
+                    if entry.isvalid():
+                        yield entry
 
 
 @event('plugin.register')


Test patch: