Repository: Flexget/Flexget

Issue:
imdb plugin failing with a 405 error code
<!---
Before opening an issue, verify:

- Is this an issue with webui? Make an issue over on https://github.com/Flexget/webui
- Did you recently upgrade? Look at the Change Log and Upgrade Actions to make sure that you don't need to make any changes to your config https://flexget.com/ChangeLog https://flexget.com/UpgradeActions
- Are you running FlexGet as a daemon? Stop it completely and then start it again https://flexget.com/CLI/daemon
- Did you search to see if the issue already exists? https://github.com/Flexget/Flexget/issues
- Did you fill out the issue template as completely as possible?

The issue template is here because it helps to ensure you submitted all the necessary information the first time, and allows us to more quickly review issues. Please fill it out correctly and do not ignore it, no matter how irrelevant you think it may be. Thanks in advance for your help with this!
--->

### Expected behaviour:

<!---
Please don't just say "it doesn't crash" or "it works". Explain what the expected result is.
--->

Flexget is parsing the imdb.com and doesn't throw an error while doing it.

### Actual behaviour:

### Steps to reproduce:
- Run the attached sample of config 
- See the error from flexget saying the HTTP method not allowed

#### Config:
```yaml
# Import sensitive data

variables: secrets.yml

####################
# Define templates #
####################

templates:

  slack-notification-movies:
    notify:
      entries:
        via:
          - slack:
              web_hook_url: '{? notify.slack ?}'
              message: "{{ movie_name }}"
              attachments:
                - title: "{{ movie_name }} ({{ movie_year }})"
                  image_url: "{{ imdb_photo }}" # better quality
                  title_link: "{{ imdb_url }}"
                  fallback: "{{ imdb_name }} ({{ imdb_year }})"
                  text: "{{ imdb_plot_outline }}"
                  color: "#e67e22"
                  footer: "{{ task }}"
                  footer_icon: "https://avatars2.githubusercontent.com/u/17483320?s=400&v=4"
                  fields:
                    - title: Score
                      value: "{{ imdb_score }}"
                      short: true
                    - title: Genres
                      value: "{{ imdb_genres | join(', ') | title }}"
                      short: true
                    - title: Resolution
                      value: "{{ quality.resolution }}"
                      short: true
                    - title: Filesize
                      value: "{{ content_size }} MB"
                      short: true


####################################
# Organizing the movies and series #
####################################

tasks:

  copy-movies:
    priority: 2
    template: [ slack-notification-movies ]
    metainfo_movie: yes
    imdb_lookup: yes
    #disable: seen
    seen: local
    accept_all: yes
    content_size:
      min: 1300
    filesystem:
      path: /transmission/downloads/complete/movies/
      regexp: '.*.(avi|mkv|mp4|m4v)$'
      recursive: yes
    copy:
      to: "/MoviesFromIMDBList/{{ movie_name|replace(' ','.')|replace(':','.')|replace('?','.') }}.{{ movie_year }}.{{ quality.resolution }}/"
      rename: "{{ movie_name|replace(' ','.')|replace(':','.')|replace('?','.') }}.{{ movie_year }}.{{ quality.resolution }}{{ location|pathext }}"
```
  
#### Log:
<details>
 <summary>(click to expand)</summary>

```
2022-11-21 11:46:50 DEBUG    backlog       copy-movies     0 entries purged from backlog
2022-11-21 11:46:50 VERBOSE  details       copy-movies     Produced 1 entries.
2022-11-21 11:46:50 DEBUG    content_size  copy-movies     Long.Lost.Christmas.2022.1080p.WEBRip.x264-RARBG size 1643 MB
2022-11-21 11:46:50 VERBOSE  task          copy-movies     ACCEPTED: `Long.Lost.Christmas.2022.1080p.WEBRip.x264-RARBG` by accept_all plugin
2022-11-21 11:46:50 DEBUG    urlrewriter   copy-movies     Checking 1 entries
2022-11-21 11:46:50 VERBOSE  details       copy-movies     Summary - Accepted: 1 (Rejected: 0 Undecided: 0 Failed: 0)
2022-11-21 11:46:50 INFO     content_size  copy-movies     Plugin is partially disabled with --test and --learn because size information may not be available
2022-11-21 11:46:50 DEBUG    imdb_lookup   copy-movies     lookup for Long.Lost.Christmas.2022.1080p.WEBRip.x264-RARBG
2022-11-21 11:46:51 VERBOSE  imdb_lookup   copy-movies     Parsing imdb for `Long.Lost.Christmas.2022.1080p.WEBRip.x264-RARBG`
2022-11-21 11:46:51 DEBUG    utils.requests copy-movies     GETing URL https://www.imdb.com/title/tt22307560/ with args () and kwargs {'allow_redirects': True, 'timeout': 30}
2022-11-21 11:46:51 DEBUG    plugin        copy-movies     decorator caught RequestException. handled traceback:
Traceback (most recent call last):

  File "/usr/lib/python3.8/threading.py", line 890, in _bootstrap
    self._bootstrap_inner()
    │    └ <function Thread._bootstrap_inner at 0x7f4c5cf27ee0>
    └ <Thread(task_queue, started daemon 139965851674368)>
  File "/usr/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
    │    └ <function Thread.run at 0x7f4c5cf27c10>
    └ <Thread(task_queue, started daemon 139965851674368)>
  File "/usr/lib/python3.8/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
    │    │        │    │        │    └ {}
    │    │        │    │        └ <Thread(task_queue, started daemon 139965851674368)>
    │    │        │    └ ()
    │    │        └ <Thread(task_queue, started daemon 139965851674368)>
    │    └ <bound method TaskQueue.run of <flexget.task_queue.TaskQueue object at 0x7f4c5716d160>>
    └ <Thread(task_queue, started daemon 139965851674368)>

  File "/usr/local/lib/python3.8/dist-packages/flexget/task_queue.py", line 47, in run
    self.current_task.execute()
    │    │            └ <function Task.execute at 0x7f4c59f66d30>
    │    └ <flexget.task.Task object at 0x7f4c56e02910>
    └ <flexget.task_queue.TaskQueue object at 0x7f4c5716d160>

  File "/usr/local/lib/python3.8/dist-packages/flexget/task.py", line 87, in wrapper
    return func(self, *args, **kw)
           │    │      │       └ {}
           │    │      └ ()
           │    └ <flexget.task.Task object at 0x7f4c56e02910>
           └ <function Task.execute at 0x7f4c59f66ca0>

  File "/usr/local/lib/python3.8/dist-packages/flexget/task.py", line 727, in execute
    self._execute()
    │    └ <function Task._execute at 0x7f4c59f66c10>
    └ <flexget.task.Task object at 0x7f4c56e02910>

  File "/usr/local/lib/python3.8/dist-packages/flexget/task.py", line 696, in _execute
    self.__run_task_phase(phase)
    │                     └ 'output'
    └ <flexget.task.Task object at 0x7f4c56e02910>

  File "/usr/local/lib/python3.8/dist-packages/flexget/task.py", line 514, in __run_task_phase
    response = self.__run_plugin(plugin, phase, args)
               │                 │       │      └ (<flexget.task.Task object at 0x7f4c56e02910>, {'to': "/home/vilius/Downloads/synonology.nas/Media/Filmai/MoviesFromIMDBList/...
               │                 │       └ 'output'
               │                 └ <PluginInfo(name=copy)>
               └ <flexget.task.Task object at 0x7f4c56e02910>

  File "/usr/local/lib/python3.8/dist-packages/flexget/task.py", line 547, in __run_plugin
    result = method(*args, **kwargs)
             │       │       └ {}
             │       └ (<flexget.task.Task object at 0x7f4c56e02910>, {'to': "/home/vilius/Downloads/synonology.nas/Media/Filmai/MoviesFromIMDBList/...
             └ <Event(name=plugin.copy.output,func=on_task_output,priority=128)>

  File "/usr/local/lib/python3.8/dist-packages/flexget/event.py", line 20, in __call__
    return self.func(*args, **kwargs)
           │    │     │       └ {}
           │    │     └ (<flexget.task.Task object at 0x7f4c56e02910>, {'to': "/home/vilius/Downloads/synonology.nas/Media/Filmai/MoviesFromIMDBList/...
           │    └ <bound method BaseFileOps.on_task_output of <flexget.plugins.output.file_operations.CopyFiles object at 0x7f4c57d56b20>>
           └ <Event(name=plugin.copy.output,func=on_task_output,priority=128)>

  File "/usr/local/lib/python3.8/dist-packages/flexget/plugins/output/file_operations.py", line 128, in on_task_output
    self.handle_entry(task, config, entry, siblings)
    │    │            │     │       │      └ {}
    │    │            │     │       └ <Entry(title=Long.Lost.Christmas.2022.1080p.WEBRip.x264-RARBG,state=accepted)>
    │    │            │     └ {'to': "/home/vilius/Downloads/synonology.nas/Media/Filmai/MoviesFromIMDBList/{{ movie_name|replace(' ','.')|replace(':','.')...
    │    │            └ <flexget.task.Task object at 0x7f4c56e02910>
    │    └ <function TransformingOps.handle_entry at 0x7f4c585f3ee0>
    └ <flexget.plugins.output.file_operations.CopyFiles object at 0x7f4c57d56b20>

  File "/usr/local/lib/python3.8/dist-packages/flexget/plugins/output/file_operations.py", line 239, in handle_entry
    dst_path = entry.render(dst_path)
               │     │      └ "/home/vilius/Downloads/synonology.nas/Media/Filmai/MoviesFromIMDBList/{{ movie_name|replace(' ','.')|replace(':','.')|replac...
               │     └ <function Entry.render at 0x7f4c5a338040>
               └ <Entry(title=Long.Lost.Christmas.2022.1080p.WEBRip.x264-RARBG,state=accepted)>

  File "/usr/local/lib/python3.8/dist-packages/flexget/entry.py", line 317, in render
    return render_from_entry(template, self, native=native)
           │                 │         │            └ False
           │                 │         └ <Entry(title=Long.Lost.Christmas.2022.1080p.WEBRip.x264-RARBG,state=accepted)>
           │                 └ "/home/vilius/Downloads/synonology.nas/Media/Filmai/MoviesFromIMDBList/{{ movie_name|replace(' ','.')|replace(':','.')|replac...
           └ <function render_from_entry at 0x7f4c5a387310>

  File "/usr/local/lib/python3.8/dist-packages/flexget/utils/template.py", line 330, in render_from_entry
    return render(template, variables, native=native)
           │      │         │                 └ False
           │      │         └ {'location': '/home/vilius/docker-compose/transmission/downloads/complete/movies/Long.Lost.Christmas.2022.1080p.WEBRip.x264-R...
           │      └ "/home/vilius/Downloads/synonology.nas/Media/Filmai/MoviesFromIMDBList/{{ movie_name|replace(' ','.')|replace(':','.')|replac...
           └ <function render at 0x7f4c5a387280>

  File "/usr/local/lib/python3.8/dist-packages/flexget/utils/template.py", line 306, in render
    result = template.render(context)
             │        │      └ {'location': '/home/vilius/docker-compose/transmission/downloads/complete/movies/Long.Lost.Christmas.2022.1080p.WEBRip.x264-R...
             │        └ <function Template.render at 0x7f4c5a395ca0>
             └ <FlexGetTemplate memory:7f4c55d40610>

  File "/usr/local/lib/python3.8/dist-packages/jinja2/environment.py", line 1299, in render
    return self.environment.concat(self.root_render_func(ctx))  # type: ignore
           │    │           │      │    │                └ <Context <flexget.utils.lazy_dict.LazyDict object at 0x7f4c55d5f430> of None>
           │    │           │      │    └ <function root at 0x7f4c55d3ee50>
           │    │           │      └ <FlexGetTemplate memory:7f4c55d40610>
           │    │           └ <built-in method join of str object at 0x7f4c5d22b2f0>
           │    └ <flexget.utils.template.FlexGetEnvironment object at 0x7f4c57080cd0>
           └ <FlexGetTemplate memory:7f4c55d40610>

  File "<template>", line 10, in root

  File "/usr/local/lib/python3.8/dist-packages/jinja2/runtime.py", line 240, in resolve_or_missing
    if key in self.parent:
       │      │    └ <flexget.utils.lazy_dict.LazyDict object at 0x7f4c55d5f430>
       │      └ <Context <flexget.utils.lazy_dict.LazyDict object at 0x7f4c55d5f430> of None>
       └ 'movie_name'

  File "/usr/lib/python3.8/_collections_abc.py", line 666, in __contains__
    self[key]
    │    └ 'movie_name'
    └ <flexget.utils.lazy_dict.LazyDict object at 0x7f4c55d5f430>

  File "/usr/local/lib/python3.8/dist-packages/flexget/utils/lazy_dict.py", line 77, in __getitem__
    return item[key]
           │    └ 'movie_name'
           └ <LazyLookup([LazyCallee(func=<bound method MetainfoQuality.get_quality of <flexget.plugins.metainfo.quality.MetainfoQuality o...

  File "/usr/local/lib/python3.8/dist-packages/flexget/utils/lazy_dict.py", line 41, in __getitem__
    callee.func(self.store, *(callee.args or []), **(callee.kwargs or {}))
    │      │    │    │        │      │               │      └ <_collections._tuplegetter object at 0x7f4c5a372f10>
    │      │    │    │        │      │               └ LazyCallee(func=<bound method ImdbLookup.lazy_loader of <flexget.components.imdb.imdb_lookup.ImdbLookup object at 0x7f4c571b9...
    │      │    │    │        │      └ <_collections._tuplegetter object at 0x7f4c5a372f40>
    │      │    │    │        └ LazyCallee(func=<bound method ImdbLookup.lazy_loader of <flexget.components.imdb.imdb_lookup.ImdbLookup object at 0x7f4c571b9...
    │      │    │    └ <Entry(title=Long.Lost.Christmas.2022.1080p.WEBRip.x264-RARBG,state=accepted)>
    │      │    └ <LazyLookup([LazyCallee(func=<bound method MetainfoQuality.get_quality of <flexget.plugins.metainfo.quality.MetainfoQuality o...
    │      └ <_collections._tuplegetter object at 0x7f4c5a372e80>
    └ LazyCallee(func=<bound method ImdbLookup.lazy_loader of <flexget.components.imdb.imdb_lookup.ImdbLookup object at 0x7f4c571b9...

  File "/usr/local/lib/python3.8/dist-packages/flexget/components/imdb/imdb_lookup.py", line 69, in lazy_loader
    self.lookup(entry)
    │    │      └ <Entry(title=Long.Lost.Christmas.2022.1080p.WEBRip.x264-RARBG,state=accepted)>
    │    └ <function internet.__call__.<locals>.wrapped_func at 0x7f4c57d6f790>
    └ <flexget.components.imdb.imdb_lookup.ImdbLookup object at 0x7f4c571b99d0>

> File "/usr/local/lib/python3.8/dist-packages/flexget/plugin.py", line 126, in wrapped_func
    return func(*args, **kwargs)
           │     │       └ {}
           │     └ (<flexget.components.imdb.imdb_lookup.ImdbLookup object at 0x7f4c571b99d0>, <Entry(title=Long.Lost.Christmas.2022.1080p.WEBRi...
           └ <function with_session.<locals>.decorator.<locals>.wrapper at 0x7f4c57d6f820>

  File "/usr/local/lib/python3.8/dist-packages/flexget/utils/database.py", line 30, in wrapper
    return func(*args, **kwargs)
           │     │       └ {'session': <sqlalchemy.orm.session.ContextSession object at 0x7f4c55d5f7f0>}
           │     └ (<flexget.components.imdb.imdb_lookup.ImdbLookup object at 0x7f4c571b99d0>, <Entry(title=Long.Lost.Christmas.2022.1080p.WEBRi...
           └ <function ImdbLookup.lookup at 0x7f4c57d6f670>

  File "/usr/local/lib/python3.8/dist-packages/flexget/components/imdb/imdb_lookup.py", line 222, in lookup
    movie = self._parse_new_movie(entry['imdb_url'], session)
            │    │                │                  └ <sqlalchemy.orm.session.ContextSession object at 0x7f4c55d5f7f0>
            │    │                └ <Entry(title=Long.Lost.Christmas.2022.1080p.WEBRip.x264-RARBG,state=accepted)>
            │    └ <function ImdbLookup._parse_new_movie at 0x7f4c57d6f8b0>
            └ <flexget.components.imdb.imdb_lookup.ImdbLookup object at 0x7f4c571b99d0>

  File "/usr/local/lib/python3.8/dist-packages/flexget/components/imdb/imdb_lookup.py", line 267, in _parse_new_movie
    parser.parse(imdb_url)
    │      │     └ 'https://www.imdb.com/title/tt22307560/'
    │      └ <function ImdbParser.parse at 0x7f4c586ec670>
    └ <flexget.components.imdb.utils.ImdbParser object at 0x7f4c55cc8760>

  File "/usr/local/lib/python3.8/dist-packages/flexget/components/imdb/utils.py", line 250, in parse
    page = requests.get(url)
           │        │   └ 'https://www.imdb.com/title/tt22307560/'
           │        └ <function Session.get at 0x7f4c5bc2e8b0>
           └ <flexget.utils.requests.Session object at 0x7f4c586e36d0>

  File "/usr/local/lib/python3.8/dist-packages/requests/sessions.py", line 600, in get
    return self.request("GET", url, **kwargs)
           │    │              │      └ {'allow_redirects': True}
           │    │              └ 'https://www.imdb.com/title/tt22307560/'
           │    └ <function Session.request at 0x7f4c59fbf820>
           └ <flexget.utils.requests.Session object at 0x7f4c586e36d0>

  File "/usr/local/lib/python3.8/dist-packages/flexget/utils/requests.py", line 271, in request
    result.raise_for_status()
    │      └ <function Response.raise_for_status at 0x7f4c5bc2c5e0>
    └ <Response [405]>

  File "/usr/local/lib/python3.8/dist-packages/requests/models.py", line 1021, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
          │         │                        └ <Response [405]>
          │         └ '405 Client Error: Not Allowed for url: https://www.imdb.com/title/tt22307560/'
          └ <class 'requests.exceptions.HTTPError'>

requests.exceptions.HTTPError: 405 Client Error: Not Allowed for url: https://www.imdb.com/title/tt22307560/
2022-11-21 11:46:51 VERBOSE  imdb_lookup   copy-movies     Requestexception: 405 client error: not allowed for url: https://www.imdb.com/title/tt22307560/
```
</details>

### Additional information:

- FlexGet version: 3.5.5
- Python version: 3.8.10
- Installation method: pip
- Using daemon (yes/no): no
- OS and version: 20.04.2-Ubuntu


<!---
In config and debug/crash logs, remember to redact any personal or sensitive information such as passwords, API keys, private URLs and so on.

Please verify that the following data is present before submitting your issue:

- Link to a paste service or paste above the relevant config (preferably full config, including templates if present). Please make sure the paste does not expire, if possible.
- Link to a paste service or paste above debug-level logs of the relevant task/s (use `flexget -L debug execute --tasks <Task_name>`).
- FlexGet version (use `flexget -V` to get it).
- Full Python version, for example `2.7.11` (use `python -V` to get it). 
- Installation method (pip, git install, etc).
- Whether or not you're running FlexGet as a daemon.
- OS and version.
- Attach crash log if one was generated, in addition to the debug-level log. It can be found in the directory with your config file.
--->



Discussion:
I got a 403 client error when use imdb_lookup here.

FlexGet version: 3.5.5
Installation method: docker/wiserain

```
  imdb_test:
    manual: yes
    mock:
      - {title: 'A.Beautiful.Day.in.the.Neighborhood.2019.BluRay.1080p.DTS-HD.MA.5.1.x265.10bit-BeiTai'}
    imdb_lookup: yes
    accept_all: yes
    exec: echo "{{imdb_name}} ({{imdb_year}})"
```

```
2022-11-23 18:38:59 INFO     manager                       Config change detected. Reloading.
2022-11-23 18:38:59 INFO     manager                       Config successfully reloaded!
2022-11-23 18:38:59 VERBOSE  details       imdb_test       Produced 1 entries.
2022-11-23 18:39:00 VERBOSE  task          imdb_test       ACCEPTED: `A.Beautiful.Day.in.the.Neighborhood.2019.BluRay.1080p.DTS-HD.MA.5.1.x265.10bit-BeiTai` by accept_all plugin
2022-11-23 18:39:00 VERBOSE  details       imdb_test       Summary - Accepted: 1 (Rejected: 0 Undecided: 0 Failed: 0)
2022-11-23 18:39:00 VERBOSE  imdb_lookup   imdb_test       Movie `A Beautiful Day in the Neighborhood` details expired, refreshing ...
2022-11-23 18:39:00 VERBOSE  imdb_lookup   imdb_test       Parsing imdb for `A.Beautiful.Day.in.the.Neighborhood.2019.BluRay.1080p.DTS-HD.MA.5.1.x265.10bit-BeiTai`
2022-11-23 18:39:00 INFO     imdb_lookup   imdb_test       Requestexception: 403 client error: forbidden for url: https://www.imdb.com/title/tt3224458/
2022-11-23 18:39:00 VERBOSE  exec          imdb_test       Executing: echo "None (None)"
2022-11-23 18:39:00 INFO     exec          imdb_test       Stdout: None (None)
```
Yeah I found the reason and I guess imdbparser needs updating?

curl -v https://www.imdb.com/title/tt14179942/

Gives the same error but if I add this header it works fine.

curl -v 'https://www.imdb.com/title/tt14179942/' -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7)'
Ditto for me. I get the same error.
Looks like it fixed itself. I guess imdb reverted back a change?  It's now working properly for me.
yep, seems fixed itself. Probably imdb.com reverted something on their side.
I'm still getting these errors. The source in flexget/components/imdb/utils.py is intentionally identifying itself as a bot:

    # IMDb delivers a version of the page which is unparsable to unknown (and some known) user agents, such as requests'
    # Spoof the old urllib user agent to keep results consistent
    requests = Session()
    requests.headers.update({'User-Agent': 'Python-urllib/2.6'})

Changing the user-agent in that last line to a standard browser agent fixes the problem for me.
@viliuspankauskas Could we please get this ticket re-opened?  I too have the above issue. Thanks!
> I'm still getting these errors. The source in flexget/components/imdb/utils.py is intentionally identifying itself as a bot:
> 
> ```
> # IMDb delivers a version of the page which is unparsable to unknown (and some known) user agents, such as requests'
> # Spoof the old urllib user agent to keep results consistent
> requests = Session()
> requests.headers.update({'User-Agent': 'Python-urllib/2.6'})
> ```
> 
> Changing the user-agent in that last line to a standard browser agent fixes the problem for me.

@grr Would you mind posting the exact line you've entered to replace the User-Agent so I can see what to edit mine to whilst waiting for a permanent fix? Thanks!

FYI, I tried this and it didn't work (note - do I have to recompile anything - I haven't as I am assuming it runs this python script):

`requests.headers.update({'User-Agent': 'Mozilla/5.0'})`
@grr actually, I got it working.  There were multiple places where the path  flexget/components/imdb/utils.py existed and I needed to change every file in the different places. I have no idea which one did the trick, but once I had changed them all, the very line in the above post worked!  Many thanks for the solution.
Aso have this issue
This problem can easily be bypassed defining the `User-Agent` in the request as mentioned above.

@gazpachoking can we please have something as default for the user agent value that is like a browser and that's that?

Resolving patch:
diff --git a/flexget/components/imdb/utils.py b/flexget/components/imdb/utils.py
--- a/flexget/components/imdb/utils.py
+++ b/flexget/components/imdb/utils.py
@@ -15,7 +15,10 @@
 # IMDb delivers a version of the page which is unparsable to unknown (and some known) user agents, such as requests'
 # Spoof the old urllib user agent to keep results consistent
 requests = Session()
-requests.headers.update({'User-Agent': 'Python-urllib/2.6'})
+requests.headers.update(
+    {'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:47.0) Gecko/20100101 Firefox/47.0'}
+)
+# the above line stating Firefox user agent is because imdb requests fail with 403 when bots query the site. Requesting with the right User-Agent seems to solve the issue
 # requests.headers.update({'User-Agent': random.choice(USERAGENTS)})
 
 # this makes most of the titles to be returned in english translation, but not all of them


Test patch: