Repository: Flexget/Flexget

Issue:
Unhandled error in plugin subliminal: decode()
<!---
Before opening an issue, verify:

- Is this a feature request? Post it on https://feathub.com/Flexget/Flexget
- Did you recently upgrade? Look at the Change Log and Upgrade Actions to make sure that you don't need to make any changes to your config https://flexget.com/ChangeLog https://flexget.com/UpgradeActions
- Are you running FlexGet as a daemon? Stop it completely and then start it again https://flexget.com/CLI/daemon
- Did you search to see if the issue already exists? https://github.com/Flexget/Flexget/issues
- Did you fill out the issue template as completely as possible?

The issue template is here because it helps to ensure you submitted all the necessary information the first time, and allows us to more quickly review issues. Please fill it out correctly and do not ignore it, no matter how irrelevant you think it may be. Thanks in advance for your help with this!
--->
### Expected behaviour:
Subliminal sometimes fallback to use Chardet ([ref.](https://github.com/Diaoul/subliminal/blob/dd74383d1cba82829ce720f2e439a65d13ffe7ef/subliminal/subtitle.py#L142)) to guess the encoding of downloaded subtitles. If Chardet could not determine the encoding and return `None`. The plugin should fail the entry and continue.

### Actual behaviour:
Subliminal plugin aborted the task and saved outstanding entries to backlog. However those backlog entries were never processed due to repeated error on next execution.

#### Config:
```yml
  # Download Subtitles
  subtitles_download:
    priority: 600
    template:
      - no_global
    seen: local
    no_entries_ok: yes
    subtitle_list:
      list: subtitles
      recursion_depth: 3
    list_match:
      from:
        - subtitle_list:
            list: subtitles
    subliminal:
      languages:
        - zho
      alternatives:
        - eng
      exact_match: no
      providers: [shooter, addic7ed, opensubtitles]
      single: no
```
  
#### Log:
```
2018-05-24 14:50 INFO     subliminal.providers.shooter subtitles_download Downloading subtitle <ShooterSubtitle u'https://www.shooter.cn/api/subapi.php?fetch=MTUyNzE0NDYzMHw4bVN3YWFRTExPWjk3YlBjOVdTNjM0NkpwVElPQlJYYlNEVW5YLXNqdnE4SVZra0phUldVbk4tYjFIUHA2WUxVWWZBQVl5UzZ5VjMxd0dlRzdjemZXQ2JSMkh2WTlwb2xBbTI4WUNBc09rRkc0YzJwOV9XajRRaDlYX0hTQ0Jxa2dRUnlOQUZVYlN4N1Y3ckVrSUxUT3RFcVdKUzQ5eGM9fMk3jwsZCix9GTnbwwNwec86iiKgtIuPscMyHVSQE2aZ&nonce=R%0F%16B%16%E7%94%F3.%1A%CC%7C%E4L%AB+' [zh]>
2018-05-24 14:50 INFO     subliminal.subtitle subtitles_download Guessing encoding for language zh
2018-05-24 14:50 WARNING  subliminal.subtitle subtitles_download Could not guess encoding from language
2018-05-24 14:50 DEBUG    chardet.charsetprober subtitles_download SHIFT_JIS Japanese prober hit error at byte 8
...
2018-05-24 15:07 DEBUG    chardet.universaldetector subtitles_download windows-1255 Hebrew confidence = 0.0144222722366
2018-05-24 15:07 DEBUG    chardet.universaldetector subtitles_download windows-1255 Hebrew confidence = 0.0144222722366
2018-05-24 15:07 INFO     subliminal.subtitle subtitles_download Chardet found encoding None
2018-05-24 15:07 INFO     subliminal.core subtitles_download Terminating provider shooter
2018-05-24 15:07 CRITICAL task          subtitles_download BUG: Unhandled error in plugin subliminal: decode() argument 1 must be string, not None
2018-05-24 15:07 CRITICAL manager       subtitles_download An unexpected crash has occurred. Writing crash report to ~/.config/flexget/crash_report.2018.05.24.150724032172.log. Please verify you are running the latest version of flexget by using "flexget -V" from CLI or by using version_checker plugin at http://flexget.com/wiki/Plugins/version_checker. You are currently using version 2.13.19
2018-05-24 15:07 DEBUG    manager       subtitles_download Traceback:
Traceback (most recent call last):
  File "/opt/lib/python2.7/site-packages/flexget/task.py", line 486, in __run_plugin
    return method(*args, **kwargs)
  File "/opt/lib/python2.7/site-packages/flexget/event.py", line 23, in __call__
    return self.func(*args, **kwargs)
  File "/opt/lib/python2.7/site-packages/flexget/plugins/output/subtitles_subliminal.py", line 197, in on_task_output
    hearing_impaired=hearing_impaired)
  File "/opt/lib/python2.7/site-packages/subliminal/core.py", line 227, in download_best_subtitles
    if self.download_subtitle(subtitle):
  File "/opt/lib/python2.7/site-packages/subliminal/core.py", line 182, in download_subtitle
    if not subtitle.is_valid():
  File "/opt/lib/python2.7/site-packages/subliminal/subtitle.py", line 85, in is_valid
    if not self.text:
  File "/opt/lib/python2.7/site-packages/subliminal/subtitle.py", line 76, in text
    return self.content.decode(self.guess_encoding(), errors='replace')
TypeError: decode() argument 1 must be string, not None
2018-05-24 15:07 WARNING  task          subtitles_download Aborting task (plugin: subliminal)
2018-05-24 15:07 DEBUG    backlog       subtitles_download Remembering all entries to backlog because of task abort.
2018-05-24 15:07 DEBUG    backlog       subtitles_download Saving The.Wrestler.2008.1080p.BluRay.x264-REFiNED
...
2018-05-24 15:07 DEBUG    task_queue                    task subtitles_download aborted: TaskAbort(reason=BUG: Unhandled error in plugin subliminal: decode() argument 1 must be string, not None, silent=False)

```

### Additional information:

- FlexGet version: 2.13.19
- Python version: 2.7.12
- Installation method: pip install flexget globally
- Using daemon (yes/no): yes
- OS and version: Embedded Linux of WD Mybook Live Duo
- Return of `uname -a`: `Linux 2.6.32.11-svn70860 #1 Thu May 17 13:32:51 PDT 2012 ppc GNU/Linux`

<!---
In config and debug/crash logs, remember to redact any personal or sensitive information such as passwords, API keys, private URLs and so on.

Please verify that the following data is present before submitting your issue:

- Link to a paste service or paste above the relevant config (preferably full config, including templates if present). Please make sure the paste does not expire, if possible.
- Link to a paste service or paste above debug-level logs of the relevant task/s (use `flexget -L debug execute --tasks <Task_name>`).
- FlexGet version (use `flexget -V` to get it).
- Full Python version, for example `2.7.11` (use `python -V` to get it). Note that FlexGet is not supported for use with Python v3.0, 3.1, 3.2 or 3.6.
- Installation method (pip, git install, etc).
- Whether or not you're running FlexGet as a daemon.
- OS and version.
- Attach crash log if one was generated, in addition to the debug-level log. It can be found in the directory with your config file.
--->

Unhandled error in plugin subliminal: decode()
<!---
Before opening an issue, verify:

- Is this a feature request? Post it on https://feathub.com/Flexget/Flexget
- Did you recently upgrade? Look at the Change Log and Upgrade Actions to make sure that you don't need to make any changes to your config https://flexget.com/ChangeLog https://flexget.com/UpgradeActions
- Are you running FlexGet as a daemon? Stop it completely and then start it again https://flexget.com/CLI/daemon
- Did you search to see if the issue already exists? https://github.com/Flexget/Flexget/issues
- Did you fill out the issue template as completely as possible?

The issue template is here because it helps to ensure you submitted all the necessary information the first time, and allows us to more quickly review issues. Please fill it out correctly and do not ignore it, no matter how irrelevant you think it may be. Thanks in advance for your help with this!
--->
### Expected behaviour:
Subliminal sometimes fallback to use Chardet ([ref.](https://github.com/Diaoul/subliminal/blob/dd74383d1cba82829ce720f2e439a65d13ffe7ef/subliminal/subtitle.py#L142)) to guess the encoding of downloaded subtitles. If Chardet could not determine the encoding and return `None`. The plugin should fail the entry and continue.

### Actual behaviour:
Subliminal plugin aborted the task and saved outstanding entries to backlog. However those backlog entries were never processed due to repeated error on next execution.

#### Config:
```yml
  # Download Subtitles
  subtitles_download:
    priority: 600
    template:
      - no_global
    seen: local
    no_entries_ok: yes
    subtitle_list:
      list: subtitles
      recursion_depth: 3
    list_match:
      from:
        - subtitle_list:
            list: subtitles
    subliminal:
      languages:
        - zho
      alternatives:
        - eng
      exact_match: no
      providers: [shooter, addic7ed, opensubtitles]
      single: no
```
  
#### Log:
```
2018-05-24 14:50 INFO     subliminal.providers.shooter subtitles_download Downloading subtitle <ShooterSubtitle u'https://www.shooter.cn/api/subapi.php?fetch=MTUyNzE0NDYzMHw4bVN3YWFRTExPWjk3YlBjOVdTNjM0NkpwVElPQlJYYlNEVW5YLXNqdnE4SVZra0phUldVbk4tYjFIUHA2WUxVWWZBQVl5UzZ5VjMxd0dlRzdjemZXQ2JSMkh2WTlwb2xBbTI4WUNBc09rRkc0YzJwOV9XajRRaDlYX0hTQ0Jxa2dRUnlOQUZVYlN4N1Y3ckVrSUxUT3RFcVdKUzQ5eGM9fMk3jwsZCix9GTnbwwNwec86iiKgtIuPscMyHVSQE2aZ&nonce=R%0F%16B%16%E7%94%F3.%1A%CC%7C%E4L%AB+' [zh]>
2018-05-24 14:50 INFO     subliminal.subtitle subtitles_download Guessing encoding for language zh
2018-05-24 14:50 WARNING  subliminal.subtitle subtitles_download Could not guess encoding from language
2018-05-24 14:50 DEBUG    chardet.charsetprober subtitles_download SHIFT_JIS Japanese prober hit error at byte 8
...
2018-05-24 15:07 DEBUG    chardet.universaldetector subtitles_download windows-1255 Hebrew confidence = 0.0144222722366
2018-05-24 15:07 DEBUG    chardet.universaldetector subtitles_download windows-1255 Hebrew confidence = 0.0144222722366
2018-05-24 15:07 INFO     subliminal.subtitle subtitles_download Chardet found encoding None
2018-05-24 15:07 INFO     subliminal.core subtitles_download Terminating provider shooter
2018-05-24 15:07 CRITICAL task          subtitles_download BUG: Unhandled error in plugin subliminal: decode() argument 1 must be string, not None
2018-05-24 15:07 CRITICAL manager       subtitles_download An unexpected crash has occurred. Writing crash report to ~/.config/flexget/crash_report.2018.05.24.150724032172.log. Please verify you are running the latest version of flexget by using "flexget -V" from CLI or by using version_checker plugin at http://flexget.com/wiki/Plugins/version_checker. You are currently using version 2.13.19
2018-05-24 15:07 DEBUG    manager       subtitles_download Traceback:
Traceback (most recent call last):
  File "/opt/lib/python2.7/site-packages/flexget/task.py", line 486, in __run_plugin
    return method(*args, **kwargs)
  File "/opt/lib/python2.7/site-packages/flexget/event.py", line 23, in __call__
    return self.func(*args, **kwargs)
  File "/opt/lib/python2.7/site-packages/flexget/plugins/output/subtitles_subliminal.py", line 197, in on_task_output
    hearing_impaired=hearing_impaired)
  File "/opt/lib/python2.7/site-packages/subliminal/core.py", line 227, in download_best_subtitles
    if self.download_subtitle(subtitle):
  File "/opt/lib/python2.7/site-packages/subliminal/core.py", line 182, in download_subtitle
    if not subtitle.is_valid():
  File "/opt/lib/python2.7/site-packages/subliminal/subtitle.py", line 85, in is_valid
    if not self.text:
  File "/opt/lib/python2.7/site-packages/subliminal/subtitle.py", line 76, in text
    return self.content.decode(self.guess_encoding(), errors='replace')
TypeError: decode() argument 1 must be string, not None
2018-05-24 15:07 WARNING  task          subtitles_download Aborting task (plugin: subliminal)
2018-05-24 15:07 DEBUG    backlog       subtitles_download Remembering all entries to backlog because of task abort.
2018-05-24 15:07 DEBUG    backlog       subtitles_download Saving The.Wrestler.2008.1080p.BluRay.x264-REFiNED
...
2018-05-24 15:07 DEBUG    task_queue                    task subtitles_download aborted: TaskAbort(reason=BUG: Unhandled error in plugin subliminal: decode() argument 1 must be string, not None, silent=False)

```

### Additional information:

- FlexGet version: 2.13.19
- Python version: 2.7.12
- Installation method: pip install flexget globally
- Using daemon (yes/no): yes
- OS and version: Embedded Linux of WD Mybook Live Duo
- Return of `uname -a`: `Linux 2.6.32.11-svn70860 #1 Thu May 17 13:32:51 PDT 2012 ppc GNU/Linux`

<!---
In config and debug/crash logs, remember to redact any personal or sensitive information such as passwords, API keys, private URLs and so on.

Please verify that the following data is present before submitting your issue:

- Link to a paste service or paste above the relevant config (preferably full config, including templates if present). Please make sure the paste does not expire, if possible.
- Link to a paste service or paste above debug-level logs of the relevant task/s (use `flexget -L debug execute --tasks <Task_name>`).
- FlexGet version (use `flexget -V` to get it).
- Full Python version, for example `2.7.11` (use `python -V` to get it). Note that FlexGet is not supported for use with Python v3.0, 3.1, 3.2 or 3.6.
- Installation method (pip, git install, etc).
- Whether or not you're running FlexGet as a daemon.
- OS and version.
- Attach crash log if one was generated, in addition to the debug-level log. It can be found in the directory with your config file.
--->



Discussion:
While the plugin should not crash, it sounds like a bug in Subliminal.
Agreed.  I did a little research on subliminal github page and found the subtitle encoding was complex enough to be resolved shortly, yet I just opened an issue (https://github.com/Diaoul/subliminal/issues/921) to get author's notice.

I am not familair with python. Just thinking that it would be nice if the plugin can ignore such error and let the task continue.
While the plugin should not crash, it sounds like a bug in Subliminal.
Agreed.  I did a little research on subliminal github page and found the subtitle encoding was complex enough to be resolved shortly, yet I just opened an issue (https://github.com/Diaoul/subliminal/issues/921) to get author's notice.

I am not familair with python. Just thinking that it would be nice if the plugin can ignore such error and let the task continue.

Resolving patch:
diff --git a/flexget/plugins/output/subtitles_subliminal.py b/flexget/plugins/output/subtitles_subliminal.py
--- a/flexget/plugins/output/subtitles_subliminal.py
+++ b/flexget/plugins/output/subtitles_subliminal.py
@@ -191,15 +191,19 @@ def on_task_output(self, task, config):
                         # Gather the subtitles for the alternative languages too, to avoid needing to search the sites
                         # again. They'll just be ignored if the main languages are found.
                         all_subtitles = provider_pool.list_subtitles(video, entry_languages | alternative_languages)
-
-                        subtitles = provider_pool.download_best_subtitles(all_subtitles, video, entry_languages,
-                                                                          min_score=msc,
-                                                                          hearing_impaired=hearing_impaired)
+                        try:
+                            subtitles = provider_pool.download_best_subtitles(all_subtitles, video, entry_languages,
+                                                                              min_score=msc,
+                                                                              hearing_impaired=hearing_impaired)
+                        except TypeError as e:
+                            log.error('Downloading subtitles failed due to a bug in subliminal. Please see'
+                                      'https://github.com/Diaoul/subliminal/issues/921. Error: %s', e)
+                            subtitles = []
                         if subtitles:
                             downloaded_subtitles[video].extend(subtitles)
                             log.info('Subtitles found for %s', entry['location'])
                         else:
-                            # only try to download for alternatives that aren't alread downloaded
+                            # only try to download for alternatives that aren't already downloaded
                             subtitles = provider_pool.download_best_subtitles(all_subtitles, video,
                                                                               alternative_languages, min_score=msc,
                                                                               hearing_impaired=hearing_impaired)


Test patch: