Repository: Flexget/Flexget

Issue:
Tvmaze error finding show
Hi I recently upgradet to flexget with enabled tvMaze by default.
running the Series Tasks returns some errors from tvmaze like:

> 2015-11-26 18:36 WARNING  est_series_tvmaze SERIES Could not found show on TVMaze: u'Orange%20is%20the%20new%20Black not found'

A quick lookup on their webpage showed thes show can be found.

After that I did recive

> 2015-11-26 18:37 CRITICAL task          SERIES_ARCHIVE BUG: Unhandled error in plugin discover: u'\xfc'
> 2015-11-26 18:37 CRITICAL manager       SERIES_ARCHIVE An unexpected crash has occurred. Writing crash report to /home/USERNAME/.flexget/crash_report.2015.11.26.183724001553.log

the chrash log is as follows

> Traceback (most recent call last):
>   File "/usr/local/lib/python2.7/dist-packages/flexget/task.py", line 438, in **run_plugin
>     return method(_args, *_kwargs)
>   File "/usr/local/lib/python2.7/dist-packages/flexget/event.py", line 22, in __call**
>     return self.func(_args, *_kwargs)
>   File "/usr/local/lib/python2.7/dist-packages/flexget/plugins/input/discover.py", line 268, in on_task_input
>     entries = self.estimated(entries)
>   File "/usr/local/lib/python2.7/dist-packages/flexget/plugins/input/discover.py", line 190, in estimated
>     est_date = estimator.estimate(entry)
>   File "/usr/local/lib/python2.7/dist-packages/flexget/plugins/est_released.py", line 29, in estimate
>     estimate = estimator(entry)
>   File "/usr/local/lib/python2.7/dist-packages/flexget/plugins/estimators/est_release_series_tvmaze.py", line 49, in estimate
>     tvmaze_show = get_show(**kwargs)
>   File "/usr/local/lib/python2.7/dist-packages/pytvmaze/tvmaze.py", line 197, in get_show
>     show_language, show_country)
>   File "/usr/local/lib/python2.7/dist-packages/pytvmaze/tvmaze.py", line 206, in get_show_by_search
>     shows = get_show_list(show_name)
>   File "/usr/local/lib/python2.7/dist-packages/pytvmaze/tvmaze.py", line 246, in get_show_list
>     shows = show_search(show_name)
>   File "/usr/local/lib/python2.7/dist-packages/pytvmaze/tvmaze.py", line 264, in show_search
>     show = url_quote(show)
>   File "/usr/lib/python2.7/urllib.py", line 1288, in quote
>     return ''.join(map(quoter, s))
> KeyError: u'\xfc'

Thanks in advance!



Discussion:
Are you sure you're on the latest version of flexget? 

yes, did the update 5 minutes before I posted the ticket

> $ flexget --version
> 1.2.395
> You are on the latest release.

Ok, thanks, i'll investigate this. 

Another question, are you running flexget from pip or git? Can you checked the installed version of pytvmaze please? 

Hi i am running from pip.
pytvmaze:

> ## $ pip show pytvmaze
> 
> Name: pytvmaze
> Version: 1.3.4
> Location: /usr/local/lib/python2.7/dist-packages
> Requires: 

EDIT:

~~What version of python are you using?~~

It seems like you may be passing unicode into the get_show() method, which urlquote doesn't like.  Can you show me what you are passing into the show_name parameter of the get_show() method?

Is get_show a public method from pytvmaze? If so, passing unicode in sounds correct. It should probably just be encoded first before handing to urllib. 

Some more testing is making it seem like this isn't the issue, but just for grins can you try to reproduce the same error with the latest dev of pytvmaze? https://github.com/srob650/pytvmaze/tree/dev

I'm having the same issue with the latest flexget and pytvmaze from pip:

> ### $ flexget execute --now --discover-now --loglevel debug
> 
> ```
> 2015-11-29 20:30 CRITICAL plugin                        Error while registering plugin est_released_movies. A plugin with the same name is already registered
> 2015-11-29 20:30 CRITICAL plugin                        Error while registering plugin estimate_release. A plugin with the same name is already registered
> 2015-11-29 20:30 VERBOSE  task_queue                    There are 6 tasks to execute. Shutdown will commence when they have completed.
> 2015-11-29 20:30 INFO     interval      discover-tv-weekly Ignoring interval because of --now
> 2015-11-29 20:30 VERBOSE  input_cache   discover-tv-weekly Restored 16 entries from db cache
> 2015-11-29 20:30 VERBOSE  discover      discover-tv-weekly Discovering 16 titles ...
> 2015-11-29 20:30 INFO     discover      discover-tv-weekly Ignoring interval because of --discover-now
> 2015-11-29 20:30 WARNING  est_series_tvmaze discover-tv-weekly Could not found show on TVMaze: u'The%20Good%20Wife not found'
> 2015-11-29 20:32 WARNING  est_series_tvmaze discover-tv-weekly Could not found show on TVMaze: u'Lost%20Girl not found'
> 2015-11-29 20:32 WARNING  est_series_tvmaze discover-tv-weekly Could not found show on TVMaze: u'Homeland not found'
> 2015-11-29 20:33 CRITICAL task          discover-tv-weekly BUG: Unhandled error in plugin discover: u'Maze id 383 not found'
> 2015-11-29 20:33 CRITICAL manager       discover-tv-weekly An unexpected crash has occurred. Writing crash report to /home/xbmc/.flexget/crash_report.2015.11.29.203349511639.log
> 2015-11-29 20:33 WARNING  task          discover-tv-weekly Aborting task (plugin: discover)
> ```
> 
> The related part from the crashlog:
> 
> ### $ cat .flexget/crash_report.2015.11.29.203349511639.log
> 
> ``` python
> 2015-11-29 20:30 WARNING  est_series_tvmaze discover-tv-weekly Could not found show on TVMaze: u'The%20Good%20Wife not found'
> 2015-11-29 20:32 WARNING  est_series_tvmaze discover-tv-weekly Could not found show on TVMaze: u'Lost%20Girl not found'
> 2015-11-29 20:32 WARNING  est_series_tvmaze discover-tv-weekly Could not found show on TVMaze: u'Homeland not found'
> 2015-11-29 20:33 CRITICAL task          discover-tv-weekly BUG: Unhandled error in plugin discover: u'Maze id 383 not found'
> Traceback (most recent call last):
>   File "/usr/local/lib/python2.7/dist-packages/flexget/task.py", line 438, in __run_plugin
>     return method(*args, **kwargs)
>   File "/usr/local/lib/python2.7/dist-packages/flexget/event.py", line 22, in __call__
>     return self.func(*args, **kwargs)
>   File "/usr/local/lib/python2.7/dist-packages/flexget/plugins/input/discover.py", line 268, in on_task_input
>     entries = self.estimated(entries)
>   File "/usr/local/lib/python2.7/dist-packages/flexget/plugins/input/discover.py", line 190, in estimated
>     est_date = estimator.estimate(entry)
>   File "/usr/local/lib/python2.7/dist-packages/flexget/plugins/est_released.py", line 29, in estimate
>     estimate = estimator(entry)
>   File "/usr/local/lib/python2.7/dist-packages/flexget/plugins/estimators/est_release_series_tvmaze.py", line 49, in estimate
>     tvmaze_show = get_show(**kwargs)
>   File "/usr/local/lib/python2.7/dist-packages/pytvmaze/tvmaze.py", line 208, in get_show
>     show_language, show_country, embed=embed)
>   File "/usr/local/lib/python2.7/dist-packages/pytvmaze/tvmaze.py", line 217, in get_show_by_search
>     shows = get_show_list(show_name, embed)
>   File "/usr/local/lib/python2.7/dist-packages/pytvmaze/tvmaze.py", line 260, in get_show_list
>     for show in shows
>   File "/usr/local/lib/python2.7/dist-packages/pytvmaze/tvmaze.py", line 343, in show_main_info
>     raise IDNotFound('Maze id ' + str(maze_id) + ' not found')
> IDNotFound: u'Maze id 383 not found'
> ```

The most weird part is the `Maze id 383`, because http://www.tvmaze.com/shows/383 points to some show called "Peep Show", which is definitely not in my flexget database:

> ### $ flexget series list
> 
> ```
> 2015-11-29 20:55 CRITICAL plugin                        Error while registering plugin est_released_movies. A plugin with the same name is already registered
> 2015-11-29 20:55 CRITICAL plugin                        Error while registering plugin estimate_release. A plugin with the same name is already registered
> -------------------------------------------------------------------------------
>  Name                           Latest     Age        Downloaded
> -------------------------------------------------------------------------------
>  Angel From Hell                N/A        N/A        N/A
>  Ascension                      N/A        N/A        N/A
>  Bones                          S11E08     9d 5h      hdtv h264
>  Brooklyn Nine-Nine             S03E08 +1  2d 4h      hdtv h264-proper
>  Castle                         S08E08 +1  2d 4h      hdtv h264
>  Dark Matter                    S01E13     87d 15h    hdtv h264
>  Faking It                      S02E15 +2  61d 9h     hdtv h264
>  Grandfathered                  S01E08 +1  2d 4h      hdtv h264
>  Homeland                       S05E08     2d 4h      hdtv h264
>  iZombie                        S02E07     11d 5h     hdtv h264
>  Lost Girl                      S05E16 +1  34d 4h     hdtv h264
>  Mission Control                N/A        N/A        N/A
>  Mr. Robot                      S01E10     87d 15h    hdtv h264
>  Orange Is the New Black        S03E06 +1  167d 21h   720p webrip h264 dd5.1
>  Rizzoli & Isles                S06E12 +1  87d 15h    hdtv h264
> Rosewood                       S01E01     1h         hdtv h264
>  Star Wars Rebels               S02E07 +1  10d 2h     hdtv h264
>  The Big Bang Theory            S09E09 +1  9d 5h      hdtv h264
>  The Good Wife                  S07E07 +2  12d 3h     hdtv h264
>  Veep                           S04E10 +1  165d 9h    hdtv h264
> -------------------------------------------------------------------------------
> ```
> 
> ### $ pip show flexget
> 
> ---
> 
> Name: FlexGet
> Version: 1.2.395
> Location: /usr/local/lib/python2.7/dist-packages
> Requires: FeedParser, SQLAlchemy, PyYAML, beautifulsoup4, html5lib, PyRSS2Gen, pynzb, progressbar, rpyc, jinja2, requests, python-dateutil, jsonschema, tmdb3, path.py, guessit, apscheduler, flask, flask-restful, ordereddict, flask-restplus, cherrypy, flask-assets, cssmin, flask-compress, flask-login, pyparsing, pyScss, pytvmaze
> 
> ### $ pip show pytvmaze
> 
> ---
> 
> Name: pytvmaze
> Version: 1.3.5
> Location: /usr/local/lib/python2.7/dist-packages
> Requires:

Very strange.  All 4 of those shows return fine when using pytvmaze alone with:

``` python
s = pytvmaze.get_show(show_name='homeland')
s = pytvmaze.get_show(show_name='the good wife')
s = pytvmaze.get_show(show_name='lost girl')
s = pytvmaze.get_show(show_name='orange is the new black')
```

Maybe it's not flexget related, because the python console output for me is the following for `homeland`:

> ### $ python
> 
> ``` python
> Python 2.7.6 (default, Jun 22 2015, 17:58:13)
> [GCC 4.8.2] on linux2
> Type "help", "copyright", "credits" or "license" for more information.
> >> import pytvmaze
> >> s = pytvmaze.get_show(show_name='homeland')
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
>   File "/usr/local/lib/python2.7/dist-packages/pytvmaze/tvmaze.py", line 211, in get_show
>     show_language, show_country, embed=embed)
>   File "/usr/local/lib/python2.7/dist-packages/pytvmaze/tvmaze.py", line 220, in get_show_by_search
>     shows = get_show_list(show_name, embed)
>   File "/usr/local/lib/python2.7/dist-packages/pytvmaze/tvmaze.py", line 260, in get_show_list
>     shows = show_search(show_name)
>   File "/usr/local/lib/python2.7/dist-packages/pytvmaze/tvmaze.py", line 284, in show_search
>     raise ShowNotFound(str(show) + ' not found')
> pytvmaze.exceptions.ShowNotFound: u'homeland not found'
> ```

And for `veep`:

``` python
>>> s = pytvmaze.get_show(show_name='veep')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python2.7/dist-packages/pytvmaze/tvmaze.py", line 211, in get_show
    show_language, show_country, embed=embed)
  File "/usr/local/lib/python2.7/dist-packages/pytvmaze/tvmaze.py", line 220, in get_show_by_search
    shows = get_show_list(show_name, embed)
  File "/usr/local/lib/python2.7/dist-packages/pytvmaze/tvmaze.py", line 263, in get_show_list
    for show in shows
  File "/usr/local/lib/python2.7/dist-packages/pytvmaze/tvmaze.py", line 346, in show_main_info
    raise IDNotFound('Maze id ' + str(maze_id) + ' not found')
pytvmaze.exceptions.IDNotFound: u'Maze id 383 not found'
```

(So somehow `veep` became _Peep Show_ with `Maze id 393`...)

However it seems to be okay with `Veep` or `Homeland` as arguments.

``` python
>>> s = pytvmaze.get_show(show_name='Veep')
>>> s = pytvmaze.get_show(show_name='Homeland')
```

Even if pytvmaze were case sensitive (AFAIK it's not), based on my previous log, flexget did pass the arguments with uppercase first letters, which is working fine for me in the console. I'm completely lost now.

I can't figure out why it would be doing that.  Here's the output from my console with the same tests:

``` python
>>> s = pytvmaze.get_show(show_name='veep')
>>> s.name
u'Veep'
>>> s.network
{u'country': {u'timezone': u'America/New_York', u'code': u'US', u'name': u'United States'}, u'id': 8, u'name': u'HBO'}


>>> s = pytvmaze.get_show(show_name='homeland')
>>> s.name
u'Homeland'
>>> s.network
{u'country': {u'timezone': u'America/New_York', u'code': u'US', u'name': u'United States'}, u'id': 9, u'name': u'Showtime'}
```

Maybe try deleting and re-installing pytvmaze? (long shot).
Any reason you're on Python 2.7.6 instead of 2.7.10?  FWIW I'm testing on 2.7.10 as well as 3.5.0

No specific reason for the Python version. This is the newest version in the Ubuntu Trusty repo and I never had a cause to manually upgrade it. Removing and reinstalling pymaze did nothing (I did it already a few times before your suggestion)
However I ran a quick network diagnostics and my latency was a bit high (around 30-50 ms, it wasn't even noticable for example browsing the web). I restarted the modem, the latency is now <10 ms and pytvmaze (and flexget) works fine. The sofware side of things is the same as before, I didn't even quit the python console. Maybe pytvmaze is this sensitive about network quality?

edit: typo

Hmm, very interesting.  I'm no expert on this kind of thing, but it doesn't seem possible that you could be getting incomplete data from tvmaze.com because of a timeout, I think it's all or nothing.  If you were really getting incomplete data you would probably see other weird errors in pytvmaze like show objects not getting initialized properly.  Someone correct me if this seems wrong

Haven't heard from the OP @kecksk0enig in a while either, was hoping they would try testing with the latest pytvmaze dev build to see if that fixed their problem.

This is very strange, my tests seems to work as well. I'll bet this relates to encoding though, this is the only thing that makes sense. 

Maybe this is something else, like tvmaze flood protection or connection issues, and not a functional error. 
@srob650 we should add some better response verification of the request to check this. 

Edit: just proved my theory, this is indeed a flooding issue. Tvmaze starts blocking requests after a certain request to time ratio has been met. 
@srob650 we'll need to explicitly check for this and raise a relevant exception. I'll try to push something tomorrow morning.

Awesome, thanks @liiight !

This is however different from the issue that this topic is about so i'll open a new issue.
@kecksk0enig please verify that the problem does not occur with pytvmaze v1.3.5

hi i upgraded pytvmaze to Version: 1.3.5
I still get an exception from pytvmaze but it looks different:

> 2015-11-30 13:41 CRITICAL task          SERIES_ARCHIVE BUG: Unhandled error in plugin discover: u'Maze id 5 not found'
> 2015-11-30 13:41 CRITICAL manager       SERIES_ARCHIVE An unexpected crash has occurred. Writing crash report to /home/USERNAME/.flexget/crash_report.2015.11.30.134136312072.log

The relevant logs from crash_report:

> 2015-11-30 13:41 CRITICAL task          SERIES_ARCHIVE_DE BUG: Unhandled error in plugin discover: u'Maze id 5 not found'
> Traceback (most recent call last):
>   File "/usr/local/lib/python2.7/dist-packages/flexget/task.py", line 438, in **run_plugin
>     return method(_args, *_kwargs)
>   File "/usr/local/lib/python2.7/dist-packages/flexget/event.py", line 22, in __call**
>     return self.func(_args, *_kwargs)
>   File "/usr/local/lib/python2.7/dist-packages/flexget/plugins/input/discover.py", line 268, in on_task_input
>     entries = self.estimated(entries)
>   File "/usr/local/lib/python2.7/dist-packages/flexget/plugins/input/discover.py", line 190, in estimated
>     est_date = estimator.estimate(entry)
>   File "/usr/local/lib/python2.7/dist-packages/flexget/plugins/est_released.py", line 29, in estimate
>     estimate = estimator(entry)
>   File "/usr/local/lib/python2.7/dist-packages/flexget/plugins/estimators/est_release_series_tvmaze.py", line 49, in estimate
>     tvmaze_show = get_show(**kwargs)
>   File "/usr/local/lib/python2.7/dist-packages/pytvmaze/tvmaze.py", line 208, in get_show
>     show_language, show_country, embed=embed)
>   File "/usr/local/lib/python2.7/dist-packages/pytvmaze/tvmaze.py", line 217, in get_show_by_search
>     shows = get_show_list(show_name, embed)
>   File "/usr/local/lib/python2.7/dist-packages/pytvmaze/tvmaze.py", line 260, in get_show_list
>     for show in shows
>   File "/usr/local/lib/python2.7/dist-packages/pytvmaze/tvmaze.py", line 343, in show_main_info
>     raise IDNotFound('Maze id ' + str(maze_id) + ' not found')
> IDNotFound: u'Maze id 5 not found'

Do you get that at the first try? Meaning, at the first episode that need an airdate?

No.
Here is the complete debug output from running the affected task:
http://paste.ubuntu.com/13574284/

I see. I'm pretty sure I know the cause of this, it's a socket error that caused urlib reply to crash. Currently there is a blanket 'except' on the response from urllib but I pushed a pr to pytvmaze that should address this issue. After it's merged, I'll push a pr to flexget to support this and not crash.

Anyways, this relates to a different issue than the one you opened originally: https://github.com/Flexget/Flexget/issues/664

The original issue can be closed IMO, as all Unicode text should now work.
BTW, what machine are your running Flexget on? I wasn't able to reproduce this on any other device other than a shitty netbook I own. When I tried using powerful machines I just couldn't get it to crash.

okay I will watch #664.
The commit which fixed the Unicode text  was in pytvmaze or in flexget?
Are the flexget requirements for pytvmaze already fixed?
Should I as Op close the bug or is ther some kind of other review before closing the bug?

You're right, i'll push a pr with just the pytvmaze version update for now.
No need to close this then, it'll closed when the fix will be merged.


Resolving patch:
diff --git a/pavement.py b/pavement.py
--- a/pavement.py
+++ b/pavement.py
@@ -33,7 +33,7 @@
     'tmdb3', 'path.py', 'guessit>=0.9.3, <0.10.4', 'apscheduler',
     'flask>=0.7', 'flask-restful>=0.3.3', 'ordereddict>=1.1', 'flask-restplus==0.7.2', 'cherrypy>=3.7.0',
     'flask-assets>=0.11', 'cssmin>=0.2.0', 'flask-compress>=1.2.1', 'flask-login>=0.3.2', 'pyparsing>=2.0.3',
-    'pyScss>=1.3.4', 'pytvmaze>=1.3.4'
+    'pyScss>=1.3.4', 'pytvmaze>=1.3.5'
 ]
 
 if sys.version_info < (2, 7):


Test patch: