Repository: pymedusa/Medusa

Issue:
Dupe logs in Warning view
Branch/Commit: Develop
What happened: Logs from backlog showing "dupped" in warning view
What you expected: To only show 1 of the same in warning view
Small snipped from warning logs, with only important dupe logs:

```
2016-05-09 12:36:10 SEARCHQUEUE-BACKLOG-78904 :: [Demonoid] :: [55c0658] Unable to find episode with date 0005-02-16 for show Australian Story, skipping
2016-05-09 12:28:43 SEARCHQUEUE-BACKLOG-287254 :: [Rarbg] :: [55c0658] Cant find search_tvdb in database. Are you sure it exists?
2016-05-09 12:23:34 SEARCHQUEUE-BACKLOG-278798 :: [Rarbg] :: [55c0658] Cant find search_tvdb in database. Are you sure it exists?
2016-05-09 12:08:11 SEARCHQUEUE-BACKLOG-254184 :: [Rarbg] :: [55c0658] Cant find search_tvdb in database. Are you sure it exists?
2016-05-09 12:06:05 SEARCHQUEUE-BACKLOG-254184 :: [Demonoid] :: [55c0658] Unable to find episode with date 0005-02-16 for show Australian Story, skipping
2016-05-09 11:51:25 SEARCHQUEUE-BACKLOG-277096 :: [Rarbg] :: [55c0658] Cant find search_tvdb in database. Are you sure it exists?
2016-05-09 11:28:54 SEARCHQUEUE-BACKLOG-110381 :: [Demonoid] :: [55c0658] Unable to find episode with date 0005-02-16 for show Australian Story, skipping
2016-05-09 10:47:37 SEARCHQUEUE-BACKLOG-82186 :: [Rarbg] :: [55c0658] Cant find search_tvdb in database. Are you sure it exists?
2016-05-09 10:42:26 SEARCHQUEUE-BACKLOG-82186 :: [Demonoid] :: [55c0658] Unable to find episode with date 0005-02-16 for show Australian Story, skipping
2016-05-09 10:26:42 SEARCHQUEUE-BACKLOG-83982 :: [Rarbg] :: [55c0658] Cant find search_tvdb in database. Are you sure it exists?
2016-05-09 10:06:53 SEARCHQUEUE-BACKLOG-100741 :: [Rarbg] :: [55c0658] Cant find search_tvdb in database. Are you sure it exists?
2016-05-09 09:59:06 SEARCHQUEUE-BACKLOG-100741 :: [Demonoid] :: [55c0658] Unable to find episode with date 0005-02-16 for show Australian Story, skipping
2016-05-09 09:56:27 SEARCHQUEUE-BACKLOG-281341 :: [Rarbg] :: [55c0658] Cant find search_tvdb in database. Are you sure it exists?
2016-05-09 09:44:10 SEARCHQUEUE-BACKLOG-271546 :: [Rarbg] :: [55c0658] Cant find search_tvdb in database. Are you sure it exists?
2016-05-09 09:40:27 SEARCHQUEUE-BACKLOG-273859 :: [Rarbg] :: [55c0658] Cant find search_tvdb in database. Are you sure it exists?
2016-05-09 09:32:42 SEARCHQUEUE-BACKLOG-263546 :: [Rarbg] :: [55c0658] Cant find search_tvdb in database. Are you sure it exists?
2016-05-09 09:28:50 SEARCHQUEUE-BACKLOG-272876 :: [Rarbg] :: [55c0658] Cant find search_tvdb in database. Are you sure it exists?
2016-05-09 09:20:04 SEARCHQUEUE-BACKLOG-292136 :: [Rarbg] :: [55c0658] Cant find search_tvdb in database. Are you sure it exists?
2016-05-09 09:13:24 SEARCHQUEUE-BACKLOG-294117 :: [Rarbg] :: [55c0658] Cant find search_tvdb in database. Are you sure it exists?
2016-05-09 08:50:12 SEARCHQUEUE-BACKLOG-268592 :: [Demonoid] :: [55c0658] Unable to find episode with date 0005-02-16 for show Australian Story, skipping
2016-05-09 08:38:27 SEARCHQUEUE-BACKLOG-280473 :: [Rarbg] :: [55c0658] Cant find search_tvdb in database. Are you sure it exists?
2016-05-09 08:07:33 SEARCHQUEUE-BACKLOG-259765 :: [Demonoid] :: [55c0658] Unable to find episode with date 0005-02-16 for show Australian Story, skipping
2016-05-09 08:02:15 SEARCHQUEUE-BACKLOG-77904 :: [Rarbg] :: [55c0658] Cant find search_tvdb in database. Are you sure it exists?
2016-05-09 07:39:48 SEARCHQUEUE-BACKLOG-72514 :: [Rarbg] :: [55c0658] Cant find search_tvdb in database. Are you sure it exists?
2016-05-09 07:32:17 SEARCHQUEUE-BACKLOG-247897 :: [Demonoid] :: [55c0658] Unable to find episode with date 0005-02-16 for show Australian Story, skipping
2016-05-09 07:16:43 SEARCHQUEUE-BACKLOG-77466 :: [Rarbg] :: [55c0658] Cant find search_tvdb in database. Are you sure it exists?
```



Discussion:
Aren't they all in different threads since the number after BACKLOG is different for all of them?

Yes  but for example  daily does not show the diferente ones just `SEARCHQUEUE-DAILY-SEARCH`,  that's probably what makes it show more than once. 
Even tough the same warning in a backlog is the same in  another backlog,  a time out this backlog  search  is the same as in the next one :P,  only need to be informed  once via  Warning view

Can't find the code that filter by error/warning message.


Resolving patch:
diff --git a/sickbeard/server/web/core/error_logs.py b/sickbeard/server/web/core/error_logs.py
--- a/sickbeard/server/web/core/error_logs.py
+++ b/sickbeard/server/web/core/error_logs.py
@@ -1,65 +1,111 @@
 # coding=utf-8
+"""Route to error logs web page."""
 
 from __future__ import unicode_literals
 
 import io
 import os
 import re
-from tornado.routes import route
+
+from mako.filters import html_escape
 import sickbeard
 from sickbeard import (
     classes, logger, ui,
 )
+from sickbeard.server.web.core.base import PageTemplate, WebRoot
 from sickrage.helper.encoding import ek
-from sickbeard.server.web.core.base import WebRoot, PageTemplate
+from tornado.routes import route
+
+
+# log regular expression: 2016-08-06 15:58:34 ERROR    DAILYSEARCHER :: [d4ea5af] Exception generated in thread DAILYSEARCHER
+log_re = re.compile(r'^(?P<year>\d{4})-(?P<month>\d{2})-(?P<day>\d{2})\s+'
+                    r'(?P<hour>\d{2}):(?P<minute>\d{2}):(?P<second>\d{2})\s+'
+                    r'(?P<log_level>[A-Z]+)\s+(?P<log_name>.+?)\s+::\s+(?P<log_message>.*)$')
+
+# log name filters
+log_name_filters = {
+    '<NONE>': html_escape('<No Filter>'),
+    'DAILYSEARCHER': 'Daily Searcher',
+    'BACKLOG': 'Backlog',
+    'SHOWUPDATER': 'Show Updater',
+    'CHECKVERSION': 'Check Version',
+    'SHOWQUEUE': 'Show Queue',
+    'SEARCHQUEUE': 'Search Queue (All)',
+    'SEARCHQUEUE-DAILY-SEARCH': 'Search Queue (Daily Searcher)',
+    'SEARCHQUEUE-BACKLOG': 'Search Queue (Backlog)',
+    'SEARCHQUEUE-MANUAL': 'Search Queue (Manual)',
+    'SEARCHQUEUE-FORCED': 'Search Queue (Forced)',
+    'SEARCHQUEUE-RETRY': 'Search Queue (Retry/Failed)',
+    'SEARCHQUEUE-RSS': 'Search Queue (RSS)',
+    'SHOWQUEUE-FORCE-UPDATE': 'Show Queue (Forced Update)',
+    'SHOWQUEUE-UPDATE': 'Show Queue (Update)',
+    'SHOWQUEUE-REFRESH': 'Show Queue (Refresh)',
+    'SHOWQUEUE-FORCE-REFRESH': 'Show Queue (Forced Refresh)',
+    'FINDPROPERS': 'Find Propers',
+    'POSTPROCESSOR': 'PostProcessor',
+    'FINDSUBTITLES': 'Find Subtitles',
+    'TRAKTCHECKER': 'Trakt Checker',
+    'EVENT': 'Event',
+    'ERROR': 'Error',
+    'TORNADO': 'Tornado',
+    'Thread': 'Thread',
+    'MAIN': 'Main',
+}
 
 
 @route('/errorlogs(/?.*)')
 class ErrorLogs(WebRoot):
+    """Route to errorlogs web page."""
+
     def __init__(self, *args, **kwargs):
+        """Default constructor."""
         super(ErrorLogs, self).__init__(*args, **kwargs)
 
-    def ErrorLogsMenu(self, level):
-        menu = [
-            {'title': 'Clear Errors',
-             'path': 'errorlogs/clearerrors/',
-             'requires': self.haveErrors() and level == logger.ERROR,
-             'icon': 'ui-icon ui-icon-trash'},
-            {'title': 'Clear Warnings',
-             'path': 'errorlogs/clearerrors/?level={level}'.format(level=logger.WARNING),
-             'requires': self.haveWarnings() and level == logger.WARNING,
-             'icon': 'ui-icon ui-icon-trash'},
-            {'title': 'Submit Errors',
-             'path': 'errorlogs/submit_errors/',
-             'requires': self.haveErrors() and level == logger.ERROR,
-             'class': 'submiterrors',
-             'confirm': True,
-             'icon': 'ui-icon ui-icon-arrowreturnthick-1-n'},
+    def _create_menu(self, level):
+        return [
+            {  # Clear Errors
+                'title': 'Clear Errors',
+                'path': 'errorlogs/clearerrors/',
+                'requires': self._has_errors() and level == logger.ERROR,
+                'icon': 'ui-icon ui-icon-trash'
+            },
+            {  # Clear Warnings
+                'title': 'Clear Warnings',
+                'path': 'errorlogs/clearerrors/?level={level}'.format(level=logger.WARNING),
+                'requires': self._has_warnings() and level == logger.WARNING,
+                'icon': 'ui-icon ui-icon-trash'
+            },
+            {  # Submit Errors
+                'title': 'Submit Errors',
+                'path': 'errorlogs/submit_errors/',
+                'requires': self._has_errors() and level == logger.ERROR,
+                'class': 'submiterrors',
+                'confirm': True,
+                'icon': 'ui-icon ui-icon-arrowreturnthick-1-n'
+            },
         ]
-        return menu
 
-    def index(self, level=logger.ERROR):
+    def index(self, level=logger.ERROR, **kwargs):
+        """Default index page."""
         try:
             level = int(level)
-        except Exception:
+        except (TypeError, ValueError):
             level = logger.ERROR
 
         t = PageTemplate(rh=self, filename='errorlogs.mako')
-        return t.render(header='Logs &amp; Errors', title='Logs &amp; Errors',
-                        topmenu='system', submenu=self.ErrorLogsMenu(level),
-                        logLevel=level, controller='errorlogs', action='index')
+        return t.render(header='Logs &amp; Errors', title='Logs &amp; Errors', topmenu='system',
+                        submenu=self._create_menu(level), logLevel=level, controller='errorlogs', action='index')
 
     @staticmethod
-    def haveErrors():
-        if classes.ErrorViewer.errors:
-            return True
+    def _has_errors():
+        return bool(classes.ErrorViewer.errors)
 
     @staticmethod
-    def haveWarnings():
-        if classes.WarningViewer.errors:
-            return True
+    def _has_warnings():
+        return bool(classes.WarningViewer.errors)
 
     def clearerrors(self, level=logger.ERROR):
+        """Clear the errors or warnings."""
         if int(level) == logger.WARNING:
             classes.WarningViewer.clear()
         else:
@@ -67,106 +113,65 @@ def clearerrors(self, level=logger.ERROR):
 
         return self.redirect('/errorlogs/viewlog/')
 
-    def viewlog(self, minLevel=logger.INFO, logFilter='<NONE>', logSearch=None, maxLines=1000):
-        min_level = minLevel
-        log_filter = logFilter
-
-        def Get_Data(Levelmin, data_in, lines_in, regex, Filter, Search, mlines):
-
-            last_line = False
-            num_lines = lines_in
-            num_to_show = min(maxLines, num_lines + len(data_in))
-
-            final_data = []
-
-            for x in reversed(data_in):
-                match = re.match(regex, x)
-
-                if match:
-                    level = match.group(7)
-                    log_name = match.group(8)
-                    if level not in logger.LOGGING_LEVELS:
-                        last_line = False
-                        continue
-
-                    if logSearch and logSearch.lower() in x.lower():
-                        last_line = True
-                        final_data.append(x)
-                        num_lines += 1
-                    elif not logSearch and logger.LOGGING_LEVELS[level] >= min_level and (log_filter == '<NONE>' or log_name.startswith(log_filter)):
-                        last_line = True
-                        final_data.append(x)
-                        num_lines += 1
-                    else:
-                        last_line = False
-                        continue
-
-                elif last_line:
-                    final_data.append('AA' + x)
+    @staticmethod
+    def _get_data(lines, min_level, log_filter, log_search, num_lines, max_lines):
+        last_line = False
+        num_to_show = min(max_lines, num_lines + len(lines))
+
+        final_data = []
+        for line in reversed(lines):
+            match = log_re.match(line)
+            if match:
+                level = match.group('log_level')
+                log_name = match.group('log_name')
+                if level not in logger.LOGGING_LEVELS:
+                    last_line = False
+                    continue
+
+                if log_search and log_search.lower() in line.lower():
+                    last_line = True
+                    final_data.append(line)
                     num_lines += 1
 
-                if num_lines >= num_to_show:
-                    return final_data
+                elif not log_search and logger.LOGGING_LEVELS[level] >= min_level and (log_filter == '<NONE>' or log_name.startswith(log_filter)):
+                    last_line = True
+                    final_data.append(line)
+                    num_lines += 1
+                else:
+                    last_line = False
+                    continue
 
-            return final_data
+            elif last_line:
+                final_data.append('AA' + line)
+                num_lines += 1
 
-        t = PageTemplate(rh=self, filename='viewlogs.mako')
+            if num_lines >= num_to_show:
+                return final_data
 
-        min_level = int(min_level)
-
-        log_name_filters = {
-            '<NONE>': '&lt;No Filter&gt;',
-            'DAILYSEARCHER': 'Daily Searcher',
-            'BACKLOG': 'Backlog',
-            'SHOWUPDATER': 'Show Updater',
-            'CHECKVERSION': 'Check Version',
-            'SHOWQUEUE': 'Show Queue',
-            'SEARCHQUEUE': 'Search Queue (All)',
-            'SEARCHQUEUE-DAILY-SEARCH': 'Search Queue (Daily Searcher)',
-            'SEARCHQUEUE-BACKLOG': 'Search Queue (Backlog)',
-            'SEARCHQUEUE-MANUAL': 'Search Queue (Manual)',
-            'SEARCHQUEUE-FORCED': 'Search Queue (Forced)',
-            'SEARCHQUEUE-RETRY': 'Search Queue (Retry/Failed)',
-            'SEARCHQUEUE-RSS': 'Search Queue (RSS)',
-            'SHOWQUEUE-FORCE-UPDATE': 'Show Queue (Forced Update)',
-            'SHOWQUEUE-UPDATE': 'Show Queue (Update)',
-            'SHOWQUEUE-REFRESH': 'Show Queue (Refresh)',
-            'SHOWQUEUE-FORCE-REFRESH': 'Show Queue (Forced Refresh)',
-            'FINDPROPERS': 'Find Propers',
-            'POSTPROCESSOR': 'PostProcessor',
-            'FINDSUBTITLES': 'Find Subtitles',
-            'TRAKTCHECKER': 'Trakt Checker',
-            'EVENT': 'Event',
-            'ERROR': 'Error',
-            'TORNADO': 'Tornado',
-            'Thread': 'Thread',
-            'MAIN': 'Main',
-        }
-
-        if log_filter not in log_name_filters:
-            log_filter = '<NONE>'
-
-        regex = r'^(\d\d\d\d)\-(\d\d)\-(\d\d)\s*(\d\d)\:(\d\d):(\d\d)\s*([A-Z]+)\s*(.+?)\s*\:\:\s*(.*)$'
+        return final_data
 
-        data = []
+    def viewlog(self, minLevel=logger.INFO, logFilter='<NONE>', logSearch=None, maxLines=1000, **kwargs):
+        """View the log given the specified filters."""
+        min_level = int(minLevel)
+        log_filter = logFilter if logFilter in log_name_filters else '<NONE>'
+        log_search = logSearch
+        max_lines = maxLines
 
-        if ek(os.path.isfile, logger.log_file):
-            with io.open(logger.log_file, 'r', encoding='utf-8') as f:
-                data = Get_Data(min_level, f.readlines(), 0, regex, log_filter, logSearch, maxLines)
+        t = PageTemplate(rh=self, filename='viewlogs.mako')
 
-        for i in range(1, int(sickbeard.LOG_NR)):
-            log_file = '{file}.{number}'.format(file=logger.log_file, number=i)
-            if ek(os.path.isfile, log_file) and (len(data) <= maxLines):
+        data = []
+        log_files = [logger.log_file] + ['{file}.{number}'.format(file=logger.log_file, number=i) for i in range(1, int(sickbeard.LOG_NR))]
+        for log_file in log_files:
+            if len(data) <= max_lines and ek(os.path.isfile, log_file):
                 with io.open(log_file, 'r', encoding='utf-8') as f:
-                    data += Get_Data(min_level, f.readlines(), len(data), regex, log_filter, logSearch, maxLines)
+                    data += ErrorLogs._get_data(f.readlines(), min_level, log_filter, log_search, len(data), max_lines)
 
-        return t.render(
-            header='Log File', title='Logs', topmenu='system',
-            logLines=''.join(data), minLevel=min_level, logNameFilters=log_name_filters,
-            logFilter=log_filter, logSearch=logSearch,
-            controller='errorlogs', action='viewlogs')
+        return t.render(header='Log File', title='Logs', topmenu='system', logLines=''.join([html_escape(line) for line in data]),
+                        minLevel=min_level, logNameFilters=log_name_filters, logFilter=log_filter, logSearch=log_search,
+                        controller='errorlogs', action='viewlogs')
 
     def submit_errors(self):
+        """Create an issue in medusa issue tracker."""
         submitter_result, issue_id = logger.submit_errors()
         logger.log(submitter_result, (logger.INFO, logger.WARNING)[issue_id is None])
         submitter_notification = ui.notifications.error if issue_id is None else ui.notifications.message


Test patch:
diff --git a/tests/test_logger.py b/tests/test_logger.py
--- a/tests/test_logger.py
+++ b/tests/test_logger.py
@@ -32,44 +32,35 @@ def sut(stream_handler):
     return sut
 
 
-def test_logger__using_brackets(sut, handler_write, handler_error):
+@pytest.mark.parametrize('p', [
+    {  # p0: curly brackets style
+        'message': 'This is an example: {arg1} {arg2}',
+        'args': [],
+        'kwargs': dict(arg1='hello', arg2='world'),
+        'expected': 'This is an example: hello world'
+    },
+    {  # p1: legacy formatter
+        'message': 'This is an example: %s %s',
+        'args': ['hello', 'world'],
+        'kwargs': dict(),
+        'expected': 'This is an example: hello world'
+    },
+    {  # p2: regression test: https://github.com/pymedusa/SickRage/issues/876
+        'message': "{'type': 'episode', 'season': 5}",
+        'args': [],
+        'kwargs': dict(),
+        'expected': "{'type': 'episode', 'season': 5}"
+    },
+])
+def test_logger__various_messages(sut, handler_write, handler_error, p):
     # Given
-    message = '{arg1} {arg2}'
-    args = {'arg1': 'hello', 'arg2': 'world'}
-    expected = (('hello world\n', ), {})
+    message = p['message']
+    args = p['args']
+    kwargs = p['kwargs']
+    expected = (('%s\n' % p['expected'], ), {})
 
     # When
-    sut.error(message, **args)
-
-    # Then
-    assert not handler_error.called
-    assert handler_write.called
-    assert handler_write.call_args == expected
-
-
-def test_logger__legacy_formatter(sut, handler_write, handler_error):
-    # Given
-    message = '%s %s'
-    args = ['hello', 'world']
-    expected = (('hello world\n', ), {})
-
-    # When
-    sut.error(message, *args)
-
-    # Then
-    assert not handler_error.called
-    assert handler_write.called
-    assert handler_write.call_args == expected
-
-
-def test_logger__regression_876(sut, handler_write, handler_error):
-    """Regression for https://github.com/pymedusa/SickRage/issues/876 issue."""
-    # Given
-    message = "{'type': 'episode', 'season': 5}"
-    expected = (("{'type': 'episode', 'season': 5}\n", ), {})
-
-    # When
-    sut.error(message)
+    sut.error(message, *args, **kwargs)
 
     # Then
     assert not handler_error.called