Repository: Flexget/Flexget

Issue:
Add support for `content` field grabbing by rss plugin
Hi there developers,

I have an issue that I cannot seem to resolve. I'm trying to extract links from a relatively detailed rss feed.

My current config looks like this:

```
tasks:
  Music Download:
    inputs:
      - rss:
          url: http://www.trancefix.nl/external.php?type=RSS2&forumids=210
          other_fields: ['content:encoded']  
          link: 'content:encoded'
    exec:
      - echo text="\"{{description}}\"" >> "/volume1/pyload/crawljob/{{title}}.crawljob"
      - echo downloadFolder="\"D:\\Downloads\\Trance\\\"" >> "/volume1/pyload/crawljob/{{title}}.crawljob"
```

However when I try and run this, flexget either doesn't recognise the field name or cannot read the rather long field "content:encoded" or it has a problem with the ":". I'm not sure but the output is the following:

```
2015-11-09 18:44 VERBOSE  task_queue                    There are 1 tasks to execute. Shutdown will commence when they have completed.
2015-11-09 18:44 WARNING  rss           Music Download  Skipped 2 RSS-entries without required information (title, link or enclosures)
2015-11-09 18:44 WARNING  inputs        Music Download  Input rss did not return anything
2015-11-09 18:44 WARNING  details       Music Download  Task didn't produce any entries. This is likely due to a mis-configured or non-functional input.
2015-11-09 18:44 VERBOSE  details       Music Download  Summary - Accepted: 0 (Rejected: 0 Undecided: 0 Failed: 0)
```

Any help would be appreciated!



Discussion:
There are a couple things.
1. you don't want to set the `link` option to that field. You can always manipulate the url field later if need be. That's what's preventing your entries from even being created `Skipped 2 RSS-entries without required information (title, link or enclosures)`
2. Seems feedparser (the library rss plugin is using) does some special stuff with the content field. In your example, that field content can be accessed from python like `feedparseritem['content'][0]['value']`. The rss plugin would need to be modified somehow to support this before it could work.

Hi thanks for your response. 
I initially did have it set up like this:

```
tasks:
  Music Download:
    inputs:
    rss:
      url: http://www.trancefix.nl/external.php?type=RSS2&forumids=210
      other_fields: ['content:encoded']
    manipulate:
      - description:
          from: 'content:encoded'
          extract: \b(([\w-]+://?|www[.])[^\s()<>]+(?:\([\w\d]+\)|([^[:punct:]\s]|/)))
    exec:
      - echo text="\"{{description}}\"" >> "/volume1/pyload/crawljob/{{title}}.crawljob"
      - echo downloadFolder="\"D:\\Downloads\\Trance\\\"" >> "/volume1/pyload/crawljob/{{title}}.crawljob"
```

but yes accessing the content:encoded field never worked. I wonder though why content:encoded is an issue but torrent:magnetURI isn't in terms of accessing it?

> I wonder though why content:encoded is an issue but torrent:magnetURI isn't in terms of accessing it?

Because feedparser is doing special handling on that field to normalize it from different kinds of feeds I think.

Please reopen in [feathub](http://feathub.com/Flexget/Flexget)
I submitted a pull request that fixed the issue for me: #1880

Resolving patch:
diff --git a/flexget/plugins/input/rss.py b/flexget/plugins/input/rss.py
--- a/flexget/plugins/input/rss.py
+++ b/flexget/plugins/input/rss.py
@@ -382,6 +382,16 @@ def add_entry(ea):
                 # fields dict may be modified during this loop, so loop over a copy (fields.items())
                 for rss_field, flexget_field in list(fields.items()):
                     if rss_field in entry:
+                        if rss_field == 'content':
+                            content_str = ''
+                            for content in entry[rss_field]:
+                                try:
+                                    content_str += decode_html(content.value)
+                                except UnicodeDecodeError:
+                                    log.warning('Failed to decode entry `%s` field `%s`', ea['title'], rss_field)
+                            ea[flexget_field] = content_str
+                            log.debug('Field `%s` set to `%s` for `%s`', rss_field, ea[rss_field], ea['title'])
+                            continue
                         if not isinstance(getattr(entry, rss_field), str):
                             # Error if this field is not a string
                             log.error('Cannot grab non text field `%s` from rss.', rss_field)


Test patch:
diff --git a/flexget/tests/rss.xml b/flexget/tests/rss.xml
--- a/flexget/tests/rss.xml
+++ b/flexget/tests/rss.xml
@@ -1,5 +1,6 @@
 <?xml version="1.0" encoding="utf-8"?>
-<rss version="2.0" xmlns:other="http://localhost/flexget">
+<rss version="2.0" xmlns:other="http://localhost/flexget"
+xmlns:content="http://purl.org/rss/1.0/modules/content/">
   <channel>
     <title>FlexGet RSS test</title>
     <ttl>15</ttl>
@@ -65,6 +66,23 @@
       <other:link>http://localhost/altlink</other:link>
     </item>
 
+    <item>
+      <title>Content</title>
+      <link>http://localhost/content</link>
+      <pubDate>Sun, 28 Dec 2008 14:20:00 -0200</pubDate>
+      <description>Description, content</description>
+      <content:encoded><![CDATA[<p>test content:encoded</p>]]></content:encoded>
+    </item>
+
+    <item>
+      <title>Multiple content items</title>
+      <link>http://localhost/multi_content</link>
+      <pubDate>Sun, 28 Dec 2008 14:20:00 -0200</pubDate>
+      <description>Description, multi_content</description>
+      <content:encoded><![CDATA[<p>test content1</p>]]></content:encoded>
+      <content:encoded><![CDATA[<p>test content2</p>]]></content:encoded>
+    </item>
+
     <item>
       <title></title>
       <link>http://localhost/empty</link>
diff --git a/flexget/tests/test_rss.py b/flexget/tests/test_rss.py
--- a/flexget/tests/test_rss.py
+++ b/flexget/tests/test_rss.py
@@ -47,6 +47,11 @@ class TestInputRSS(object):
               title: "other:Title"
               other_fields:
               - "Other:field"
+          test_content:
+            rss:
+              <<: *rss
+              other_fields:
+                - content
     """
 
     def test_rss(self, execute_task):
@@ -145,6 +150,13 @@ def test_field_sanitation(self, execute_task):
         assert entry['url'] == 'http://localhost/altlink'
         assert entry['other:field'] == 'otherfield'
 
+    def test_content(self, execute_task):
+        task = execute_task('test_content')
+        assert task.find_entry(title='Content', content='<p>test content:encoded</p>'), \
+            'RSS entry missing: content:encoded'
+        assert task.find_entry(title='Multiple content items', content='<p>test content1</p><p>test content2</p>'), \
+            'RSS entry missing: multiple content tags'
+
 
 @pytest.mark.xfail(reason="silverorange changed some stuff")
 @pytest.mark.online