Package: feed-to-exporter
Feed to Wordpress PostThis project allow to get RSS Feed and create new Wordpress post from them.Pre-requisitesWordpress ModeYou must install the Wordpress plugin:Application PasswordsYou must create a new Application PasswordPython 3.6 or abovemapping.json formatWordpress ModeThe mapping file has this format:{"feed":"http://www.xxxx.es/blogs/xxxx/feed/","exportMethod":"wordpress","mapping":{"body":"summary"},"fixed":{"tags":["one","two","general"],"categories":[{"category":"subcategory","parent":"top-category"}],"post_status":"publish"}}Mongo Mode{"feed":"http://www.xxxx.es/blogs/xxxx/feed/","exportMethod":"mongo","fixed":{"categories":[{"category":"subcategory","parent":"top-category"}]}}Mapping keymappingindicates howf2emust match the input feed values to the wordpress result.Left values of mapping will be the variables names of exported objects.Right values are the key names in feed wheref2emust map to the output.For example:Suppose this RSS:<?xml version="1.0" encoding="ISO-8859-1"?><rssversion="2.0"><channel><title>Trabajoyempleo-BOE-BoletínOficialdelEstado</title><link>http://www.boe.es/diario_boe/</link><description>LegislaciónrelativaaTrabajoyempleoingresadaenlosúltimosdosmesesenlabasededatosdelBoletínOficialdelEstado</description><language>es-es</language><pubDate>Mon,06Aug201800:00:00+0200</pubDate><lastBuildDate>Mon,06Aug201814:55:03+0200</lastBuildDate><webMaster>webmaster@boe.es</webMaster><item><title>Pleno.Sentencia78/2018,de5dejuliode2018.Recursodeinconstitucionalidad3720-2017.InterpuestoporelPresidentedelGobiernoenrelaciónconlosartículos13y36delaLey10/2016,de27dediciembre,delpresupuestodelaComunidadAutónomadeAndalucíaparaelaño2017.Competenciassobreordenacióngeneraldelaeconomía,haciendageneralyfunciónpública:nulidadparcialdelpreceptolegalautonómicorelativoalaofertadeempleopúblicode2017uotroinstrumentosimilardegestióndelaprovisióndenecesidadesdepersonal(STC142/2017).Votoparticular.</title><link>http://www.boe.es/diario_boe/txt.php?id=BOE-A-2018-11276</link><description>TribunalConstitucional-Publicadoel06/08/2018-Referencia:BOE-A-2018-11276</description><guidisPermaLink="true">http://www.boe.es/boe/dias/2018/08/06/pdfs/BOE-A-2018-11276.pdf</guid><pubDate>Mon,06Aug201800:00:00+0200</pubDate></item></channel></rss>Where each item has this format:<item><title>Pleno.Sentencia78/2018,de5dejuliode2018.Recursodeinconstitucionalidad3720-2017.InterpuestoporelPresidentedelGobiernoenrelaciónconlosartículos13y36delaLey10/2016,de27dediciembre,delpresupuestodelaComunidadAutónomadeAndalucíaparaelaño2017.Competenciassobreordenacióngeneraldelaeconomía,haciendageneralyfunciónpública:nulidadparcialdelpreceptolegalautonómicorelativoalaofertadeempleopúblicode2017uotroinstrumentosimilardegestióndelaprovisióndenecesidadesdepersonal(STC142/2017).Votoparticular.</title><link>http://www.boe.es/diario_boe/txt.php?id=BOE-A-2018-11276</link><description>TribunalConstitucional-Publicadoel06/08/2018-Referencia:BOE-A-2018-11276</description><guidisPermaLink="true">http://www.boe.es/boe/dias/2018/08/06/pdfs/BOE-A-2018-11276.pdf</guid><pubDate>Mon,06Aug201800:00:00+0200</pubDate></item>This implies that we’ll have these keys:titlelinkdescriptionpubDateguidThen, if we want to recover the title, description and published date, we must write thismapping.json:{"mapping":{"body":"description"}}Fixed keysSome times it should be interesting to add some static values to the result dict object. You can use fixed keys for this purpose.Wordpress modeIn Wordpress there’re two special keys:tagsandcategories.You can specifytagsandcategories.f2ewill try to resolve the tag/category or create if it doesn’t exits in the Wordpress site.FiltersThere’s situations where you may want to apply some advanced filters. To do that we must add some Python code.You can use any name for the filter file, but for convention we’ll usefilters.py. There a basic example:Basicsfromfeed_to_exporter.filtersimportFeedInfofromfeed_to_exporter.exceptionsimportFeedToWordpressNotValidInfoFounddeflink_filter(field_value:str)->dict:"""
    this filter will download the link pointing by the field and replace
    the content of the web page.

    Also try to check if some keywords are available and generate some tags

    Content filter must return a dictionary type, otherwise, engine will
    release an exception
    """response=request.get(field_value)results={'body':response.content}# Try to find tagsifany(xinresponse.contentforxin('hacking','security','pentesting')):results['tags']=['security']returnresultsdefbody_filter(field_value:str)->dict:"""
    This filter remove the words 'SEO' from the body field and return
    the new 'body' value for the field. The engine will update that
    with this information.

    Content filter must return a dictionary type, otherwise, engine will
    release an exception
    """return{'body':field_value.replace('SEO','')}defglobal_filter(feed_info:FeedInfo)->\dictorFeedToWordpressNotValidInfoFound:"""
    Global filter enables a validation with the context of all of fields
    values. This filter must return a dictionary or an exception.

    If one exception is returned, engine will interpret that the current
    feed must not be processed and continue to the next feed.

    Global filter will executed after the individual filters.
    """ifnotfeed_info.titleornotfeed_info.body:raiseFeedToWordpressNotValidInfoFound()if"security"infeed_info.titleand"hacking"infeed_info.body:return{"category":["hard-security"]}elif"ciso"infeed_info.body.lower():return{"category":["ciso-news"]}else:return{}## Order of filters are following the definition in the bellow dictionary## The name of the variable must be the following for the individual filtersINDIVIDUAL_VALIDATORS={'link':link_filter,'body':body_filter}# The name of the variable must be the following for global validatorGLOBAL_VALIDATOR=global_filterAs you can see you must define the var nameINDIVIDUAL_VALIDATORSindicates the field where it will apply the filter.Filtersalwaysmust return a dictionary and it can overwrite the original content of a field.Filters execution order are defined by the order indicated in theINDIVIDUAL_VALIDATORSvar.The parameters passed in each individual filter function is the value of the field.Result data structuref2eexport collected data from feed to a dictionary. Depending of the export method you want, it need some different keys:WordpressFeedInfoWordpress has these properties:title: strslug: strlink: strfeed_source: strbody: str -> raw information from Feed mappingcontent: str -> content that will send to the Wordpress Post. By default is a composition of: body + html link + feed_source. You can see at internal filters (feed_to_exporter.filters.py)raw_feed_info: dict -> raw content of feedping_status: str (default: closed)feed_source: str (default: closed)post_status: str (default: draft)comment_status: str (default: closed)date: str (default: now time, with format: %Y-%m-%dT%H:%M:%S)For fieldsping_status,feed_source,post_statusandcomment_statusyou can check valid values at Wordpress REST API doc:https://developer.wordpress.org/rest-api/MongoMongo doesn’t need any special value for the result dictionary. The whole dict will be stored into Mongo “as is”.Validation ruleSome times you could want to use a global validation rule. This validation could implies more than one field. If this is the case then you must use the a new function and map toGLOBAL_VALIDATORvariable.This function must returns adictvalue or a Exception.Working modesSimpleSimple mode is the usual mode. Explained above.Discovery modeDiscover mode discover recursively the directories, form a base dir given. The engine will get each directory and manage it as and independent running.For this mode works well each crawler must in an independent directory and have only 2 files:filters.pyandmapping.json.To enable this mode you must use the-Doption and each m¡mapping.jsonmust have an additional entry:feed:{"feed":"http://www.mysite.com/feed/","mapping":{"body":"summary"},"fixed":{"categories":["myCategory"]}}Example of directory structure>treeexamples/examples├──__init__.py├──site1.com│├──__ini__.py│├──filters.py│└──mapping.json└──other-site.com│├──__ini__.py├──f2eSkip├──filters.py└──mapping.jsonIMPORTANT: all of folders need the file__init__.pycon convert it into a Python package.Ignoring directoryIf you want that a directory will be ignored, only create a file calledf2eSkipinto the directory and the engine will ignore it.Running ExamplesWithout DockerInstall:>pipinstall-Ufeed-to-exporterWordpress modeBasic Usage:>f2ewordpress-Whttps://mywordpress.com-Uuser-A"XXXX XXXX XXXX XXXX XXXX XXXX"examples/Where-Aindicates the Application PasswordFor more help type-h:>f2ewordpress-husage:f2ewordpress[-h]--wordpress-urlWORDPRESS_URL--userUSER--app-authAPP_AUTH[--devel][feed_source[feed_source...]]positionalarguments:feed_sourcetargeturlorpathoptionalarguments:-h,--helpshowthishelpmessageandexit--wordpress-urlWORDPRESS_URL,-WWORDPRESS_URLwordpressurl--userUSER,-UUSERusertoaccesstoWordpress--app-authAPP_AUTH,-AAPP_AUTHappauthcode(from"Application Passwords"plugin)--develrunningindevelopmodedoesn'tpublishWordpressPostMongoDB modeWith default parameters (mongo in localhost without authentication, database=f2e, collection=f2e)>f2emongoexamples/Setting some parameters:>f2emongo-UmongoAdmin-Mmongodb://10.0.0.1:27017examples/For more help type-h:>f2emongo-husage:f2emongo[-h][--userUSER][--passwordPASSWORD][--collectionCOLLECTION][--databaseDATABASE][--mongo-urlMONGO_URN][feed_source[feed_source...]]positionalarguments:feed_sourcetargeturlorpathoptionalarguments:-h,--helpshowthishelpmessageandexit--userUSER,-UUSERmongodbuser--passwordPASSWORD,-PPASSWORDmongodbpassword--collectionCOLLECTION,-CCOLLECTIONmongocollection--databaseDATABASE,-DDATABASEmongodatabase--mongo-urlMONGO_URN,-MMONGO_URNmongoURL.(Default:mongodb://127.0.0.1:27017/f2e)Using DockerDocker only run in discovery mode and can schedule a new run each some seconds.You can mount a dir with the filters/mapping, but it’s highly recommended to put it into a git repository.Environment varsf2e_CMD_PARAMETERS: f2e running optionsf2e_CHECK_TIME: time to launch in secondsf2e_FILTERS_GIT: git where to download filters and mappingRunning examplesRun feed each 3600 seconds:>dockerrun--rm\-ef2e_FILTERS_GIT=https://XXXXXXXXXXXXXX@github.com/cr0hn/myfeeds-repo.git\-ef2e_CMD_PARAMETERS='wordpress -W https://mywordpress.com -U admin -A "XXXX XXXX XXXX XXXX XXXX XXXX"'\-ef2e_CHECK_TIME=3600f2eContributingAny collaboration is welcome!There’re many tasks to do.You can check theIssuesand send us a Pull Request.LicenseThis project is distributed underBSD 3 license