Package: bors
BorsA highly flexible and extensible service integration framework for
scraping the web or consuming APIs.UsageCreate your model based on the data you expect to incorporate.Decide on what you want to do with your data, and add it.Create or use an existing API integration library.Create your root application to tie it all together.Object ModelWe usemarshmallowfor the underlying object schema definitions. Here’s an example model:frommarshmallowimportSchema,fieldsclassNewsItemSchema(Schema):"""News item"""id=f.Str(required=True)url=f.Str(required=True)title=f.Str(required=True)pubDate=f.Str(required=True)timestamp=f.Str(required=True)feed_id=f.Int(required=True)published_date=f.Str(required=True)feed_name=f.Str(required=True)feed_url=f.Str(required=True)feed_enabled=f.Int(required=True)feed_description=f.Str(required=True)url_field=f.Str(required=True)title_field=f.Str(required=True)date_field=f.Str(required=True)feed_image=f.Str(required=True)See themarshmallowdocs for more information.Middleware StrategiesMiddleware API is implemented in the form of strategies and follows this
basic layout:"""
Simple context display strategy
"""frombors.app.strategyimportIStrategyclassPrint(IStrategy):"""Print Strategy implementation"""defbind(self,context):"""
        Bind the strategy to the middleware pipeline,
        returning the context
        """print(f"""PrintStrategy:{context}""")# just a pass-throughreturncontextThe important things to note here: * We’re inheriting fromIStrategy. * We’re implementing abindmethod. * The bind
method receives, potentially augments, and then returns thecontext.API IntegrationRequest SchemaBecause our API is simple, we’re going to use this as-is.frombors.generics.requestimportRequestSchemaResponse SchemaOur API sends us data in the following format:{"data":...,"status":"OK"}For this, we’ll need to supplement a bit, removing the root fields and
returning thedatavalue:frommarshmallowimportfieldsfrombors.generics.requestimportResponseSchemaclassMyAPIResponseSchema(ResponseSchema):"""Schema defining how the API will respond"""status=fields.Str()defget_result(self,data):"""Return the actual result data"""returndata.get("data","")classMeta:"""Add 'data' field"""strict=Trueadditional=("data",)API Classfrombors.api.requestorimportReqclassMyAPI(LoggerMixin):name="my_api"def__init__(self,context):self.create_logger()self.request_schema=RequestSchemaself.result_schema=MyAPIResponseSchemaself.context=contextself.req=Req("http://some.api.endpoint/v1",payload,self.log)# We don't need to deal directly with requests, so we pass them throughself.call=self.req.calldefshutdown(self):"""Perform last-minute stuff"""passHere we use the built-inReqclass to issue requests to the API, we
assign therequest_schemaandresult_schemato classes in our
object, and we set thename,context, andcallattributes.
The results passed through on the API are referencable from within the
middleware context under the keymy_api.Pulling it all togetherfrombors.app.builderimportAppBuilderfrombors.app.strategyimportStrategydefmain():strat=Strategy(Print())app=AppBuilder([MyAPI],strat)app.run()if__name__=="__main__":main()Here, we set as many strategies and API’s as we want, then create and
run theapp.Architecture+------------+
+-+ MIDDLEWARE +------> out
| +------------+
|                       API/WEB
| +------------+
+-+ PREPROCESS +<------ in
  +------------+At its most basic level, aborsintegrator engages with an
integration library (API) passing incoming data through a prepocessor to
generate and validate incoming objects, then passes that data through
middlewares. Outgoing interactions are initiated from within a
middleware and passed directly to an API, allowing easily for
request/response type behavior in addition to observe and react.Ingesting Data^
      |
+-----+------+
| MIDDLEWARE |
+-----+------+
      ^
+-----+------+
| PREPROCESS |
+-----+------+
      ^
      |
      +
     API/
     WEBIngested data provokes calls along the pipeline.Outgoing DataAPI/
     WEB
      ^
      |
+-----+------+
| MIDDLEWARE |
+------------+Enacted events stimulate API or web actions.PreprocessingPreprocessing is nothing more than an object-ization of the incoming
data. This provides two benefits: 1. Data can be generalized across API
interfaces. 2. Data structure can be validated and enforced.MiddlewaresMiddlewares allow for a data processing pipeline to pass data through.+-+  +-+  +-+
  |M|  |M|  |M|
  |I|  |I|  |I|
  |D|  |D|  |D|
  |D|  |D|  |D|
->+L+->+L+->+L+->
  |E|  |E|  |E|
  |W|  |W|  |W|
  |A|  |A|  |A|
  |R|  |R|  |R|
  |E|  |E|  |E|
  +-+  +-+  +-+With this model, we gain a lot of flexibility in the behavior of our
integration. Middleware is up to the developer to create, and can be any
of the following:Data post-processing, filtering, aggregation, or augmentationExternal integrations and interfacesStimulate an API/web transaction from external actors or time-based
criteriaHooks and callbacksHistory0.3.5 (2018-06-26)Added badges.Cleaned up deps.0.3.4 (2018-06-26)Pruned and upgraded all dependencies.0.3.3 (2018-06-26)Cleaned up tox.ini0.3.2 (2018-06-26)Setup example.py script for others to use.0.3.1 (2018-06-26)Added several unit tests, building out some of the framework.0.3.0 (2018-06-25)Reboot packaging using cookiecutter-pypackage.0.2.0 (2018-05-17)Initialize the repository, breaking it out from nombot.