Package: os-pywf
os-pywfWorkflowand it's Python bindingPyWorkfloware great async frameworks.This project is trying to explore the power of Workflow, provide command line tools and high level APIs for real world development.Installpip install os-pywfCommandsos-pywfcommand can be used after installation. You can get help information with--helpoption.  Global settings of Workflow can be specified, ENVs is not supported yet.The subcommands withplanningtag will be developed later, can not be used right now.$ os-pywf --help
Usage: os-pywf [OPTIONS] COMMAND [ARGS]...

  Command line tool for os-pywf.

Options:
  --version                       Show the version and exit.
  Workflow:                       Workflow global settings.
    --compute-threads INTEGER     Number of compute threads.  [default: 4]
    --handler-threads INTEGER     Number of handler threads.  [default: 4]
    --poller-threads INTEGER      Number of poller threads.  [default: 4]
    --dns-threads INTEGER         Number of dns threads.  [default: 4]
    --dns-ttl-default INTEGER     Default seconds of dns ttl.  [default:
                                  43200]

    --dns-ttl-min INTEGER         Min seconds of dns ttl.  [default: 180]
    --max-connections INTEGER     Max number of connections.  [default: 200]
    --connection-timeout INTEGER  Connect timeout(ms).  [default: 10000]
    --response-timeout INTEGER    Response timeout(ms).  [default: 10000]
    --ssl-connect-timeout INTEGER
                                  SSL connect timeout(ms).  [default: 10000]
  --help                          Show this message and exit.

Commands:
  curl    HTTP client inspired by curl (beta).
  mysql   MySQL client (planning).
  proxy   HTTP proxy (planning).
  redis   Redis client (planning).
  run     Run runnable objects of pywf (planning).
  spider  Web spider (planning).
  web     Web server (planning).curlThis subcommand is inspired by curl. It works as curl and provides more useful features especially invoke Python function as response callback, which make it flexible and easy to extend.$ os-pywf curl --help
Usage: os-pywf curl [OPTIONS] [URLS]...

  HTTP client inspired by curl (beta).

Options:
  Curl options:                   Options same as curl.
    -0, --http1.0                 Use HTTP 1.0
    -A, --user-agent TEXT         User-Agent to send to server.  [default: os-
                                  pywf/0.0.1]

    -b, --cookie TEXT             String or file to read cookies from.
    -c, --cookie-jar FILENAME     Write cookies to this file after operation.
    -d, --data TEXT               HTTP POST data.
    --data-urlencode TEXT         HTTP POST data url encoded.
    -e, --referer TEXT            Referer URL.
    -F, --form TEXT               Specify HTTP multipart POST data.
    -H, --header TEXT             Custom header to pass to server.
    -L, --location                Follow redirects.
    --max-filesize INTEGER        Maximum data size (in bytes) to download.
    --max-redirs INTEGER          Maximum number of redirects allowed.
                                  [default: 30]

    -u, --user TEXT               Specify the user name and password to use
                                  for  server  authentication.

    --no-keepalive                Disable keepalive.
    --retry INTEGER               Maximum retries when request fail.
                                  [default: 0]

    --retry-delay FLOAT           Time between two retries(s).  [default: 0]
    -x, --proxy TEXT              Specify proxy.
    -X, --request [DELETE|GET|HEAD|OPTIONS|PATCH|POST|PUT]
                                  Request method. [default: GET]
  Additional options:             Additional options.
    --send-timeout FLOAT          Send request timeout(s).  [default: -1]
    --receive-timeout FLOAT       Receive response timeout(s).  [default: -1]
    --startup TEXT                Function invoked when startup.  [default:
                                  os_pywf.commands.curl.startup]

    --cleanup TEXT                Function invoked when cleanup.  [default:
                                  os_pywf.commands.curl.cleanup]

    --callback TEXT               Function invoked when response received.
                                  [default: os_pywf.commands.curl.callback]

    --errback TEXT                Function invoked when request fail (callback
                                  will be invoked when no errback).

    --parallel                    Send requests parallelly.
    --log-level [CRITICAL|ERROR|WARNING|INFO|DEBUG]
                                  Log level.  [default: INFO]
    --debug                       Enable debug mode.
  --help                          Show this message and exit.Example:# app.py
def callback(task, request, response):
    print(request, response)os-pywf curl http://www.example.com/ --callback app.callbackFeatures:Same options as curl, command line can be used by curl directlySupport HTTP version 1.0/1.1Auto manipulate cookies. Cookies can be specified by command line or read from file. Cookies can be saved to fileSupport post urlencode dataSupport upload files as multipart formSupport redirect. Response history can be accessed withresponse.historySupport retry and retry interval.  The program can be quickly canceled when retryingAll requests can be send parallelly (async not multithread)Custom startup/cleanup/callback/errback function as pluginsCallback with request and response parameters of the most famousRequestslibrarySupport auto decompress response data (v0.0.2)Support set proxy for http (not https) request (v0.0.3)Generate requests from callback and download continuously (v0.0.4)Issues/Not support:Configure proxyUse your own certCtrl+C to quit program slowly when downloading slow responseThe command provides two types of options,curl optionsandadditional options. Runos-pywf curl --helpto get the full help information.curl optionsare same as the options of curl.  Usage can be found on man page of curl and help descriptions.additional optionsenhance curl and provide additional features.--send-timeout, send request timeout (second), default (-1) is no timeout--receive-timeout, receive response timeout (second), default (-1) behavior depends on some other settings such as response timeout--startup, a function invoked when startup, before download pages. The function have only one parameter which is the series or the parallel of Workflow--cleanup, a function invoked when cleanup, after all downloads finish. The function have only one parameter same as startup function# app.py
def startup(runner):
    pass

def cleanup(runner):
    passos-pywf curl http://www.example.com/ --startup app.startup --cleanup app.cleanup--callback,--errbackfunctions invoked when response received or fail, seemore details--parallel,  requests will be send parallelly. Attention, the framework is asynchronous, all callback/errback invoked in one thread. Block operations in callback/errback will block the whole worldAPIsos_pywf.http.clientThis module provides hight level HTTP client APIs. Inspired by the most famous Python HTTP libraryRequests, the APIs are nearly the same.All of the request APIs do not send request and block wait response, they all return HttpTask object for Workflow and invoke callback function when response downloaded.We wrap the PyWorkflow HttpTask and provide more convenient callback withrequestandresponseas additional parameters, they all typical instance of Requests library as you know.import pywf
from os_pywf.http import client

def callback(task, request, response):
    print(request, response)

task = client.get("http://www.example.com/", callback=callback)
task.start()
pywf.wait_finish()We provide more useful features which PyWorkflow not support directly:session with cookies persistenceredirect responses historyretry interval and quick cancelauthenticationpost urlencode data and multipart files uploadauto decompress response data (v0.0.2)set proxy for http (not https) request (v0.0.3)You can use Session to configure same settings of  a group tasks, it also auto manipulate cookies and provide cancel function to cancel all tasks create by the same session. You can create Session as normal class or as a context manager:import pywf
from os_pywf.http import client
from os_pywf.utils import create_series_work

def callback(task, request, response):
    print(request, response)
    
series = create_series_work()
headers = {"User-Agent": "os-pywf/beta"}
with client.Session(headers=headers, callback=callback) as session:
    for url in ["http://www.example.com/", "http://httpbin.org/"]:
        task = session.get(url)
        series.push_back(task)
series.start()
pywf.wait_finish()Session can be canceled, when canceled the tasks created by the session which not started  will be destroyed, running task will still run until finish but callback will not invoked....
# register cancel for Ctrl+C 
with client.Session() as session:
    def _cancel(signum, frame):    
        session.cancel()
    for sig in (signal.SIGTERM, signal.SIGINT):
        signal.signal(sig, _cancel)
...callback/errbackFor callback async type of Workflow, we provide two functions as request/session parameters for framework: callback and errbackWe wrap PyWorkflow with most famous Python HTTP libraryRequestsand provide more powerful callback and errbackcallback, invoked when response received,  three parameters: task, request, responsedef callback(task, request, response):
    passtask, the PyWorkflow HttpTask objectrequest,requests.PreparedRequestobject, it is the original request even though there are retries and redirectsresponse,requests.Responseobject, it is the final response when there are retries and redirects.  You can get all the response when redirect occur. If not set errback function, the response will beos_pywf.exceptions.Failureobject when transaction fail (all HTTP response treat as success)errback, invoked when transaction fail. It can be ignored, all of the response and fail will invoke callback function, three parameters: task, request, failuredef errback(task, request, failure):
    passtask, the PyWorkflow HttpTask objectrequest, same as the parameter of callbackFailure,os_pywf.exceptions.Failureobject, it has two properties: exception and value. The value property maybe None orrequests.Responsedepends on the fail situationboth callback and errback can have return value (from v0.0.4) for framework to schedule. There are several types object can be returnedstr，must be URL，it will be wrapped with session as HttpTask and add to the head of the seriesrequests.Request, it will be wrapped with session as HttpTask and add to the head of the seriesrequests.PreparedRequest, it will be wrappedwithoutsession as HttpTask and add to the head of the seriespywf.SubTask, it will be add to the head of the serieslist, the elements will be treated as above object add to the head of the series from last to firsttuple, first element treated as above object, second element will add to the tail of the seriesos_pywf.utilscreate_series_work, wrap the create_series_work of PyWorkflow, you can pass arbitrary tasks to create series.create_timer_task, wrap the create_timer_task of PyWorkflow. It split the wait time into small time pieces, so it can be canceled as soon as possible.You can pass a threading.Event object as cancel parameter.os_pywf.exceptionsFailure, failure for usually for errback, two properties: exception and value. The real value object depend on fail situationWFException, exception about task fail,  two properties: state and code. state come fromtask.get_state(), code come fromtask.get_error(). You can get human readable error string by  use built-in str function.Unit Testssh scripts/test.shLicenseMIT licensed.