Package: nipyapi
NiPyApiNifi-Python-Api: A rich Apache NiFi Python Client SDKFeaturesThree layers of Python support for working with Apache NiFi:High-level Demos and example scriptsMid-level Client SDK for typical complex tasksLow-level Client SDKs for the full API implementation of NiFi and selected sub-projectsFunctionality Highlights:Detailed documentation of the full SDK at all levelsCRUD wrappers for common task areas like Processor Groups, Processors, Templates, Registry Clients, Registry Buckets, Registry Flows, etc.Convenience functions for inventory tasks, such as recursively retrieving the entire canvas, or a flat list of all Process GroupsSupport for scheduling and purging flows, controller services, and connectionsSupport for fetching and updating Variable RegistriesSupport for import/export of Versioned Flows from NiFi-RegistryDocker Compose configurations for testing and deploymentA scripted deployment of an interactive environment, and a secured configuration, for testing and demonstration purposesPlease see theissueregister for more information on current development.Quick StartThere are several scripts to produce demo environments innipyapi.demo.*The mid-level functionality is innipyapi.canvas / nipyapi.security / nipyapi.templates / nipyapi.versioningYou can access the entire API using the low-level SDKs innipyapi.nifi / nipyapi.registryThe easiest way to install NiPyApi is with pip:# in bash
pip install nipyapiYou can set the config for your endpoints in the central config file:# in python
import nipyapi
nipyapi.config.nifi_config.host = 'http://localhost:8080/nifi-api'
nipyapi.config.registry_config.host = 'http://localhost:18080/nifi-registry-api'Then import a module and execute tasks:nipyapi.canvas.get_root_pg_id()
>'4d5dcf9a-015e-1000-097e-e505ed0f7fd2'You can use the Docker demos to create a secured interactive console showing many features:from nipyapi.demo.secured_console import *
from nipyapi.demo.console import *You can also explore the scripts to get ideas for how NiPyAPi can be used to automate your environment.Please check out theContribution Guideif you are interested in contributing to the feature set.Background and DocumentationFor more information on Apache NiFi, please visithttps://nifi.apache.orgFor Documentation on this package please visithttps://nipyapi.readthedocs.io.NiFi Version SupportCurrently we are testing against NiFi versions 1.1.2 - 1.15.0, and NiFi-Registry versions 0.1.0 - 1.15.0.If you find a version compatibility problem please raise anissuePython RequirementsPython 2.7 or 3.6-9 supported, though other versions may work.We will shortly stop supporting Python2There are known issues on Python 3.10Tested on AL2 and OSX 10.14.x - Windows automated testing not attemptedOutside of the standard Python modules, we make use of lxml, DeepDiff, ruamel.yaml and xmltodict in processing, and Docker for demo/tests.History0.19.1 (2022-08-08)Updated client for NiFi & Registry 1.17.0 releaseUpdate docker resources to the latest 1.17 container imageFix issue in docker volume mounts where certificates were not readable in new versions of DockerFix issue in test_get_processor_type where new Twitter processor broke existing logic assumptionsUpdate readme to note issues with Python 3.10 and removing reference to python 3.5 testing0.19.0 (2022-05-13)Updated client for NiFi & Registry 1.16.0 releaseModify utils.check_version to accept a default version to check against, and remove nested error when version check fails as it is overzealousUpdate test file paths in test_utils in case someone runs tests as root which actually can write to fake devices and therefore break the testsAdded pylint exclusions for known but unimportant complaintsAdded workaround for NiFi 9470 by providing default inherited_parameter_contexts per advice from Chris Sampson in NiPyAPI Issue #305Updated NiFi and Registry versions in Docker compose configs to use latest 1.16.1 builds0.18.0 (2021-11-29)Updated client for NiFi & Registry 1.15.0 releaseMoved testing to secured single user by default for NiFiConsolidated clients to the 1.15.0 release version for both NiFi and RegistryUpdated logic to support changed working modes in testsCorrected minor functional issues for 1.15 in login handling, SSL certs, Parameter Updates and TestingBackwards compatibility and Regression tested as requiring no breaking changes0.17.1 (2021-10-21)Added gzip as default request header (PR from rsaggino)Update ruamel.yaml to 0.17.16Update file read/write to explicitly handle encoding, default to UTF8, configurable in nipyapi.configLinting and style changes to update to Python 3.9 standards, except where it would break backwards compatibility to Python 2.7Test support for Amazon Linux 2, mostly better error handling for filesystem responses0.17.0 (2021-10-13)Updated NiFi version and client to 1.13.2Updated NiFi-Registry version and client to 0.8.0Note that these are the last versions where NiFi and NiFi-Registry are separate codebases0.16.3 (2021-10-11)Removed force reset of configuration.password and configuration.username to empty string. This was not increasing security, and was causing unexpected errors for users connecting to multiple services in a single script.Add greedy control to versioning.get_registry_bucket and versioning.get_flow_in_bucket to avoid undesirable partial string match.Update readme to reflect switch from ‘master’ branch naming to ‘main’.Update tox to pin testing to Python 3.8, as Python 3.9 is producing unexpected and unrelated SSL failuresMinor lint formatting improvements0.16.2 (2021-02-10)NOTE: If you are using secured Registry, this release will enforce access controls for the swagger interface which is used to determine which version of Registry is connected in order to correctly provide features - you may have to update your authorizationsUpdate requirements.txt to unpin future and lxmlUpdate lxml to 4.6.2 or newer to resolve vulnerabilityPin watchdog to <1.0.0 per their docs to maintain Python2.7 compatibilityRevert 0.14.3 changes to Authentication handling which introduced basicAuth support but resulted in some NiFi connections appearing incorrectly as AnonymousAdded simpler basicAuth control to force it via a config switch without changing tokenAuth and other Authorization header behavior during normal usagenipyapi.config.global_force_basic_auth is now available for use for this purposeSecured Registry users will now require the authorization policy to retrieve the swagger so we may use it to validate which version ofRegistry is in use for feature enablementMoved all Security controls in config.py to a common area at the foot of the fileRemoved auth_type from security.service_login as it is now redundantAdded controls to handle certificate checking behavior which has become more strict in recently versions of Python3, ssl_verify and check_hostname are now handledsecurity.set_service_auth_token now has an explicit flag for ssl host checking as wellFix oversight where improved model serialisation logic was not correctly applied to RegistryRemoved unusused parameter refresh from parameters.update_parameter_contextReduced unecessary complexity in utils.dump with no change in functionalityUpdated client gen mustache templates to reflect refactored security and api client codeMinor linting and docstring and codestyle improvementsSet pyUp to ignore Watchdog as it must stay between versions to statisfy py2 and py3 compatibilityIf Client is not instantiated, optimistically instantiate for version checkingadd socks proxy support0.16.3 (2021-10-11)Removed force reset of configuration.password and configuration.username to empty string. This was not increasing security, and was causing unexpected errors for users connecting to multiple services in a single script.Add greedy control to versioning.get_registry_bucket and versioning.get_flow_in_bucket to avoid undesirable partial string match.Update readme to reflect switch from ‘master’ branch naming to ‘main’.Update tox to pin testing to Python 3.8, as Python 3.9 is producing unexpected and unrelated SSL failuresMinor lint formatting improvements0.16.2 (2021-02-10)NOTE: If you are using secured Registry, this release will enforce access controls for the swagger interface which is used to determine which version of Registry is connected in order to correctly provide features - you may have to update your authorizationsUpdate requirements.txt to unpin future and lxmlUpdate lxml to 4.6.2 or newer to resolve vulnerabilityPin watchdog to <1.0.0 per their docs to maintain Python2.7 compatibilityRevert 0.14.3 changes to Authentication handling which introduced basicAuth support but resulted in some NiFi connections appearing incorrectly as AnonymousAdded simpler basicAuth control to force it via a config switch without changing tokenAuth and other Authorization header behavior during normal usagenipyapi.config.global_force_basic_auth is now available for use for this purposeSecured Registry users will now require the authorization policy to retrieve the swagger so we may use it to validate which version ofRegistry is in use for feature enablementMoved all Security controls in config.py to a common area at the foot of the fileRemoved auth_type from security.service_login as it is now redundantAdded controls to handle certificate checking behavior which has become more strict in recently versions of Python3, ssl_verify and check_hostname are now handledsecurity.set_service_auth_token now has an explicit flag for ssl host checking as wellFix oversight where improved model serialisation logic was not correctly applied to RegistryRemoved unusused parameter refresh from parameters.update_parameter_contextReduced unecessary complexity in utils.dump with no change in functionalityUpdated client gen mustache templates to reflect refactored security and api client codeMinor linting and docstring and codestyle improvementsSet pyUp to ignore Watchdog as it must stay between versions to statisfy py2 and py3 compatibilityIf Client is not instantiated, optimistically instantiate for version checkingadd socks proxy support0.15.0 (2020-11-06)Updated NiFi client and helpers to 1.12.1, Registry client to 0.7.0Release to include new fixes and features in baseline, work continues on improving different Authentication methodsAdded new Parameter contexts API to docsResolved bug where funnel position did not honour requested location (thanks @geocali)Fixed issue where users expected exact search by default but some functions were silently using greedy search (thanks @razdob15)Change deploy_template to use floats for deployment instead of int (thanks @bgeisberger)Fixed creation of empty user groups (thanks @razdob15)0.14.0 (2019-11-06)Updated NiFi client and helpers to 1.10.00.13.3 (2019-10-09)Updated NiFi-Registry client for 0.5.0Several Issues closed as bugfixesMany canvas operations sped-up through refactoring of recursive code to fast iterators0.13.0 (2019-04-22)Updated NiFi client for 1.9.1Major rework of security.py to handle TLS and BasicAuth scenariosMajor rework for test_security.py to cover Issues and common use casesUpdate ‘set_endpoint’ to easily handle TLS and BasicAuth scenarios if https is setAdd default BasicAuth params to configAdd default ‘safe chars’ to config for URL encoding bypass where ‘/’ is in a stringAdd ‘bypass_slash_encoding’ to utils.py to simplify conditionally allowing ‘/’ in a stringUpdate Docker compose files for Secure and tox-full environments to latest NiFi versionsAdd global test controls to top of conftest for default, security, and regression test modesAdd fixtures to conftest for user and usergroup testing in secure scenariosUpdate fixtures to better handle mixed secure and insecure test environments0.12.0 (2018-12-20)Updated NiFi client for 1.8.0Updated NiFi-Registry client for 0.3.0Added Controller Service Management (experimental)Added Connections Management (experimental)New Project Logo! Kindly provided by KDoranFixed several bugs around how the special root Process Group is handled when listing all Project Groups for various methodsVarious backwards compatibility improvements for handling calls going back to NiFi-1.1.2Various speedups for NiFi-1.7+ using descendants functionality to recurse the canvasAbility for various methods to specify a Process Group to use as the parent instead of always using rootBetter username/password handling in security.py and config.pySupport for global ssl_verify squashing in config.pyAdded swagger for 1.8.0 to project resources against potential future validation requirementsAdded versioned deployment convenience functions for finding sensitive and invalid processors, should make it easier to update properties when importing to a new canvasAdded summary options to several calls to return simple objects suitable for quick processing rather than full objects that need to be parsedAdded utils.infer_object_label_from_class to make it easier to create connections between objectsUpdated compound methods like delete_process_group to also handle connections and controllers elegantly if requestedVarious codestyle and testing improvements0.11.0 (2018-10-12)Added steps to fdlc demo to show sensitive and invalid processor testing and behavior during deploymentAdded list_sensitive_processors and list_invalid_processors to nipyapi.canvasAdded simple caching capability for certain calls to nipyapi.configAdded placeholder tests for new functionality against next refactoring and integration runMissing assertion test in get_process_group_statusdeprecated use of tests_require setup.py as current best-practiceUpdate ruamel.yaml to support Python 3.7 with passing testsAdded test for docker image already present to avoid excessive downloadingAdded option to recurse from a given pg_id, rather than always from root, to several canvas functionsAdded default verify_ssl and ssl error squashing to config for user convenienceAdded filter option to specify whether exact or greedy matching should be used, still greedy by defaultAdded hard logout when changing endpoint to ensure tokens are refreshedUpdated testsupdating travis to build all branchesFix travis for Python 3.7 testing supportFix edge case in delete process group where templates stop the revision from being refreshedFixed test case to decode string correctly in old python versionsFixed race condition in test where not all processors started before test executesbugfix for missing status value in Processor DTOUpdating pylint to ignore import errors on standard packagesadded logging to docker image controlBump version: 0.10.3 → 0.11.0Install requirements reset0.10.3 (2018-08-28)Minor bugfix for versioning/deploy_flow_version to resolve additional edge case for version number type0.10.2 (2018-08-27)BugFix for Issue #66 in security/get_access_policy_for_resource where NiFi Api is not expecting a resource_id to be submitted0.10.1 (2018-08-21)Minor bugfix for versioning/deploy_flow_version where version number should be a str instead of int0.10.0 (2018-08-03)Updated NiFi client for 1.7.1 releaseUpdated NiFi-Registry client for 0.2.0 releaseKey ChangesReworked NiFi-Registry pytest setup to support multiple versionsChanged schedule_processor to use component. rather than status. tests as they are more reliableSwtiched Docker configs to use explicit versions instead of latest for more consistent behavior across environentsVersion ChangesDeprecated testing against NiFi-1.5.0 due to host headers issue - recommend users to upgrade to at least NiFi-1.6.0Deprecated testing against NiFi-1.4.0 as superfluousAdded testing for NiFi-1.7.1 and NiFi-Registry-0.2.00.9.1 (2018-05-18)Updated Demos for 0.9 releaseNew FeaturesAdded a new demo for Flow Development LifeCycle which illustrates the steps a user might automate to promote Versioned Flows between NiFi environmentsCheck out nipyapi.demo.fdlc to see more details0.9.0 (2018-05-16)Updated NiFi client to 1.6.0 releasePotentially Breaking ChangesUsers should check the updated documentation and ensure their tests pass as expectedSeveral NiFi client API calls were inconsistently CamelCase’d and have been renamed in the upstream NiFi release, I have honoured those changes in this release. If you use them please check your function names if you get an errorNew FeaturesAdded functionality to Deploy a versioned flow to the canvas. This was an oversight from the 0.8.0 release. Function is creatively named ./versioning/deploy_flow_versionOther NotesUpdated the Issue Template to also ask how urgent the problem is so we can priortise workWhere possible we have switched to using the Apache maintained Docker containers rather than our own, there should be no impact to this unless you were relying on some edge part of our test compose files0.8.0 (2018-03-06)Introducing Secured environment support, vastly expanded Versioning support including import/export.Fixed Templates, better documentation, more demos, and NiFi version backtesting.Potentially Breaking ChangesUsers should check the updated documentation and ensure their tests pass as expectedImport/Export of Flow Versions was reworked significantly and renamed to correct bugs and remove coding complications and be generally more obvious in its behaviorTemplate upload/download reworked significantly to remove direct reliance on requests and correct bugs in some environmentsReworked many list/get functions to be more standardised as we stabilise the approaches to certain tasks. This should not change again in futureStandardised bad user submission on AssertionError, bad API submission errors on ValueError, and general API errors on ApiException. This standard should flow forwardsSwitched ruamel.yaml from >15 to <15 as advised in the project documentation, as >15 is not considered production readyKnown IssuesPython2 environments with older versions of openssl may run into errors like ‘SSLV3_ALERT_HANDSHAKE_FAILURE’ when working in secured environments. This is not a NiPyApi bug, it’s a problem with py2/openssl which is fixed by either upgrading openssl or moving to Python3 like you know you shouldNew FeaturesAdded support for working with secured NiFi environments, contributed by KevDoranAdded demo compatibility between secured_connection and console to produce a rich secured and version-controlled demo environmentAdded many secured environment convenience functions to security.pyIntegrated tokenAuth support throughout the low-level clientsAdded simple Docker deployment support in utils module for test, demo, and developmentStandardised all documentation on more readable docstrings and rst templates across the entire codebaseSignificantly expanded versioning support, users should consult the refreshed documentationAdded experimental support for cleaning queues, process_groups, and setting scheduling of various componentsMany calls now have an auto-refresh before action option to simplify applying changesImplemented short and long wait controls for relevant functions to allow more deterministic changesImplemented generic object-list-filtering-for-a-string-in-a-field for many response get/list typesStandardised many responses to conform to a common response contract: None for none, object for single, and list-of-objects for manyImplemented import/export to json/yaml in versioningAdded regression/backtesting for many functions going back through major release versions to NiFi-1.1.2. More details will be obvious from reading tests/conftest.pyTest suites now more reliably clean up after themselves when executed on long-running environmentsApparently logging is popular, so standard Python logging is now includedOther notesVarious low-level SDK bugfixes corrected in the swagger spec and updated in the provided clientEnhanced Template and Flow Versioning to handle significantly more complex flowsSignificantly enhanced testing fixturesRefactored several common functions to utils.py, and moved several common configurations to config.pyversioning.get_flow will now export the raw Registry object for convenience when serialising flowsSignificantly improved Py2/Py3 compatibility handling, and import management within the packageRemoved docs dependency on M2R by converting everything over to reStructuredText0.7.0 (2018-01-30)Updated project to support NiFi-1.5.0 and NiFi-Registry-0.1.0Merged api clients into main codebase, deprecated external client requirementCreated centralised project configuration and test configurationUpdated automated test environment to consistent docker for local and TravisRemoved procedurally generated boilerplate stub tests to improve readabilityMoved pytest fixtures into conftest and expanded dramaticallyAdded limited support for processor and process group schedulingAdded support for all common Nifi-Registry callsAdded a demo package to provide an interactive test and demo consoleSignificant readme, contribution, and other documentation refreshExpanded CRUD support for most processor, process group and related tasks0.6.1 (2018-01-04)Added requested functions to find and list Processors on the canvasFixed list all process groups to include the root special case properly0.6.0 (2017-12-31)Refactored many functions to use native NiFi datatypes instead of genericsStandardised several call names for consistencyUpdated examplesCreated additional tests and enhanced existing to capture several exceptions0.5.1 (2017-12-07)Added template import/export with working xml parsing and testsAdded a ton of testing and validation stepsCleared many todos out of code by either implementing or moving to todo doc0.5.0 (2017-12-06)migrated swagger_client to separate repo to allow independent versionsrefactored wrapper Classes to simpler functions insteadcleaned up documentation and project administrivia to support the split0.4.0 (2017-10-29)Added wrapper functions for many common Template commands (templates.py)Added new functions for common Process Groups commands (canvas.py)Significant test framework enhancements for wrapper functionsMany coding style cleanups in preparation for filling out test suiteAdded lintingCleaned up docs layout and placement within projectIntegrated with TravisCIDropped Python2.6 testing (wasn’t listed as supported anyway)Updated examples and Readme to be more informative0.3.2 (2017-09-04)Fixed bug where tox failing locally due to coveralls expecting travisFixed bug where TravisCI failing due to incorrectly set install requirementsFixed bug where swagger_client not importing as expected0.3.1 (2017-09-04)Fixed imports and requirements for wheel install from PyPi0.3.0 (2017-09-04)Created basic wrapper structure for future developmentAdded simple usage functions to complete todo taskAdded devnotes, updated usage, and various sundry other documentation cleanupsSplit tests into subfolders for better management and clarityAdded Coveralls and License BadgeRemoved broken venv that ended up in project directory, added similar to ignore fileChanged default URL in the configuration to default docker url and port on localhost0.2.1 (2017-08-26)Fixed up removal of leftover swagger client dependencies0.2.0 (2017-08-25)Merge the nifi swagger client into this repo as a sub packageRestructured tests into package subfoldersConsolidate package configurationSetup package import structureUpdated usage instructionsIntegrate documentation0.1.2 (2017-08-24)Created basic integration with nifi-python-swagger-client0.1.1 (2017-08-24)Cleaned up base project and integrations ready for code migration0.1.0 (2017-08-24)First release on PyPI.