Package: vdk-confluence-data-source
confluence-data-sourceVDK data source plugin for ConfluenceThis plugin enables Versatile Data Kit (VDK) users to easily integrate and ingest data from Atlassian Confluence into their data processing jobs. It provides a streamlined way to access Confluence page content, utilizing the Confluence API to fetch pages from specified spaces or across the entire Confluence site. The plugin supports multiple authentication methods and is designed to work seamlessly with VDK, allowing for the configuration-driven ingestion of Confluence data.UsageTo use thevdk-confluence-data-sourceplugin, install it using pip:pip install vdk-confluence-data-sourceConfigurationConfiguration options can be set in VDK by using thevdk config-helpcommand, which provides a comprehensive list of all configurable options for your VDK installation. For theconfluence-data-sourceplugin, the following configurations are available:ConfigurationConfiguration options for theconfluence-data-sourceplugin can be set using VDK's configuration commands. These options allow you to specify the details required to connect to and interact with your Confluence instance. The available configurations are detailed below:NameDescriptionExample Valueconfluence_urlThe base URL of your Confluence instance."https://your-confluence-instance.atlassian.net/wiki"space_keyThe key of the Confluence space from which to fetch content. Leave empty to fetch from all spaces."DEMO"usernameConfluence username for authentication. Required if using username and API token for authentication."user@example.com"api_tokenConfluence API token for authentication. Can be used instead ofpersonal_access_token."your_api_token"personal_access_tokenConfluence personal access token for authentication. Can be used instead ofapi_token."your_personal_access_token"oauth2OAuth2 credentials for authentication in dictionary format. Includes keys: 'access_token', 'access_token_secret', 'consumer_key', 'key_cert'.{"access_token": "...", "access_token_secret": "...", "consumer_key": "...", "key_cert": "..."}cloudFlag indicating if the Confluence instance is cloud-based.Trueconfluence_kwargsAdditional keyword arguments for the Confluence client.{"timeout": 60}Example usage in configuration:[confluence-data-source-config]confluence_url=https://your-confluence-instance.atlassian.net/wikispace_key=DEMOusername=user@example.comapi_token=your_api_token# personal_access_token = your_personal_access_token# oauth2 = {"access_token": "token", "access_token_secret": "secret", "consumer_key": "key", "key_cert": "cert"}cloud=True# confluence_kwargs = {"timeout": 60}These configurations allow you to tailor the Confluence data source to your specific Confluence instance and authentication method, ensuring secure and efficient access to your Confluence content.ExampleTo use theconfluence-data-sourcein a VDK job:fromvdk.plugin.confluence_data_sourceimportConfluenceDataSource# Configuration exampleconfig={"confluence_url":"https://your-confluence-instance.atlassian.net/wiki","api_token":"your_api_token","space_key":"DEMO",}# Initialize the data sourceconfluence_data_source=ConfluenceDataSource()confluence_data_source.configure(config)# Use the data source in your VDK jobconfluence_data_source.connect()forstreaminconfluence_data_source.streams():forpage_contentinstream.read():print(page_content.data,page_content.metadata)confluence_data_source.disconnect()Build and testingTo build and test theconfluence-data-sourceplugin, follow these steps:pip install -r requirements.txt
pip install -e .
pytestFor plugins that are part of theVersatile Data Kit Plugin repo, you can also use thebuild-plugin.shscript located in the VDK repo for building and testing:../build-plugin.shNote about the CICD:For CI/CD,.plugin-ci.yamlis required for plugins that are part of the Versatile Data Kit Plugin repository. The CI/CD process is divided into two stages: build and release. Each stage consists of jobs that are configured to run on different Python versions (3.7, 3.8, 3.9, and 3.10), ensuring compatibility. The CI/CD runs according to rules designed to trigger the plugin's CI pipeline when changes are made within its directory.