Package: odd-collector-sdk
ODD Collector SDKRoot project for ODD collectorsDomainCollectorConfigMain config file for collectorclassCollectorConfig(pydantic.BaseSettings):default_pulling_interval:int# pulling interval in minutestoken:str# token for requests to odd-platformplugins:Anyplatform_host_url:strCollectorArgs:config_path: str - path to collector_config.yaml (i.e.'/collector_config.yaml')root_package: str - root package for adapters which will be loaded (i.e.'my_collector.adapters')plugins_union_type- Type variable for pydantic model.PluginIs a config for adapterclassPlugin(pydantic.BaseSettings):name:strdescription:Optional[str]=Nonenamespace:Optional[str]=NonePlugin class inherited from Pydantic's BaseSetting,it means it can take any field, which was skipped incollector_config.yaml, from env variables.Fieldtype: Literal["custom_adapter"]is obligatory for each plugin, by convention literalMUSThave same name with adapter packagePlugins example:# plugins.pyclassAwsPlugin(Plugin):aws_secret_access_key:straws_access_key_id:straws_region:strclassS3Plugin(AwsPlugin):type:Literal["s3"]buckets:Optional[List[str]]=[]classGluePlugin(AwsPlugin):type:Literal["glue"]# For Collector's plugins_union_type argumentAvailablePlugin=Annotated[Union[GluePlugin,S3Plugin,],pydantic.Field(discriminator="type"),]AbstractAdapter
Abstract adapter whichMUSTbe implemented by generic adaptersCollector exampleRequirenmentsUse the package managerpoetryto install add odd-collector-sdk and asyncio.poetryaddodd-collector-sdkA typical top-level collector's directory layout (as an example we took poetry project).
├── my_collector            
│   ├── adapters            # Adapters
│   │   ├── custom_adapter  # Some adapter package
│   │   │   ├── adapter.py  # Entry file for adapter
│   │   │   └── __init__.py
│   │   ├── other_custom_adapter
│   │   ├── ...             # Other adapters
│   │   └── __init__.py
│   ├── domain              # Domain models
│   │   ├── ...
│   │   ├── plugins.py      # Models for available plugins
│   │   └── __init__.py
│   ├── __init__.py         
│   └── __main__.py         # Entry file for collector
├── ...
├── collector_config.yaml
├── pyproject.toml
├── LICENSE
└── README.mdAdapters folderEach adapter inside adapters folder must have anadapter.pyfile with anAdapterclass implementingAbstractAdapter# custom_adapter/adapter.py examplefromodd_collector_sdk.domain.adapterimportAbstractAdapterfromodd_models.modelsimportDataEntityList#classAdapter(AbstractAdapter):def__init__(self,config:any)->None:super().__init__()defget_data_entity_list(self)->DataEntityList:returnDataEntityList(data_source_oddrn="test")defget_data_source_oddrn(self)->str:return"oddrn"PluginsEach plugin must implementPluginclass from sdk# domain/plugins.pyfromtypingimportLiteral,Unionfromtyping_extensionsimportAnnotatedimportpydanticfromodd_collector_sdk.domain.pluginimportPluginclassCustomPlugin(Plugin):type:Literal["custom_adapter"]classOtherCustomPlugin(Plugin):type:Literal["other_custom_adapter"]# Needs this type variable for Collector initializationAvailablePlugins=Annotated[Union[CustomPlugin,OtherCustomPlugin],pydantic.Field(discriminator="type"),]collector_config.yamldefault_pulling_interval:10token:""platform_host_url:"http://localhost:8080"plugins:-type:custom_adaptername:custom_adapter_name-type:other_custom_adaptername:other_custom_adapter_nameUsage# __main__.pyimportasyncioimportloggingfromosimportpathfromodd_collector_sdk.collectorimportCollector# Union type of avalable pluginsfrommy_collector.domain.pluginsimportAvailablePluginslogging.basicConfig(level=logging.INFO,format="[%(asctime)s]%(levelname)sin%(module)s:%(message)s")try:cur_dirname=path.dirname(path.realpath(__file__))config_path=path.join(cur_dirname,"../collector_config.yaml")root_package="my_collector.adapters"loop=asyncio.get_event_loop()collector=Collector(config_path,root_package,AvailablePlugin)loop.run_until_complete(collector.register_data_sources())collector.start_polling()loop.run_forever()exceptExceptionase:logging.error(e,exc_info=True)loop.stop()And runpoetryrunpython-mmy_collector