Package: insights-analytics-collector
Insights Analytics CollectorThis package helps with collecting data by user-defined collector methods.
It packs collected data to one or more tarballs and sends them to user-defined URL.Some data and classes has to be implemented.
By function:persisting settingsdata like credentials, content type etc. for shipping (POST request)By Classes:CollectorPackagecollector_module:functions with@registerdecorator, one withconfig=True, format='json'slicing functions (optional) for splitting large data (db tables) by time intervalsCollectorEntrypoint with "gather()" method.ImplementationCollector is an Abstract class, implement abstract methods._package_class: Returns class of your implementation of Package_is_valid_license: Check for valid license specific to your service_is_shipping_configured: Check if shipping to cloud is configured_last_gathering: returns datetime. Loading last successful run from some persistent storage_save_last_gather: Persisting last successful run_load_last_gathered_entries: Has to fill dictionaryself.last_gathered_entries. Load from persistent storage
Dict contains keys equal to collector's registered functions' keys (with @register decorator)_save_last_gathered_entries: Persistingself.last_gathered_entriesAn example can be found inTest collectorPackageOne package represents one.tar.gzfile which will be uploaded to Analytics.
Registered collectors are placed to collections as JSON/CSV files this way:Upload limit is 100MB. The maximum bytes of uncompressed data is MAX_DATA_SIZE (by guess 200MB, redefine if needed)JSON collectors are processed first, it's not expected they'll exceed this sizeif yes, use CSV format insteadCSV files can be collected in two modes:with slicing functionsplitting data by custom function - usually time intervalthe purpose is to have reasonable SQL query in big databases@register(fnc_slicing=...)without slicing functionCSV files are expected to be large (db data), so they can be split byCsvFileSplitterin the collector function.How are files included into packages:JSON files are in first packageCSVs without slicing are included to first free package with enough size (can be added to JSON files)if function collects i.e. 900MB, it's sent in first 5 packagestwo functions cannot have the same name in@register()decoratorCSVs with slicing are sent after each slice is collected (with respect to smaller volume size if running in OpenShift/docker)each slice can be also split by CsvFileSplitter, if bigger than MAX_DATA_SIZEthen each part of slice is sent immediatelytwo functions can have the same name in@register()decoratorNumber of packages (tarballs) is bigger of:number of files collected by one biggest registered CSV collector without slicingnumber of files collected by all registered CSV collectors with slicingcan be +1 for JSON filesSee thetest_gathering.pyfor detailsImplementationPackage is also abstract class. You have to implement basically info for POST request to cloud.PAYLOAD_CONTENT_TYPE: contains registered content type for cloud's ingress serviceMAX_DATA_SIZE: maximum size in bytes ofuncompresseddata for one tarball. Ingress limits uploads to 100MB. Defaults to
200MB.get_ingress_url: Cloud's ingress service URL_get_rh_user: User for POST request_get_rh_password: Password for POST request_get_x_rh_identity: X-RH Identity Used for local testing instead of user and password_get_http_request_headers: Dict with any custom headers for POST requestAn example can be found inTest packageCollector moduleModule with gathering functions is the main part you need to implement.
It should contain functions returning data either indictformat or list of CSV files.Function is registered by@registerdecorator:frominsights_analytics_collectorimportregister@register('json_data','1.0',format='json',description="Data description")defjson_data(**kwargs):return{'my_data':'True'}Decorator@registerhas following attributes:key: (string) name of output file (usually the same as function name)version: (string) i.e. '1.0'. Version of data - added to the manifest.json for parsing on cloud's sidedescription: (string)  not used yetformat: (string) Default: 'json' extension of output file, can be "json" of "csv". Also determines function output.config: (bool) Default: False. therehas to be onefunction withconfig=True, format=jsonfnc_slicing: Intended for large data. Described inSlicing functionbelowshipping_group: (string) Default: 'default'. Splits data to packages by group, if required.from<your-namespace>importCollector# your implementationcollector=Collectorcollector.gather()Slicing functionCollectorsRegistered collectorsAbstract classesTarballs