Package: dspreview
Tool for merging DSP data from many providers in a single view. There are CLI
tools for launching the workers responsible for parsing.csvfiles and
storing them in a MySQL database. There is also a web app where it is possible
to have a complete report of the operation.ConfigurationYou must specify the following environment variables prior to usage:DB_HOSTthe url or ip for the mysql serverDB_PORTthe port for the mysql serverDB_NAMEthe name of the databaseDB_USERa user with writing permissionDB_PASSthe user’s passwordIf you inted to develop or change something, it is also need:DB_TEST_NAMEthe name of the database (for testing purposes)DB_TEST_USERa user with writing permission (for testing purposes)DB_TEST_PASSthe user’s password (for testing purposes)Since this project has support only for GCP (currently), the following
environment variables are also mandatory:GOOGLE_APPLICATION_CREDENTIALSthe json file for an account with admin permissions for theStorageservice.GCP_BUCKETthe bucket where the.csvfile will be placedIt is possible to set the workers to consume a RabbitMQ queue, so it is necessary
to specify the server info:MQ_HOSTthe RabbitMQ server ip addressMQ_PORTthe RabbitMQ portMQ_VHOSTthe RabbitMQ virtual hostMQ_USERthe RabbitMQ userMQ_PASSthe RabbitMQ passwordMQ_QUEUEthe RabbitMQ queue name (we might change for workers operating through exchanges later, so we can have multiple workers operating at the same time, but for now lets use one queue).A much better option would be to set all these variables in a file named.dspreview.csgin the users home folder:{
    "GOOGLE_APPLICATION_CREDENTIALS": "/home/user/service_account.json",
    "GCP_BUCKET": "...",
    "DB_HOST": "...",
    "DB_PORT": "3306",
    "DB_NAME": "...",
    "DB_USER": "...",
    "DB_PASS": "..."
}If the above environment variables are set, you can initialize the system.
It will create the database, tables, and so on. It might be donne through:$ dspreview initThere are currently two workers:dcmanddsp. Thedcmworker expects
to find a file nameddcm.csvinside theGCP_BUCKET, with the
following structure:[date, campaign_id, campaign, placement_id, placement, impressions, clicks, reach]where:dateshould be in format YYYY-MM-DDcampaign_idis an integercampaignis a string with the informationbrand_subbrandplacement_idis an integerplacementis a string with the informationdsp_adtypeimpressionsis an integerclicksis an integerreachis an float, please take care to not repeat this, since it is a calculated metricWhile thedspworker expect to find a file with the dsp’s name (likedbm.csvormediamath.csv) and the following structure:[date, campaign_id, campaign, impressions, clicks, cost]where:dateshould be in format YYYY-MM-DDcampaign_idis an integercampaignis a string with the informationbrand_subbrand_adtypeimpressionsis an integerclicksis an integercostis a floatIn order to launch a worker, you might use the command:$ dspreview --worker dcmor:$ dspreview --worker dspIf the DSP is known in advance, you might run:$ dspreview --worker dsp --dsp dbmor$ dspreview --worker dsp --dsp mediamathWhen all files are stored in the MySQL database, the following command generates
the full report:$ dspreview --generate-reportThe web app might be run through:$ dspreview serve --port 80The default port is80Finally, it is possible to put the worker to run in a loop, in this case, it
will consume a queue in the RabbitMQ. The messages must be:dcmfor the DCM workerdspfor running all DSP workersdsp.dbmfor running a specific DSP worker (DBM in this case)reportfor generating full reportThe worker might be launched as:$ dspreview operateIt is possible to add itens to the queue through:$ dspreview --poke "dsp.dbm"ClassificationsBesides the configuration described above, it is also important to understand theclassifications. The classifications might be managed through the webserver
in the underlying section.The idea is that each line in the DCM and DSP files might be classified according
to abrand, asub brand, and adsp. You can create a regex that
will be checked against a combination of fields.For instance, it is a line in the DBM file:datecampaign_idcampaignimpressionsclickscost2018-01-01128115acme_asprin_youtube60110701188940334.2797You might choose to apply the regex.*acme.*for classifying the brand asAcme, and the only necessary field iscampaign.Chances are that you have another brand, sayUmbrella Corp, and for some reason, you end up with a line like:datecampaign_idcampaignimpressionsclickscost2018-03-01475987acme_solution48678674645487897.4558Now, the regex above would classify both this lines asAcme.A solution would be change the regex to^128115acme.*applied to the
concatenation of fieldscampaign_idandcampaign, in order to  identify
ads belonging to the brandAcme.For identifying ads belonging toUmbrella Corpthe regex could be^475987acme.*applied to the concatenation of fieldscampaign_idandcampaign.The fields that can be used are:campaign_idcampaignplacement_id (form DCM files only)placement (form DCM files only)Fields will be concatenated in this order.The regex patterns will be applied in the order in which they are registered
in the database. The first matching a combination of fields will define the
classification, so it is necessary to avoid ubiquitous regex.Preparing for DevelopmentEnsurepipandpipenvare installed.Make sure you also havedefault-libmysqlclient-devorlibmysqlclient-devinstalled.Clone repository:https://github.com/thiagolcmelo/dspreviewFetch development dependencies:make installRunning TestsRun tests locally usingmakeif virtualenv is active:$ makeIf virtualenv isn’t active then use$ pipenv run make