Package: cubestories
CubeStoriesCubeStories allows querying Linked Open Statistical by Providing parameters in a form of Python dictionaries(JSON).Reserach paper underpinning the implementationResearch paperHigh-level encapsulation of datastories librabryDataStories repoTest sripts are available at:Test reporInstallationpipinstallcubestoriesRequirements will be automatically installed with packageImport/UsagefromCubeStoriesimport*UsageLibrary implements 3 artifacts required for Data analysisMetadata Parameters - metadata required for SPARQL queries{"sparqlEndPointUrl":"[SPARQL ENDPOINT URL]","jsonMetaDataFile":"[directory of JSON file with metadata]"}Cube Parameters - what properties of cube to be retrieved from endpoint(based on JSON file provided in Metadata Parameters).
Values highlighted as: ```--- --- `` has to be specified by a user - replaced to value only{"cube":"---Key of Cube ---","dimensions":["---List of dimensions---"],"measures":["---List of Measures---"],"hierdimensions":{"---DimKey---":{"selected_level":"---levelkey---"}}Analysis Pipeline - JSON-based list of pattern analysis to be performed. Each Pattern will have such template provided{"---PatternName---":{"parameter1":["---list of values---"],"parameter2":"---value---"},"---PatternName----":{"parameter1":"---pattern1 value---","parameter2":["---list of values---"]}}JSON Template - one of the metadata parameters{"---cube_key---":{"title":"---title of cube---","dataset_structure":"---URI for cube structure---","dimensions":{"---dimension_key---":{"dimension_title":"---Title of diemnsion---","dimension_url":"---URI for dimension---","dimension_prefix":"---URI for dimension's values---"},"---dimension_key---":{"dimension_title":"---Title of diemnsion---","dimension_url":"---URI for dimension---","dimension_prefix":"---URI for dimension's values---"}},"hierarchical_dimensions":{"---dimension_key---":{"dimension_title":"---Title of diemnsion---","dimension_url":"---URI for dimension---","dimension_prefix":"---URI for dimension's values---","dimension_levels":{"---level_key---":{"description":"---description of granularity level---","granularity":"---integer level of granularity---"},"---level_key---":{"description":"---description of granularity level---","granularity":"---integer level of granularity---"}}}},"measures":{"---measure_key---":{"measure_title":"---Title of measure---","measure_url":"---URI for measure---"}}}}Patterns DescriptionComments after ## are just for descriptive purposes. REMOVE THEM WHEN SPECIFYING PIPELINEMeasurement and CountingLeague TableInternal ComprisonProfile OutliersDissect FactorsHighlight ContrastStart Big Drill DownStart Small Zoom OutAnalysis By CategoryExplore IntersectionNarrating Change Over TimeMCountingMeasurement and Counting
Arithemtical operators applied to whole dataset - basic information regarding dataAttributes"MeasCount":{"count_type":"count value"}ParameterTypeDescriptioncount_typeStringType of Count to performOutputBased on count_type valueCount_typeDescriptionrawdata without any analysis performedsumsum across all numeric columnsmeanmean across all numeric columnsminminimum values from all numeric columnsmaxmaximum values from all numeric columnscountamount of recordsLTableLeagueTable - sorting and extraction specific amount of recordsAttributes"LeagueTab":{"columns_to_order":["list of columns to order by"],"order_type":"type of order by","number_of_records":5}ParameterTypeDescriptioncolumns_to_orderlist[String]Set of columns to order byorder_typeStringType of order (asc/desc)number_of_recordsIntegerAmount of records to retrieveOutputBased on sort_type valueSort_typeDescriptionascascending order based on columns provided incolumns_to_orderdescdescending order based on columns provided incolumns_to_orderInternalComparisonInternalComparison - comparison of numeric values related to textual values within one columnAttributes"IntComp":{"dim_to_compare":"dimension to compare","meas_to_compare":"measure to compare","comp_type":"comparison type"}ParameterTypeDescriptiondim_to_compareStringDimension, which values will be investigatedmeas_to_compareStringMeasure, which numeric values related todim_to_comparewill be processedcomp_typeStringType of comparison to performOutputIndependent fromcomp_typeselected, output data will have additional column with numerical columnmeas_to_compareprocessed in specific way.Available types of comparisoncomp_typeComp_typeDescriptiondiffmaxdifference with max value related to specific textual valuediffmeandifference with arithmetic mean related to specific textual valuesdiffmindifference with minimum value related to specific textual valueProfileOutliersProfileOutliers - detection of unusual values within data (anomalies)Attributes"ProfOut":{"display_type":"Gender"}ParameterTypeDescriptiondisplay_typeStringWhat information display are bound to display (with/without anomalies)OutputPattern analysis usingpython scipylibrary will perform quick exploration in serach of unusual values within data.Based ondisplay_typeparameter data will be displayed with/without ddetected unusual values.Available types of displayingdisplay_typedisplay_typeDescriptionoutliers_onlyreturns rows from dataset where unusual values were detectedwithout_outliersreturns dataset with excluded rows where unusual values were detectedDissectFactorsDissectFactors - decomposition of data based on values in dim_to_dissectAttributes"DissFact":{"dim_to_dissect":"dimension to dissect"}ParameterTypeDescriptiondim_to_dissectStringBased on which dimension data should be decomposedOutputAs an output, data will be decomposed in a form of a dictionary, where each subset have values only related to specific value.
Dictionary of subdataset will be constructed as a series of paiers where key per each susbet will values fromdim_to_dissectand this key value will be data, where yhis key value was occurring.HighlightContrastHighlightContrast - partial difference within values related to one textual columnAttributes"HighCont":{"dim_to_contrast":"dimension to contrast","meas_to_contrast":"measure to contrast","contrast_type":"type of contrast"}ParameterTypeDescriptiondim_to_contrastStringTextual column, from which values will be contrastedmeas_to_contrastStringNumerical column, which values are contrastedcontrast_typeStringType of contrast to presentOutputIndependent fromcontrast_typeselected, output data will have additional column with numerical columnmeas_to_contrastprocessed in specific way.Available types of comparisoncontrast_typeContrast_typeDescriptionpartofwholedifference with max value related to specific textual valuepartofmaxdifference with arithmetic mean related to specific textual valuespartofmindifference with minimum value related to specific textual valueStartBigDrillDownStartBigDrillDown - data retrieval from multiple hierachical levels.This pattern can be only applied to data not stored already in DataFrameAttributes"StBigDrillDown":{"hierdim_drill_down":{"Key of hierarchical dimension":["dimlevel1key","dimlevel2key","dimlevel3key"]}}ParameterTypeDescriptionhierdim_drill_downdict{hierdim:list[str]}Hierarchical dimension with list of hierarchy levels to inspectOutputAs an output, data will be retrieved in a form of a dictionary, where each dataset will be retrieved from different hierachy level. List will be provided inhierdim_drill_down. Hierachy levels provided by in parameter will automatically sorted in order from most general to most detailed level based on metadata provided.StartSmallZoomOutStartSmallZoomOut - data retrieval from multiple hierachical levels.This pattern can be only applied to data not stored already in DataFrameAttributes"StSmallZoomOut":{"hierdim_zoom_out":{"Key of hierarchical dimension":["dimlevel1key","dimlevel2key","dimlevel3key"]}}ParameterTypeDescriptionhierdim_zoom_outdict{hierdim:list[str]}Hierarchical dimension with list of hierarchy levels to inspectOutputAs an output, data will be retrieved in a form of a dictionary, where each dataset will be retrieved from different hierachy level. List will be provided inhierdim_zoom_out. Hierachy levels provided by in parameter will automatically sorted in order from most detaile to most general level based on metadata provided.AnalysisByCategoryAnalysisByCategory - ecomposition of data based on values in dim_for_category with analysis performed on each susbetAttributes"AByCategory":{"dim_for_category":"dimension for categorisation","meas_to_analyse":"measure to perform analysis","analysis_type":"type of analysis"}ParameterTypeDescriptiondim_for_categoryStringDimension, based on which input data will be categorisedmeas_to_analyseStringMeasure, which will be analysedanalysis_typeStringType of analysis to performOutputAs an output, data will be decomposed in a form of a dictionary, where each subset have values only related to specific value. Such subset will get analysed based onanalysis_typeparameterAvailable types of analysisanalysis_typeAnalysis_typeDescriptionminMinimum per each categorymaxMaximum per each categorymeanArithmetical mean per each categorysumTotal value from each categoryExploreIntersectionAttributes"ExpInt":{"dim_to_explore":"dimension to explroe across cubes"}ParameterTypeDescriptiondim_to_exploreStringDimension, which existence within enpoint is going to be investigatedOutputPattern will return series of datasets, where each will represent occurence ofdim_to_explorein one cubeNarratingChangeOverTimePresenting difference between 2 numerical properties of dataAttributes"NarrChangeOT":{"meas_to_narrate":["list of two dimensions to present change"],"narr_type":"type of narration"}ParameterTypeDescriptionmeas_to_narrateStringSet of 2 measures, which change will be narratednarr_typeStringType of narration to performOutputIndependent fromnarr_typeselected, output data will have additional column with numerical values processed in specific way.Available types of analysisnarr_typeNarr_typeDescriptionpercchangePercentage change between first nad second propertydiffchangeQuantitive change between first and second property