DATA DRIVEN SYNTHESIZER

A data driven synthesizer (DDS) may receive one or more instructions included in a recipe, wherein the one or more instructions specify a template for retrieving data from one or more data sources. In response to receiving the one or more instructions, the DDS may retrieve the data from the one or more data sources using the template, including inserting one or more values specified by the one or more instructions in place of one or more parameters in the template. The DDS may enable one or more transformation to be performed on the data based on the one or more instructions. The DDS may insert the transformed data into one or more locations in a data store specified by the one or more instructions.

TECHNICAL FIELD

The disclosure relates to a system for data analysis and data reporting.

BACKGROUND

An analyst often analyzes data and presents the data in reports. The analyst may need to retrieve data from multiple data sources. To access the data sources, such as databases or application programs, the analyst may be required to write separate computer code to access each of these different data sources. After retrieving the necessary data, the analyst may use a statistical analysis application to analyze and transform the retrieved data before the data can be placed in the appropriate parts of a report.

SUMMARY

In general, the disclosure is directed to techniques for automating report generation and for separating the tasks of an analyst from that of a database programmer, so that the analyst does not have to know how to program a database to retrieve data. A digital data synthesizer (DDS) may automate the process of data retrieval and report generation, and may remove the burden of data collection from the analyst. The DDS introduces two concepts: recipes and templates. The DDS may receive a recipe that includes a sequence of instructions that directs the DDS to perform a series of tasks to generate a desire report. An analyst may create a recipe that specifies the data to be collected, the data source from which the data is collected, transformations to be performed on the data, and where the data may be stored in a data store. A subject matter expert for a specific data source may create a template that includes code to retrieve data from the specific data source. An analyst can take advantage of a template by referencing the template in the recipe to retrieve data from the data source associated with the recipe. The DDS may receive the recipe and may execute the sequence of instructions included in the recipe. In this way, the process for report generation can be automated.

Furthermore, by separating recipes from templates, analysts do not need to become subject matter experts for the various data sources used by the DDS. Instead, analysts can create custom recipes for their own analytics purposes that references already created templates created by subject matter experts to easily be able to retrieve, manipulate, and present a variety of data from a variety of data sources in a flexible manner. In this way, an analyst may control the entire process of retrieving, transforming, and storing data by writing the appropriate instructions in a recipe.

Benefits of the techniques disclosed in the present disclosure may include workflow efficiencies and time savings compared to previous techniques. The techniques disclosed herein may reduce errors, and may produce reports in a standardized format and in a consistently timely manner. Analysts may no longer have the burden of data collection and may have more control in producing specific reports.

In one aspect, the disclosure is directed to a method. The method may include receiving, by a data driven synthesizer running on at least one processor, one or more instructions included in a recipe, wherein the one or more instructions specify a template for retrieving data from one or more data sources. The method may further include in response to receiving the one or more instructions, retrieving, by the data driven synthesizer, the data from the one or more data sources using the template, including inserting one or more values specified by the one or more instructions in place of one or more parameters in the template. The method may further include performing one or more transformations on the data based on the one or more instructions. The method may further include inserting the transformed data into one or more locations in a data store specified by the one or more instructions.

In another aspect, the disclosure is directed to a computer-readable medium. The computer-readable medium may include instructions that, when executed by at least one processor, cause the at least one processor to perform operations. The operations may include receiving, by a data driven synthesizer, one or more instructions included in a recipe, wherein the one or more instructions specify a template for retrieving data from one or more data sources. The operations may further include, in response to receiving the one or more instructions, retrieving, by the data driven synthesizer, the data from the one or more data sources using the template, including inserting one or more values specified by the one or more instructions in place of one or more parameters in the template. The operations may further include inserting the data into one or more locations in a data specified by the one or more instructions. The operations may further include triggering one or more macros that operate on the data stored in the data store based on the one or more instructions.

In another aspect, the disclosure is directed to a system. The system may include at least one processor. The system may further include at least one memory device. The system may further include at least one module stored by the at least one memory device and executable by the at least one processor, wherein the at least one module is configured to perform operations comprising: receiving, by a data driven synthesizer, one or more instructions included in a recipe, wherein the one or more instructions specify a template for retrieving data from one or more data sources; in response to receiving the one or more instructions, retrieving, by the data driven synthesizer, the data from the one or more data sources using the template, including inserting one or more values specified by the one or more instructions in place of one or more parameters in the template; performing one or more transformations on the data based on the one or more instructions; and inserting the transformed data in place of one or more paste tags in a document based on the one or more instructions.

DETAILED DESCRIPTION

FIG. 1is a block diagram illustrating a system100for generating reports of data according to some aspects of the present disclosure. As shown inFIG. 1, system100includes a data driven synthesizer (DDS)102and one or more recipes104. Each recipe104includes a set of instructions for retrieving and/or transforming data from one or more of data sources110. Each of recipes104may refer to one or more templates106, each of which is associated with one or more of data sources110. Each of recipes104may further refer to one or more transformation modules114, which govern transformation of data retrieved from data sources110according to the instructions included in the recipe104. The transformed data may be stored in one or more data stores112as specified in the recipe104. One or more handlers108, each associated with one of data sources110or one of data stores112, permit DDS102to communicate with data sources110and/or data stores112.

Recipes104may be processed by DDS102to retrieve data from one or more data sources110. The retrieved data may be transformed by transformation modules114, and the transformed data may be stored in data stores112. Reports may be generated as specified by recipes104based on data received from data sources110and/or on transformed data stored in data stores112. By creating or editing recipes104, a user such as a business analyst may be able to produce reports based on data from multiple data sources110. The reports may be generated on a scheduled or as needed basis.

Data driven synthesizer (DDS)102may act as an interchange from the recipes104to retrieve data from data sources requested by recipes104, to enable the retrieved data to be transformed or otherwise processed as specified by recipes104, and to place that retrieved and transformed data into specified locations within data stores and/or other documents as specified by recipes104.

Recipes104may include domain-specific language that may be used to create instructions to control DDS102. For example, recipes104may include documents such as text files that include a set of instructions written in a domain-specific language for processing by DDS102. The set of instructions included in recipes104may include one or more of requests for data from data sources, file system commands, data transformation and processing, database persistence, copying of data to data stores, execution of macros or other instructions to operate on the stored data, and the like.

The instructions in recipes104may include a reference to templates106. Each of templates106may be uniquely associated with one of data sources110and may include code specific to the data source. For example, if one of data sources110is based on a relational database model (RDBMS), then the associated template may include instructions written in structured query language (SQL) designed to manage data held in such a relational database. In other examples, templates106may include code written in shell scripting language, Perl, a data source specific application programming interface, and the like. Because in this example, each of templates106are unique to one of data sources110, each of recipes104may, by specifying a particular template, also specify a data source associated with the template from which to retrieve data. Templates106may be parameterized so that they may include one or more parameters, such as [PageURL]. Recipes104may provide the values of the parameters in the referenced template, such as by including an instruction [PageURL]=www.target.com. DDS102may insert values for the parameters into the referenced template based on the values specified for those parameters in the recipe, and DDS102and may pass the template to a handler of handlers108. The handler may execute the template passed by DDS102to retrieve data from the associated one of data sources110.

Data sources106may include databases, applications, file systems, calendars, text files, application programming interfaces, and any other suitable source of data from which data may be retrieved. DDS102may communicate with a data source data sources110through a handler of handler108associated with the data source to request and receive data from the data source. DDS102is extendable to communicate with and retrieve data from any data source by using an appropriate handler for the data source.

Handlers108may include software that allows DDS102to communicate with data sources108. Each of handlers108is uniquely associated with one of data sources110or one of data stores112. Handlers108may provide a common set of commands that DDS102may use to communicate with data sources110and/or data stores112. Example handlers may include Open Database Connectivity (ODBC) handlers, SiteCatalyst API handlers, file system handers, file transfer protocol (FTP) handlers, Microsoft Excel macro handlers, data warehouse API handlers, ad server handlers, and the like. Handlers108may be attachable, so that a specific handler for a data source may be attached to DDS102in response to DDS102requesting data from the data source associated with the specific handler. Similarly, DDS102may also use handlers108to communicate with data stores112, so that DDS102may utilize a specific hander of handlers108associated with a specific data store of data stores112to communicate with data stores112. In some examples, DDS102may not use a handler of handlers108to communicate with a specific data store. For example, if a data store is a Microsoft Office document, DDS102may use Apache POI to directly manipulate the data model of the document.

Transformation modules114may transform the data retrieved from data sources110according to the instructions included in recipes104. Recipes104may specify one or more transformation modules114to transform the retrieved data. Recipes104may further specify the functionality of the specified transformation module114to be performed on the retrieved data. Transformation modules114may include, for example, statistical analysis applications, data analysis software, and the like. Transformation modules114may also include, for example, functionality such as arithmetic functions, financial functions, statistical functions, probability functions, and the like.

Recipes104may specify that the data retrieved from data sources110and transformed by transformation modules114be stored in one or more of data stores112. Examples of data stores112may include databases, spreadsheets, slide show presentations, word processing documents, flat files, file systems, and the like. In one example, if one of data stores112includes one or more Microsoft Office documents, such as a Microsoft Excel spreadsheet, DDS102may utilize Apache POI to interface and directly manipulate the data model of the document.

Recipes104may also specify that the data be placed in specified locations within a specified one of data stores112via use of paste tags. For example, if the specified data store is a spreadsheet, certain cells of the spreadsheet may include paste tags such as [Impressions], [Revenue], and the like. Recipes104may specify that portions of the data retrieved from the data store corresponding to impressions be stored in place of [Impressions] paste tags in the spreadsheet, and that the data retrieved from the data store corresponding to revenue be stored in place of [Revenue] plate tags in the spreadsheet, for example.

Recipes104may also specify additional processes that may operate on the data stored in the data store. For example, if the specified data store is a spreadsheet, recipe104may specify spreadsheet macros that operate on data stored in a spreadsheet. As another example, the processes that may be specified by recipes104to operate on the data stored in the data store may include processes that generate a report based on the data stored in the data store.

In accordance with an aspect of the present disclosure, DDS102may receive one or more instructions included in one of recipes104. The one or more instructions in the recipe may specify at least one template106, for retrieving data from an associated one of data sources110. In response to receiving the one or more instructions included in the recipe, DDS102may process the one or more instructions and may retrieve the data from the associated one of data sources110using the template106. Retrieving the data from the data source may include inserting one or more parameters specified by the one or more instructions in the recipe into the template106, and may further include passing the template to an associated one of handlers108. The handler may execute the code included in the template106against the data source to request and receive data from the data source.

DDS102may persist the data retrieved from the data source. The one or more instructions included in the recipe may specify that one or more transformation modules of transformation module114perform one or more transformations on the data retrieved from the data source. In response to receiving data from the data source, DDS102may process the one or more instructions in the recipe that specifies the one or more transformations to be performed by the transformation module, and may enable the transformation module to perform one or more transformations to the data based on the one or more instructions included in the recipe. DDS102may process the one or more instructions in the recipe that specifies the location in one of data stores112into which the data is placed, and DDS102may insert the transformed data into one or more locations in the data store specified by the one or more instructions included in the recipe. DDS102may further process one or more instructions in the recipe that specifies a macro or other processes to be executed against the data in the data store and may execute the specified macro or process against the data in the data store based on the one or more instructions included in the recipe.

FIG. 2is a block diagram illustrating an example computing device according to some aspects of the present disclosure. In some examples, computing device2can include fewer, additional, or different components compared to those illustrated inFIG. 2. As shown in the example ofFIG. 2, computing device200may include one or more processors240, one or more input devices242, one or more communication units244, one or more output devices246, user interface device204, and one or more storage devices248. In this example, storage devices248of computing device200may include operating system254, and an application206. Communication channels50may interconnect each of the components204,240,242,244,246, and248for inter-component communications (physically, communicatively, and/or operatively). In some examples, communication channels250may include a system bus, a network connection, an inter-process communication data structure, or any other method for communicating data.

One or more input devices242of computing device200may receive input. Examples of input are tactile, audio, and video input. Input devices242of computing device200, in one example, include a presence-sensitive display, touch-sensitive screen, mouse, keyboard, voice responsive system, video camera, microphone or any other type of device for detecting input from a human or machine.

One or more output devices246of computing device200may generate output. Examples of output are tactile, audio, and video output. Output devices246of computing device200, in one example, include a presence-sensitive display, sound card, video graphics adapter card, speaker, cathode ray tube (CRT) monitor, liquid crystal display (LCD), or any other type of device for generating output to a human or machine. In some examples, user interface device204may include one of one or more output devices246. For example, user interface device204may include a display.

One or more communication units244of computing device200may communicate with wired or wireless external devices via one or more networks by transmitting and/or receiving network signals on the one or more networks. For example, computing device200may use communication units244to transmit and/or receive radio signals on a radio network such as a cellular radio network. Likewise, communication units244may transmit and/or receive satellite signals on a satellite network such as a GPS network. Examples of communication units244include a network interface card (e.g., such as an Ethernet card), an optical transceiver, a radio frequency transceiver, a GPS receiver, or any other type of device that can send and/or receive information. Other examples of communication units244may include Bluetooth®, GPS, 3G, 4G, and Wi-Fi® radios found in mobile devices as well as Universal Serial Bus (USB) controllers.

One or more storage devices248within computing device200may store information required for use during operation of computing device200. Storage devices248, in some examples, have the primary purpose of being short term and not long-term computer-readable storage mediums. Storage devices248on computing device200may be configured for short-term storage of information as volatile memory and therefore not retain stored contents if powered off. Examples of volatile memories include random access memories (RAM), dynamic random access memories (DRAM), static random access memories (SRAM), and other forms of volatile memories known in the art. Storage devices248may further be configured for long-term storage of information as non-volatile memory space and retain information after power on/off cycles. Examples of non-volatile memories include magnetic hard discs, optical discs, floppy discs, flash memories, or forms of electrically programmable memories (EPROM) or electrically erasable and programmable (EEPROM) memories. Storage devices248may store program instructions and/or data associated with one or more of application206.

One or more processors240may implement functionality and/or execute instructions within computing device200. For example, processors240on computing device200may read and execute instructions stored by storage devices248that execute the functionality of application206. These instructions executed by processors240may cause computing device200to store information within storage devices248during program execution. Processors240may execute application206to perform various actions.

Storage devices248may include application206. Application206may execute on processors240to receive a document via communication units244and to render the content specified by the received document for display at a display device, such as user interface device204included or otherwise operably coupled to computing device200.

Application206may include data driven synthesizer (DDS)262, similar to DDS102shown inFIG. 1. Computing device200may receive, via communication units244or input devices242, a recipe, such as recipe104shown inFIG. 1. The recipe may include one or more instructions written in a domain specific language for DDS262. DDS262may process the one or more instructions in the recipe, including translating the one or more instructions into machine language for computing device200so that the one or more instructions may be executed by processors240. DDS262may retrieve any templates specified by the recipe from templates266, similar to templates106shown inFIG. 1, and may replace parameters in the template(s) with the values specified in the recipe. DDS262may also select an appropriate handler out of handlers264, similar to handlers108shown inFIG. 1, to communicate with the data source associated with the template.

DDS262may, via communication units244and the appropriate handler for the data source, communicate with the data source to retrieve data. DDS262may, after retrieving the data, persist the data in storage device248. DDS262may also, based on the recipe, trigger transformation application268to perform one or more functions, such as statistical analysis, to the retrieved data. DDS262may also, based on the recipe, store the data in a document and/or data store in storage device248.

FIG. 3is a block diagram illustrating an example system for generating reports of coupon usage according to some aspects of the present disclosure. As shown inFIG. 3, an example recipe304, similar to one of recipes104shown inFIG. 1, may be communicated to DDS302, similar to DDS102shown inFIG. 1. Example recipe304includes the following example instructions:TemplateName=coupon_template.txt[StartDate]=‘01/01/2014’[EndDate]=‘01/07/2014’[CouponID]=1234Transform=StatAnalysis.Function1OutputFileName=couponstats.xlsxPlace 1234 into [CouponID]Place TransformedOutputData into [CouponStats]Run Macro1 on [CouponStats]

Template306may include instructions for retrieving specified data from a data store, and may include, for example, parameters [StartDate], [EndDate] and [CouponID] in its instructions. Template306may retrieve coupon use details for a coupon specified by the parameter [CouponID] between the dates specified by the parameters [StartDate] and [EndDate].

DDS302may process the instructions in example recipe304and may retrieve the coupon template.txt template based on the instructions in recipe304. Because recipe304defines values for parameters [StartDate], [EndDate] and [CouponID] as [StartDate]=‘01/01/2014’, [EndDate]=‘01/07/2014’, and [CouponID]=1234, DDS302may replace the parameters [StartDate], [EndDate] and [CouponID] in template306with the values ‘01/01/2014’, ‘01/07/2014’, and 1234, respectively.

DDS302may, based on the instructions in recipe304, determine that recipe304includes a request for data from coupon database310. DD302may select an associated handler, such as handler308, similar to one of handlers108shown in FIG.1., to communicate with coupon database310. DDS302may pass filled-in template306to handler308, and handler308may execute the instructions in template306against coupon database310to retrieve data. The data retrieved from coupon database310may be returned to DDS302, and DDS302may persist the retrieved data.

Recipe304may specify a specific statistical analysis application314and a specific function of statistical analysis application314to perform on the retrieved data. In the example ofFIG. 3, recipe304includes the instruction Transform=StatAnalysis.Function1 that specifies the StatAnalysis application to perform Function1 on the retrieved data. DDS302may, based on the instruction in recipe304, pass the retrieved data to statistical analysis application314and may pass instructions for statistical analysis application314to perform the specified Function1 on the data, and statistical analysis application314may return the transformed data back to DDS302.

Recipe304may specify a location of a specific data store into which the transformed data is placed. In the example ofFIG. 3, recipe304includes the instructions of OutputFileName=couponstats.xlsx, Place1234into [CouponID], and Place TransformedOutputData into [CouponStats]. DDS302may, based on the instructions in recipe304, place the value of1234into the [CouponID] paste tag in couponstats.xlsx312, and may place the transformed output data into the [CouponStats] paste tag in couponstats.xlsx312. As discussed above, because couponstats.xlsx312is a Microsoft Excel spreadsheet, DDS304may directly operate on the data model of couponstats.xlsx312to place the data into the paste tags by using Apache POI.

Recipe304may further specify additional operations that may be performed on the data placed into couponstats.xlsx312. In the example ofFIG. 3, recipe304includes the instruction of Run Macro1 on [CouponStats], which specifies that Macro1 be performed on the data placed into the [CouponStats] paste tag in couponstats.xlsx312. DDS302may, based on the instruction in recipe304, enable the specified Macro1 to be performed on the data placed into the [CouponStats] paste tag in couponstats.xlsx312. In this way, the instructions in recipe304enable DDS302to perform a process of retrieving data from a data source, transforming the data, and placing the data into specified locations of a data store.

FIG. 4is a block diagram illustrating an example system for generating reports of data using nested recipes according to some aspects of the present disclosure. As shown inFIG. 4, an example recipe404A, similar to one of recipes104shown inFIG. 1, may be communicated to DDS402, similar to DDS102shown inFIG. 1. Example recipe404A may include the following instructions:REQUEST=EXPORTTemplateName=VS_SpeciaPage_Impressions[PageURL]=www.target.com[ExportOutput]=[MC_PoP_Recent:dvm_id]-HomePageImpressions[FiscalWeekBegin]=[MC_PoP_Recent:FiscalWeekBegin][FiscalWeekEnd]=[MC_PoP_Recent:FiscalWeekEnd]

Recipe404A may define values for parameters [PageURL], [ExportOutput], [FiscalWeekBegin], and [FiscalWeekEnd]. However, in this example, recipe404A does not directly define values for parameters [ExportOutput], [FiscalWeekBegin], and [FiscalWeekEnd]. Instead, recipe404A refers to another recipe MC_PoP_Recent for the values of these parameters. In the example ofFIG. 4, the value of parameter [ExportOutput] is based in part on the value defined by recipe MC_PoP_Recent for its parameter [dvm_id], the value of [FiscalWeekBegin] is the value defined by recipe MC_PoP_Recent for its parameter [FiscalWeekBegin], and the value of [FiscalWeekEnd] is the value defined by recipe MC_PoP_Recent for its parameter [FiscalWeekEnd]. Recipe404B may include recipe MC_PoP_Recent and may, for example, define the value of [dvm_id] as 1234, define the value of [FiscalWeekBegin] as 01/01/2014, and define the value of [FiscalWeekEnd] as 1/07/2014.

DDS402may receive example recipes404A and404B. Based on recipe404A, DDS402may retrieve template VS_SpecialPage_Impressions406. DDS402may replace the parameters in template406with the values defined in recipe404A. Because some of the values defined for parameters by recipe404A depends on the values of parameters in recipe404B, DDS402may read recipe404B in conjunction with reading404A to insert the proper values in place of parameters [ExportOutput], [FiscalWeekBegin], and [FiscalWeekEnd] in template406. DDS402may send template406to handler408, and handler408may execute template406to retrieve data from file system410. In this way, recipe404A may nest recipe404B so that recipe404A may define values for parameters in terms of the values defined for parameters in recipe404B.

FIG. 5is a conceptual diagram illustrating an example generated report according to some aspects of the present disclosure. A report may be generated using data retrieved, transformed, and stored via the techniques described above with respect toFIGS. 1-4. As shown inFIG. 5, a report502may include, for example, a summary of impressions received for a particular product that is being sold online and in retail stores during a marketing campaign for the product. An impression, in one example, may include a delivered basic advertising unit for the product. Report502may, for example, include the number of impressions the product received, and may break down the impressions received by whether the impression was received through the store's website, e-mail campaigns, a user's search results, or a physical display at the retail store. Report502may, for example, also include charts and graphs regarding the impressions received by the product via the store's website and the number of website clicks received for the product. Report502may also include coupon redemption statistics for the product. Report502may also include additional information, such as the marketing program summary for the product, the product coupons used to promote the product, as well as the product's web page on the store's website.

FIG. 6is a conceptual diagram illustrating an example user interface for generating reports according to some aspects of the present disclosure. As shown inFIG. 6, user interface602may be provided by a software application to a user, such as a business analyst, so that the business analyst may be able to create a report, such as report502shown inFIG. 5, via an easy-to-use user interface. For example, user interface602may include a user interface for creating a report regarding the number of impressions for a product from a variety of channels. For example, user may select the channels to include in a report, such as by clicking buttons or otherwise interacting with user interface elements of user interface602. In response, a recipe for retrieving data regarding channel impressions may be edited so that it retrieves data for the channels selected by the user. User interface602may also display the data retrieved in response to the user input. If the user is satisfied with the results, the user may interact with user interface602to create the report, such as by clicking “Create Report” button604.

FIG. 7is a flowchart illustrating a method for generating reports of data according to some aspects of the present disclosure. As shown inFIG. 7, data driven synthesizer (DDS)102may receive one or more instructions, wherein the one or more instructions specify a template106for retrieving data from one or more data sources110(702). In response to receiving the one or more instructions, DDS102may retrieve the data from the one or more data sources110using the template106, including inserting one or values specified by the one or more instructions in place of one or more parameters in the template106(704). Transformation module114may perform one or more transformations on the data based on the one or more instructions (706). DDS102may insert the transformed data into one or more locations in a data store112specified by the one or more instructions (708).

In some examples, retrieving the data from the one or more data sources may further include sending, by DDS102, the template106to one or more handlers108associated with the one or more data sources110and executing, by the one or more handlers108, instructions included in the template106. In some examples the data store112comprises a document. In some examples, the process may further include triggering one or more macros that operate on the data stored in the document based on the one or more instructions. In some examples, inserting the transformed data into one or more locations in a data store112specified by the one or more instructions further includes placing the transformed data in the document in place of one or more paste tags in the document based on the one or more instructions. In some examples, the process may further include generating a report based on the transformed data stored in the data store112. In some examples, the one or more instructions included in the recipe104specify the one or more values for the one or more instructions by referring to a second recipe that defines the one or more values for the one or more instructions.