Package: data-decomposition-platform
gTech Ads - Data Decomposition PackageData Decomposition Package is an open-source Python package to decompose aggregated observations down to a single observation level.This is not an official Google product.Introduction•Decomposition processIntroductionIn a privacy-first world, an increasing volume of data is be collected on an aggregated level which means that certain data points will not contain information on a single observational level. Instead these values are aggregated by a shared id where shared id could be a purchase date, a purchase time window, and others.This level of data aggregation can be a major challenge when working with data pipelines and projects expecting data structures oriented on an individual-level observation. The single observation levels are representative of the aggregated whole, allowing for privacy-first methods to be used with existing data pipelines and methods which otherwise may break with new privacy-first, aggregated upstream methods.gTech Ads Data Decomposition Platform is an open-source Python package to decompose aggregated observations down to a statistically representative single observation level.The package implements 2 methods: 1) Bayesian Disaggregator and 2) Neural Networks Disaggregator. The former allows for decomposing continuous values, while the latter can handle both the continuous and categorical values. Both methods use a similar API.Decomposition process with Disaggregator methodsFirst, the input data needs to be either a comma-separated CSV file or a Pandas DataFrame.dataframe = pd.read_csv(“path/to/your/file.csv”)Decomposition with Bayesian Disaggregator methodNumerical aggregated dataThe default Bayesian Disaggregator is compatible with only one input feature. It requires the dataframe to have a column with shared id (e.g. date, but represented as an integer), a single feature X (e.g. items in a basket), a feature X aggregated by shared id and a numerical aggregated value Y to be later decomposed in the process.This method simply estimates a weight to distribute aggregated value Y into value Y, given learned weights from aggregated feature X to feature X. It assumes the weighting is the same between X and Y and that each group always has the same number of items.Example 1. Sample dataframe compatible with the Bayesian Disaggregator (numerical aggregated data).shared_idfeature_Xaggregated_feature_Xaggregated_value_Y01310.502310.515916.814916.8The following step is to provide a mapping betweenshared_id,x_aggregates,x_valuesandy_aggregatesand the respective column names in the input dataframe.column_name_mapping = preprocessors.BayesianColumnNameMapping(
    shared_id="shared_id",
    x_aggregates="aggregated_feature_X",
    x_values="feature_X",
    y_aggregates="aggregated_value_Y",
)Next, you initialize the Bayesian model that you wish to use in the decomposition process. You can replace the default model with a customized one as long as it’s compatible with theBayesianDisaggregatorclass.bayesian_model = methods.BayesianModel()The final steps are to initialize theBayesianDisaggregatorclass and decompose the input dataframe.bayesian_disaggregator = methods.BayesianDisaggregator(model=bayesian_model)
disaggregated_dataframe = bayesian_disaggregator.fit_transform(
    dataframe=dataframe, column_name_mapping=column_name_mapping
)Example 2. Sample output from the Bayesian Disaggregator  (numerical aggregated data).shared_idx_aggregatesx_valuesy_aggregatesdisaggregated_y_values03110.55.003210.55.519516.86.819416.810.0Decomposition with Neural Networks Disaggregator methodNumerical aggregated dataThe default Neural Networks Disaggregator is compatible with multiple input features.
It requires the dataframe to have a column with shared id (e.g. date, but represented as an integer), one or more features (e.g. items in a basket, browser type, device type etc.) and numerical aggregated value Y to be later decomposed in the process.Example 3. Sample dataframe compatible with the Neural Networks Disaggregator (numerical aggregated data).shared_idfeature_1feature_2aggregated_value_Y01.51.310.503.45.610.5110.10.016.814.59.916.8The following step is to provide a mapping betweenshared_id,featuresandy_aggregatesand the respective column names in the input dataframe.column_name_mapping = preprocessors.NeuralNetworksColumnNameMapping(
    shared_id="shared_id",
    features=["feature_1", “feature_2”],
    y_aggregates="aggregated_value_Y",
)Next, you initialize the Neural Networksregressionmodel that you wish to use in the decomposition process. You can replace the default model with a customized TensorFlow ModelTensorFlow Model.task_type = "regression"
model = methods.NeuralNetworksModel(task_type=task_type)The final steps are to initialize theNeuralNetworksDisaggreagtorclass and decompose the input dataframe. Thecompileandfitarguments accept the exact same kwargs as thecompileandfitmethods of aTensorFlow Model. Make sure that you use a loss type compatible with the regression problems when decomposing numerical data.disaggregated_dataframe = neural_network_disaggregator.fit_transform(
    dataframe=input_dataframe,
    column_name_mapping=column_name_mapping,
    compile_kwargs=dict(loss="mean_absolute_error", optimizer="adam"),
    fit_kwargs=dict(epochs=30, verbose=False),
)Example 4. Sample output from the Neural Networks Disaggregator (numerical aggregated data).shared_idfeature_1feature_2y_aggregatesdisaggregated_y_values01.51.310.55.003.45.610.55.5110.10.016.86.814.59.916.810.0Categorical aggregated dataThe statistical windowing process of categorical aggregated data with the Neural Networks Disaggregator is almost identical. At this point the Neural Networks Disaggregator can work only with binary problems. E.g. there is an aggregated number of conversions per given day and the model will only be able to assign 0 to users who did not convert and 1 otherwise.Example 5. Sample dataframe compatible with the Neural Networks Disaggregator (categorical aggregated data).shared_idfeature_1feature_2aggregated_value_Y01.51.3203.45.62110.10.0114.59.91The following step is also to provide a mapping betweenshared_id,featuresandy_aggregatesand the respective column names in the input dataframe.column_name_mapping = preprocessors.NeuralNetworksColumnNameMapping(
    shared_id="shared_id",
    features=["feature_1", “feature_2”],
    y_aggregates="aggregated_value_Y",
)Next, you initialize the Neural Networksclassificationmodel that you wish to use in the decomposition process.task_type = "classification"
model = methods.NeuralNetworksModel(task_type=task_type)The final steps are to initialize theNeuralNetworksDisaggreagtorclass and decompose the input dataframe. Make sure that you use a loss type compatible with the binary classification problems when decomposing categorical data.neural_network_disaggregator = methods.NeuralNetworksDisaggregator(
    model=model,
    task_type=task_type,
)
disaggregated_dataframe = neural_network_disaggregator.fit_transform(
    dataframe=input_dataframe,
    column_name_mapping=column_name_mapping,
    compile_kwargs=dict(loss="bce", optimizer="adam"),
    fit_kwargs=dict(epochs=30, verbose=False),
)Example 5. Sample output from the Neural Networks Disaggregator (categorical aggregated data).shared_idfeature_1feature_2y_aggregatesdisaggregated_y_values01.51.310.5103.45.610.51110.10.016.8014.59.916.81