Package: artificial-data-generator
Data generator for synthetic data including artificial classes, intraclass correlations, pseudo-classes and random data -Sphinx DocTable of ContentsPurposeData structureDifferent parts of the data setData distribution and effect sizesCorrelationsPseudo-classesRandom FeaturesInstallationLicensingPurposeIn order to develop new methods or to compare existing methods for feature selection, reference data with known dependencies and importance of the individual features are needed. This data generator can be used to simulate biological data for example artificial high throughput data including artificial biomarkers. Since commonly not all true biomarkers and internal dependencies of high-dimensional biological datasets are known with
certainty, artificial dataenables to know the expected outcome in advance. In synthetic data, the feature importances and the distribution of each class are known. Irrelevant features can be purely random or belong to a pseudo-class. Such data can be used, for example, to make random effects observable.Data structureDifferent parts of the data setThe synthetic-data-generator produces data sets consisting of up to three main parts:Relevant featuresbelonging to an artificial class (for example artificial biomarkers)[optional]Pseudo-classes(for example a patient's height or gender, which have no association with a particular disease)[optional]Random datarepresenting the features (for example biomarker candidates) that are not associated with any classThe number of artificial classes is not limited. Each class is generated individually and then combined with the others.
In order to simulate artificial biomarkers in total, all individual classes have the same number of features in total.This is an example of simulated binary biological data including artificial biomarkers:Data distribution and effect sizesFor each class, either thenormal distribution or the log normal distributioncan be selected. The differentclasses can be shiftedto regulate the effect sizes and to influence the difficulty of data analysis.The normally distributed data could, for example, represent the range of values of healthy individuals.
In the case of a disease, biological systems are in some way out of balance.
Extreme changes in values as well as outliers can then be observed (Concordet et al., 2009).
Therefore, the values of a diseased individual could be simulated with a lognormal distribution.Example of log-normal and normal distributed classes:CorrelationsIntra-class correlation can be generated for each artificial class. Any number of groups
containing correlated features can be combined with any given number of uncorrelated features.However, a high correlation within a group does not necessarily lead to
a high correlation to other groups or features of the same class. An example of a class with three
highly correlated groups but without high correlations between all groups:It is probably likely that biomarkers of healthy individuals usually have a relatively low correlation. On average,
their values are within a usual "normal" range. In this case, one biomarker tends to be in the upper normal range and another biomarker in the lower normal range. However, individually it can also be exactly the opposite, so that the correlation between healthy individuals would be rather low. Therefore, thevalues of healthy people
could be simulated without any special artificially generated correlations.In the case of a disease, however, a biological system is brought out of balance in a certain way and must react to it.
For example, this reaction can then happen in a coordinated manner involving several biomarkers,
or corresponding cascades (e.g. pathways) can be activated or blocked. This can result in arather stronger
correlation of biomarkers in patients suffering from a disease. To simulate these intra-class correlations,
a class is divided into a given number of groups with high internal correlation
(the respective strength can be defined).Pseudo-classesOne option for an element of the generated data set is a pseudo-class. For example, this could be a
patient's height or gender, which are not related to a specific disease.The generated pseudo-class contains the same number of classes with identical distributions as the artificial biomarkers.
But after the generation of the individual classes, all samples (rows) are randomly shuffled.
Finally, combining the shuffled data with the original, unshuffled class labels, the pseudo-class no longer
has a valid association with any class label. Consequently, no element of the pseudo-class should be
recognized as relevant by a feature selection algorithm.Random FeaturesThe artificial biomarkers and, if applicable, the optional pseudo-classes can be combined with any number
of random features. Varying the number of random features can be used, for example, to analyze random effects
that occur in small sample sizes with a very large number of features.InstallationThe artificial-data-generator is available atthe Python Package Index (PyPI).
It can be installed with pip:$pipinstallartificial-data-generatorLicensingCopyright (c) 2022 Sigrun May, Helmholtz-Zentrum für Infektionsforschung GmbH (HZI)Copyright (c) 2022 Sigrun May, Ostfalia Hochschule für angewandte WissenschaftenLicensed under theMIT License(the "License"); you may not use this file except in compliance with the License.
You may obtain a copy of the License by reviewing the fileLICENSEin the repository.