Package: pymfe
pymfe: Python Meta-Feature ExtractorThe pymfe (pythonmeta-featureextractor) provides a
comprehensive set of meta-features implemented in python. The package brings
cutting edge meta-features, following recent literature propose. The pymfe
architecture was thought to systematically make the extraction, which can
produce a robust set of meta-features. Moreover, pymfe follows recent
meta-feature formalization aiming to make MtL reproducible.Here,  you can use different measures and summary functions, setting their
hyperparameters, and also measuring automatically the elapsed time. Moreover,
you can extract meta-features from specific models, or even extract
meta-features with confidence intervals using bootstrap. There are a lot of
other interesting features and you can see more about it looking at the
documentation.Meta-featureIn the Meta-learning (MtL) literature, meta-features are measures used to
characterize data sets and/or their relations with algorithm bias."Meta-learning is the study of principled methods that exploit meta-knowledge to obtain efficient models and solutions by adapting the machine learning and data mining process." - (Brazdil et al. (2008))Meta-features are used in MtL and AutoML tasks in general, to
represent/understand a dataset,  to understanding a learning bias, to create
machine learning (or data mining) recommendations systems, and to create
surrogates models, to name a few.Pinto et al. (2016)andRivolli et al. (2018)defined a meta-feature as
follows. Let $D \in \mathcal{D}$ be a dataset, $m\colon \mathcal{D} \to \mathbb{R}^{k'}$
be a characterization measure, and $\sigma\colon \mathbb{R}^{k'} \to \mathbb{R}^{k}$
be a summarization function. Both $m$ and $\sigma$ have also hyperparameters associated,
$h_m$ and $h_\sigma$ respectively. Thus, a meta-feature $f\colon \mathcal{D} \to \mathbb{R}^{k}$
for a given dataset $D$ is$$
f\big(D\big) = \sigma\big(m(D,h_m), h_\sigma\big).
$$The measure $m$ can extract more than one value from each data set, i.e.,
$k'$ can vary according to $D$, which can be mapped to a vector of fixed length
$k$ using a summarization function $\sigma$.In this package, We provided the following meta-features groups:General: General information related to the dataset, also known as simple
measures, such as the number of instances, attributes and classes;Statistical: Standard statistical measures to describe the numerical
properties of data distribution;Information-theoretic: Particularly appropriate to describe discrete
(categorical) attributes and their relationship with the classes;Model-based: Measures designed to extract characteristics from simple
machine learning models;Landmarking: Performance of simple and efficient learning algorithms.Relative Landmarking: Relative performance of simple and efficient
learning algorithms;Subsampling Landmarking: Performance of simple and efficient learning
algorithms from a subsample of the dataset;Clustering: Clustering measures extract information about dataset based
on external validation indexes;Concept: Estimate the variability of class labels among examples and the
examples density;Itemset: Compute the correlation between binary attributes; andComplexity: Estimate the difficulty in separating the data points into
their expected classes.In the pymfe package, you can use different measures and summary functions,
setting their hyperparameters, and automatically measure the elapsed time.
Moreover,  you can extract meta-features from specific models, or even obtain
meta-features with confidence intervals using bootstrap.
There are many other exciting features. You can see more about it looking at
thedocumentation.DependenciesThe mainpymferequirement is:Python (>= 3.6)InstallationThe installation process is similar to other packages available on pip:pipinstall-UpymfeIt is possible to install the development version using:pipinstall-Ugit+https://github.com/ealcobaca/pymfeorgitclonehttps://github.com/ealcobaca/pymfe.gitcdpymfe
pythonsetup.pyinstallExample of useThe simplest way to extract meta-features is by instantiating theMFEclass.
It computes five meta-features groups by default using mean and standard
deviation as summary functions:  General, Statistical, Information-theoretic,
Model-based, and Landmarking. Thefitmethod can be called by passing theXandy. Then theextractmethod is used to extract the related measures.
A simple example usingpymfefor supervised tasks is given next:# Load a datasetfromsklearn.datasetsimportload_irisfrompymfe.mfeimportMFEdata=load_iris()y=data.targetX=data.data# Extract default measuresmfe=MFE()mfe.fit(X,y)ft=mfe.extract()print(ft)# Extract general, statistical and information-theoretic measuresmfe=MFE(groups=["general","statistical","info-theory"])mfe.fit(X,y)ft=mfe.extract()print(ft)# Extract all available measuresmfe=MFE(groups="all")mfe.fit(X,y)ft=mfe.extract()print(ft)You can simply omit the target attribute for unsupervised tasks while fitting
the data into theMFEmodel. Thepymfepackage automatically finds and
extracts only the metafeatures suitable for this type of task. Examples are
given next:# Load a datasetfromsklearn.datasetsimportload_irisfrompymfe.mfeimportMFEdata=load_iris()y=data.targetX=data.data# Extract default unsupervised measuresmfe=MFE()mfe.fit(X)ft=mfe.extract()print(ft)# Extract all available unsupervised measuresmfe=MFE(groups="all")mfe.fit(X)ft=mfe.extract()print(ft)Several measures return more than one value. To aggregate the returned values,
summarization function can be used. This method can computemin,max,mean,median,kurtosis,standard deviation, among others. The default
methods are themeanand thesd. Next, it is possible to see an example of
the use of this method:## Extract default measures using min, median and maxmfe=MFE(summary=["min","median","max"])mfe.fit(X,y)ft=mfe.extract()print(ft)## Extract default measures using quantilemfe=MFE(summary=["quantiles"])mfe.fit(X,y)ft=mfe.extract()print(ft)You can easily list all available metafeature groups, metafeatures, summary
methods and metafeatures filtered by groups of interest:frompymfe.mfeimportMFE# Check all available meta-feature groups in the packageprint(MFE.valid_groups())# Check all available meta-features in the packageprint(MFE.valid_metafeatures())# Check available meta-features filtering by groups of interestprint(MFE.valid_metafeatures(groups=["general","statistical","info-theory"]))# Check all available summary functions in the packageprint(MFE.valid_summary())It is possible to pass custom arguments to every metafeature usingMFEextractmethod kwargs. The keywords must be the target metafeature name, and
the value must be a dictionary in the format {argument:value}, i.e., each
key in the dictionary is a target argument with its respective value. In the
example below, the extraction of metafeaturesminandmaxhappens as
usual, but the metafeaturessd,nr_normandnr_cor_attrwill receive user
custom argument values, which will interfere in each metafeature result.# Extract measures with custom user argumentsmfe=MFE(features=["sd","nr_norm","nr_cor_attr","min","max"])mfe.fit(X,y)ft=mfe.extract(sd={"ddof":0},nr_norm={"method":"all","failure":"hard","threshold":0.025},nr_cor_attr={"threshold":0.6},)print(ft)If you want to extract metafeatures from a pre-fitted machine learning model
(fromsklearn package), you can use theextract_from_modelmethod without
needing to use the training data:importsklearn.treefromsklearn.datasetsimportload_irisfrompymfe.mfeimportMFE# Extract from modeliris=load_iris()model=sklearn.tree.DecisionTreeClassifier().fit(iris.data,iris.target)extractor=MFE()ft=extractor.extract_from_model(model)print(ft)# Extract specific metafeatures from modelextractor=MFE(features=["tree_shape","nodes_repeated"],summary="histogram")ft=extractor.extract_from_model(model,arguments_fit={"verbose":1},arguments_extract={"verbose":1,"histogram":{"bins":5}})print(ft)You can also extract your metafeatures with confidence intervals using
bootstrap. Keep in mind that this method extracts each metafeature several
times, and may be very expensive depending mainly on your data and the number
of metafeature extract methods called.# Extract metafeatures with confidence intervalmfe=MFE(features=["mean","nr_cor_attr","sd","max"])mfe.fit(X,y)ft=mfe.extract_with_confidence(sample_num=256,confidence=0.99,verbose=1,)print(ft)DocumentationWe write a greatDocumentationto guide you on how to use the pymfe library.
You can find in the documentation interesting pages like:Getting startedAPI documentationExamplesNews about pymfeDeveloper notesWe are glad to accept any contributions, please checkContributingand theDocumentation.To submit bugs and feature requests, report atproject issues.LicenseThis project is licensed under the MIT License - see theLicensefile for
details.Cite UsIf you use thepymfein scientific publication, we would appreciate citations
to the following paper:Edesio Alcobaça, Felipe Siqueira, Adriano Rivolli, Luís P. F. Garcia,
Jefferson T. Oliva, & André C. P. L. F. de Carvalho (2020).
MFE: Towards reproducible meta-feature extraction.
Journal of Machine Learning Research, 21(111), 1-5.You can also use the bibtex format:@article{JMLR:v21:19-348,author={Edesio Alcobaça andFelipe Siqueira andAdriano Rivolli andLuís P. F. Garcia andJefferson T. Oliva andAndré C. P. L. F. de Carvalho},title={MFE: Towards reproducible meta-feature extraction},journal={Journal of Machine Learning Research},year={2020},volume={21},number={111},pages={1-5},url={http://jmlr.org/papers/v21/19-348.html}}AcknowledgmentsWe would like to thank everyContributorthat directly or indirectly has make this project to happen. Thank you all.ReferencesBrazdil, P., Carrier, C. G., Soares, C., & Vilalta, R. (2008). Metalearning:
Applications to data mining. Springer Science
and Business Media.Pinto, F., Soares, C., & Mendes-Moreira, J. (2016, April). Towards automatic
generation of metafeatures. In Pacific-Asia Conference on Knowledge Discovery
and Data Mining (pp. 215-226). Springer,
Cham.Rivolli, A., Garcia, L. P. F., Soares, C., Vanschoren, J., and de Carvalho,
A. C. P. L. F. (2018). Characterizing classification datasets: a study of
meta-features for meta-learning.
arXiv:1808.10406.