Package: maxjoshua
maxjoshuaFeature selection for hard voting classifier and NN sparse weight
initialization.PrefaceI am naming this software package in memory of my late nephew Max Joshua
Hamster (* 2005 to † June 18, 2022).UsageForward Selection for Hard Voting ClassifierLoad toy data set and convert features to binary.fromsklearn.datasetsimportload_breast_cancerfromsklearn.preprocessingimportscaleX=scale(load_breast_cancer().data,axis=0)>0# convert to binary featuresy=load_breast_cancer().targetSelect binary features. Each row in theresultslist contains then_selectcolumn indices ofX, the notice if the binary features
were negated, and the sum of absolute MCC correlation coeffcients
between the selected features.importmaxjoshuaasmhidx,neg,rho,results=mh.binsel(X,y,preselect=0.8,oob_score=True,subsample=0.5,n_select=5,unique=True,n_draws=100,random_state=42)Algorithm. The task is to select e.g.n_selectfeatures from a
pool of many features. These features might be the prediction of binary
classifiers. The selected features are then combined into one
hard-voting classifier.A voting classifier should have the following propertieseach voter (a binary feature) should be highly correlated to the
target variablethe selected features should be uncorrelated.The algorithm works as followsGenerate multiple correlation matrices by bootstrapping. This
includescorr(X_i, X_j)as well ascorr(Y, X_i)computation.
Also store the oob samples for evaluation.For each correlation matrix do …Preselect thei*with the highestabs(corr(Y, X_i))estimates (e.g. pick then_pre=?highest absolute
correlations)Slice a correlation matrixcorr(X_i*, X_j*)and find the least
correlated combination ofn_selectfeatures. (see`korr.mincorr<https://github.com/kmedian/korr/blob/master/korr/mincorr.py>`__)Compute the out-of-bag (OOB) performance (see step 1) of the
hard-voter with the selectedn_select=?featuresSelect the feature combination with the best OOB performance as final
model.Forward Selection for Linear RegressionLoad toy dataset.fromsklearn.preprocessingimportscalefromsklearn.datasetsimportfetch_california_housinghousing=fetch_california_housing()X=scale(housing["data"],axis=0)y=scale(housing["target"])Select real-numbered features. Each row in theresultslist contains
then_selectcolumn indices ofX, the ridge regression
coefficentsbetaand the RMSEloss. Warning! Please note that
the featuresXand the targetymust be scaled becausemh.fltseluses an L2-penalty onbetacoefficients, and doesn’t
used an intercept term to shifty.importmaxjoshuaasmhfromsklearn.preprocessingimportscaleidx,beta,loss,results=mh.fltsel(scale(X),scale(y),preselect=0.8,oob_score=True,subsample=0.5,n_select=5,unique=True,n_draws=100,random_state=42,l2=0.01)Initialize Sparse NN LayerThe idea is to runmh.fltselto generate an ensemble of linear
models, and combine them in a sparse linear neural network layer, i.e.,
the number of output neurons is the ensemble size. In case of small
datasets, the sparse NN layer is non-trainable because because each
submodel was already estimated and selected with two-way data splits inmh.fltsel(seeoob_scoresandsubsample). The sparse NN
layers basically produces submodel predictions for meta model in the
next layer, i.e., a simple dense linear layer. The inputs of the sparse
NN layer must be normalized for which a layer normalization layers is
trained.importmaxjoshuaasmhimporttensorflowastfimportsklearn.preprocessing# create toy datasetimportsklearn.datasetsX,y=sklearn.datasets.make_regression(n_samples=1000,n_features=100,n_informative=20,n_targets=3)# feature selection# - always scale the inputs and targets -indices,values,num_in,num_out=mh.pretrain_submodels(sklearn.preprocessing.scale(X),sklearn.preprocessing.scale(y),num_out=64,n_select=3)# specify the modelmodel=tf.keras.models.Sequential([# sub-modelsmh.SparseLayerAsEnsemble(num_in=num_in,num_out=num_out,sp_indices=indices,sp_values=values,sp_trainable=False,norm_trainable=True,),# meta modeltf.keras.layers.Dense(units=3,use_bias=False,# kernel_constraint=tf.keras.constraints.NonNeg()),# scale upmh.InverseTransformer(units=3,init_bias=y.mean(),init_scale=y.std())])model.compile(optimizer=tf.keras.optimizers.Adam(learning_rate=3e-4,beta_1=.9,beta_2=.999,epsilon=1e-7,amsgrad=True),loss='mean_squared_error')# trainhistory=model.fit(X,y,epochs=3)AppendixInstallationThemaxjoshuagit repois
available asPyPi packagepipinstallmaxjoshuaInstall a virtual environmentpython3.7-mvenv.venvsource.venv/bin/activatepipinstall--upgradepippipinstall-rrequirements.txtpipinstall-rrequirements-dev.txtpipinstall-rrequirements-demo.txt(If your git repo is stored in a folder with whitespaces, then don’t use
the subfolder.venv. Use an absolute path without whitespaces.)Python commandsJupyter for the examples:jupyter labCheck syntax:flake8--ignore=F401--exclude=$(grep-v'^#'.gitignore | xargs | sed-e's//,/g')Run Unit Tests:pytestPublishpandocREADME.md--frommarkdown--torst-s-oREADME.rstpythonsetup.pysdisttwineupload-rpypidist/*Clean upfind . -type f -name "*.pyc" | xargs rm
find . -type d -name "__pycache__" | xargs rm -r
rm -r .venvSupportPleaseopen an issuefor support.ContributingPlease contribute usingGithub
Flow. Create a branch,
add commits, andopen a pull
request.