Package: datasieve
DataSieveDataSieve is very similar to the SKlearn Pipeline in that it:fits an arbitrary series of transformations to an array Xtransforms subsequent arrays of the same dimension according to the fit from the original Xinverse transforms arrays by inverting the series of transformationsThis means that it follows the SKLearn API very closely, and in fact users can use SKLearn transforms directly without making any modifications.The maindifferenceis that DataSieve allows for the manipulation of the y and sample_weight arrays in addition to the X array. This is useful if you find yourself wishing to use the SKLearn pipeline for:removing outliers across your X, y, and sample_weights arrays according to simple or complex criteriaremove feature columns based on arbitrary criteria (e.g. low variance features)change feature column names at certain transformations (e.g. PCA)needing outlier classification without removal (seeoulier_check)passing dynamic parameters to individual transforms of the pipelinepassing dataframes/arrays without worrying about converting to arrays and maintaining the proper feature columnscustomizing backend for parallelization (e.g. Dask, Ray, loky, etc.)These improved flexibilities allow for more customized/creative transformations. For example, the includedDBSCANhas automated parameter fitting and outlier removal based on clustering.UsageThe user builds the pipeline similarly to SKLearn, and can even use SKLearn transforms directly with theSKLearnWrapper:fromdatasieve.pipelineimportPipelineimportdatasieve.transformsasdstfromsklearn.preprocessingimportMinMaxScalerfeature_pipeline=Pipeline([("detect_constants",dst.VarianceThreshold(threshold=0)),("pre_svm_scaler",dst.SKlearnWrapper(MinMaxScaler(feature_range=(-1,1)))),("svm",dst.SVMOutlierExtractor()),("pca",dst.PCA(n_components=0.95)),("post_pca_scaler",dst.SKlearnWrapper(MinMaxScaler(feature_range=(-1,1))))])Once the pipeline is built, it can be fit and transformed similar to a SKLearn pipeline:X,y,sample_weight=feature_pipeline.fit_transform(X,y,sample_weight)This pipeline demonstrates the various components ofDataSievewhich are missing from SKLearn's pipeline. A dataframeX(if desired, else users can input a numpy array without column names) is input with its associatedyandsample_weightarrays/vectors (these are also optional). TheVarianceThresholdwill first detect and remove any features that have zero variance in X, theSVMOutlierExtractorwill fitSGDOneClassSVMtoXand then remove the detected outliers inX, while also propagating those row removals fromyandsample_weight. Finally, thePCAwill be fit to the remainingXarray with the features count changing and getting renamed. The returnedXdataframe will have the correctly named column features/count, and equal row counts across theX,y, andsample_weightarrays.Next, thefeature_pipelinecan then be used to transform other datasets with the same input feature dimension:Xprime,_,_=feature_pipeline.transform(X2)Finally, similar to SKLearn's pipeline, thefeature_pipelinecan be used to inverse_transform the arrayXprimearray that has the same dimensions as the returnedX2/Xarray from the pipeline:X2,_,_=feature_pipeline.inverse_transform(Xprime)Creating a custom transformAn example would be someone who wants to useSGDOneClassSVMto detect and remove outliers from their data set before training:classSVMOutlierExtractor(BaseTransform):"""A subclass of the SKLearn SGDOneClassSVM that adds a transform() methodfor removing detected outliers from X (as well as the associated y andsample_weight if they are also furnished."""def__init__(self,**kwargs):self._skl=SGDOneClassSVM(**kwargs)deffit_transform(self,X,y=None,sample_weight=None,feature_list=None,**kwargs):self.fit(X,y,sample_weight=sample_weight)returnself.transform(X,y,sample_weight,feature_list)deffit(self,X,y=None,sample_weight=None,feature_list=None,**kwargs):self._skl.fit(X,y=y,sample_weight=sample_weight)returnX,y,sample_weight,feature_listdeftransform(self,X,y=None,sample_weight=None,feature_list=None,outlier_check=False,**kwargs):y_pred=self._skl.predict(X)y_pred=np.where(y_pred==-1,0,y_pred)ifnotoutlier_check:X,y,sample_weight=remove_outliers(X,y,sample_weight,y_pred)num_tossed=len(y_pred)-len(X)ifnum_tossed>0:logger.info(f"SVM detected{num_tossed}data points ""as outliers.")else:y+=y_predy-=1returnX,y,sample_weight,feature_listdefinverse_transform(self,X,y=None,sample_weight=None,feature_list=None,**kwargs):"""Unused"""returnX,y,sample_weight,feature_listAs shown here, thefit()method is actually identical to the SKLearnfit()method, but thetransform()removes data points from X, y, and sample_weight for any outliers detected in theXarray.Data removalThe commandfeature_pipeline.fit_transform(X, y, sample_weight)fits each pipeline step toX, and transformsXaccording to each step'stransform()method. In some cases, this will not affectyorsample_weight. For example,MinMaxScalersimply scalesXand saves the normalization information.  Meanwhile, in theSVMOutlierExtractor,.fit()will fit an SVM toXand.transform()will remove any detected outliers fromX. TypicalScikit-Learnpipelines do not remove those data points fromyandsample_weight. Luckily, thePipelinetakes care of the "associated removal" of the same outlier data points fromyandsample_weight.Feature modificationAnother feature is demonstrated in thePCA, which fits a PCA transform toXand then transformsXto principal components. This dimensionality reduction means that the features are no longer the same, instead they are nowPC1,PC2...PCX.Pipelinehandles the feature renaming at that step (which is not a feature available in theScikit-Learnpipeline). Similar toFlowdaptPCA, theVarianceThresholdsubclasses theScikit-LearnVarianceThresholdwhich is geared toward removing features that have a low variance.VarianceThresholdensures that the removed features are properly handled whenXpasses through this part of the pipeline.Outlier checkingDataSieve also allows users to fit a pipeline that can be used to flag outliers in datawithoutremoving them from the dataset. This may be handy in a variety of cases where keeping the data point is important but having some indication of which points are outliers is also important. In order to use this functionality, you can take an alreadyfitpipeline and calltransform(X, outlier_check=True)which will return X as well as a vector of 1s and 0s indicating which points are outliers. This is demonstrated in the following example:pipeline=Pipeline([("pre_svm_scaler",transforms.DataSieveMinMaxScaler()),("svm",transforms.SVMOutlierExtractor())])pipeline.fit(X)X,outliers,_=pipeline.transform(X,outlier_check=True)Now X willnothave any of the outlier data points removed, but the vectoroutlierswill be an indication of which points were classified as outliers, where 0 means the point was an outlier and 1 means that the point was an inlier.InstallationThe easiest way to installdatasieveis with:pip install datasievebut you can also clone this repository and install it with:git clone https://github.com/emergentmethods/datasieve.git
cd datasieve
poetry installLicenseCopyright (c) 2023 DataSievePermission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.