Package: genbase
Generation base dependencyBase functions, generation functions and generic wrappers.© Marcel Robeer, 2021Module overviewModuleDescriptiongenbaseReadable data representations and meta information class.genbase.dataWrapper functions for working with data.genbase.decoratorBase support for decorators.genbase.internationalizationi18ninternationalization.genbase.mixinMixins for seeding (reproducibility) and state machines.genbase.modelWrapper functions for working with machine learning models.genbase.uiExtensible user interfaces (UIs) forgenbasedependencies.InstallationMethodInstructionspipInstall fromPyPIviapip3 install genbase.LocalClone this repository and install viapip3 install -e .or locally runpython3 setup.py install.Releasesgenbaseis officially released throughPyPI.SeeCHANGELOG.mdfor a full overview of the changes for each version.Packages usinggenbaseTheexplaboxaims to support data scientists and machine learning (ML) engineers in explaining, testing and documenting AI/ML models, developed in-house or acquired externally. The explabox turns youringestibles(AI/ML model and/or dataset) intodigestibles(statistics, explanations or sensitivity insights)! Thetext_explainabilitypackage is available throughPyPIand fully documented athttps://explabox.readthedocs.io/.text_explainabilityprovides ageneric architecturefrom which well-known state-of-the-art explainability approaches for text can be composed. This modular architecture allows components to be swapped out and combined, to quicklydevelop new types of explainability approachesfor (natural language) text, or toimprove a plethora of approachesby improving a single module. Thetext_explainabilitypackage is available throughPyPIand fully documented athttps://text-explainability.readthedocs.io/.text_explainabilitycan be extended to also performsensitivity testing, checking for machine learning model safety, robustness and fairness. Thetext_sensitivitypackage is available throughPyPIand fully documented athttps://text-sensitivity.readthedocs.io/.APIgenbaseReadable data representations and meta information class.ClassDescriptionReadableEnsure that a class has a readable representation.ConfigurableAdds working with configs (.from_config(),from_json(),from_yaml(), ...,read_json(), ...,to_yaml()) to a class.MetaInfoAddstype,subtype,callargsand other meta descriptors to a class (subclass ofConfigurable).silence_tqdmSilence output oftqdmin a module.Examples:>>>fromgenbaseimportMetaInfo>>>classReturnCls(MetaInfo):...def__init__(self,value,**kwargs):...super().__init__(self,...type='special_test',...subtype='special',...**kwargs)...self.value=value......@property...defcontent(self):...return{'value':self.value}>>>obj=ReturnCls(value=5)>>>obj.to_config(){'META':{'type':'special_test','subtype':'special'},'CONTENT':{'value':5}}Silence the output oftqdmin awithstatement.>>>importinstancelib>>>fromgenbaseimportsilence_tqdm>>>withsilence_tqdm(instancelib):...model.predict(instances)genbase.dataWrapper functions for working with data.FunctionDescriptionimport_data()Import dataset into aninstancelib.Environment(containing instances and ground-truth labels).train_test_split()Split a dataset into training and test data.Examples:
Import from an online .csv file for theBBC News datasetwith data in the 'text' column and labels in 'category':>>>fromgenbaseimportimport_data>>>import_data('https://storage.googleapis.com/dataset-uploader/bbc/bbc-text.csv',...data_cols='text',label_cols='category')TextEnvironment()Convert a pandas DataFrame to instancelib Environment:>>>fromgenbaseimportimport_data>>>importpandasaspd>>>df=pd.read_csv('./Downloads/bbc-text.csv')>>>import_data(df,data_cols=['text'],label_cols=['category'])TextEnvironment()Download a .zip file of theDrugs.com review datasetand convert each file in the ZIP to an instancelib Environment:>>>fromgenbaseimportimport_data>>>import_data('https://archive.ics.uci.edu/ml/machine-learning-databases/00462/drugsCom_raw.zip',...data_cols='review',label_cols='rating')TextEnvironment(named_providers=['drugsComTest_raw.tsv','drugsComTrain_raw.tsv'])Convert a huggingface Dataset (SST2 in Glue) to an instancelib Environment:>>>fromgenbaseimportimport_data>>>fromdatasetsimportload_dataset>>>import_data(load_dataset('glue','sst2'),data_cols='sentence',label_cols='label')TextEnvironment(named_providers=['test','train','validation'])genbase.decoratorBase support for decorators.DecoratorDescription@add_callargsDecorator that passes__callargs__to a function if available. Useful in conjunction withMetaInfo.Example:>>>fromgenbaseimportMetaInfo,add_callargs>>>classReturnCls(MetaInfo):...def__init__(self,value,callargs=None,**kwargs):...super().__init__(self,...type='special_test',...subtype='special',...callargs=callargs,...**kwargs)...self.value=value......@property...defcontent(self):...return{'value':self.value}>>>@add_callargs...defexample_fn(x:int,y:int,z:int=5,t='str',**kwargs):...callargs=kwargs.pop('__callargs__',None)...returnReturnCls(value=x+y+z,callargs=callargs)>>>example_fn(x=1,y=2).callargs{'x':1,'y':2,'z':5,'t':'str'}genbase.internationalizationi18ninternationalization.FunctionDescriptionget_locale()Get current locale.set_locale()Set current locale .translate_list()Get a list based onlocale, as defined in the './locale' folder.translate_string()Get a string based onlocale, as defined in the './locale' folder.Example:>>>fromgenbase.internationalizationimportset_locale,translate_list>>>set_locale('en')>>>translate_list('stopwords')['a','an','the']>>>set_locale('nl')>>>translate_list('stopwords')['de','het','een']genbase.mixinMixins for seeding (reproducibility) and state machines.ClassDescriptionSeedMixinAdds working with ._seed and ._original_seed for reproducibility.CaseMixinAdds working with title-, sentence-, upper- and lowercase for random data generation.Example:>>>fromgenbase.mixinimportSeedMixin>>>classRandomCls(SeedMixin):...def__init__(self,seed:int=0):...self._seed=self._original_seed=seed>>>rc=RandomCls(seed=10)>>>rc.seed10>>>rc._seed+=20>>>rc.seed30>>>rc._original_seed10genbase.modelWrapper functions for working with machine learning models.FunctionDescriptionimport_data()Import a model with instancelib or instancelib-onnx.Examples:
Make a scikit-learn text classifier and train it on SST2>>>fromgenbaseimportimport_data,import_model>>>fromdatasetsimportload_dataset>>>ds=import_data(load_dataset('glue','sst2'),data_cols='sentence',label_cols='label')>>>fromsklearn.pipelineimportPipeline>>>fromsklearn.naive_bayesimportMultinomialNB>>>fromsklearn.feature_extraction.textimportTfidfVectorizer>>>pipeline=Pipeline([('tfidf',TfidfVectorizer()),...('clf',MultinomialNB())])>>>import_model(pipeline,ds,train='train')SklearnDataClassifier()Load apretrained ONNX modelwith labels 'Bedrijfsnieuws', 'Games' and 'Smartphones'>>>fromgenbaseimportimport_model>>>import_model('data-model.onnx',label_map={0:'Bedrijfsnieuws',1:'Games',2:'Smartphones'})SklearnDataClassifier()genbase.uiExtensible user interfaces (UIs) forgenbasedependencies.FunctionDescriptionget_color()Get color from amatplotlibcolorscale.plot.matplotlib_available()Check ifmatplotlibis installed.plot.plotly_available()Check ifplotlyis installed.notebook.format_label()Format label as title.notebook.format_instances()Format multipleinstancelibinstances.notebook.is_colab()Check if environment is Google Colab.notebook.is_interactive()Check if the environment is interactive (Jupyter Notebook).ClassDescriptionplot.ExpressPlotPlotter forplotly.express.notebook.RenderBase class for rendering configs (configuration dictionaries).Example:>>>fromgenbase.ui.notebookimportRender>>>classCustomRender(Render):...def__init__(self,*configs):...super().__init__(*configs)...self.default_title='My Custom Explanation'...self.main_color='#ff00000'...self.package_link='https://git.io/text_explainability'......defformat_title(self,title:str,h:str='h1',**renderargs)->str:...returnf'<{h}style="color: red;">{title}</{h}>'......defrender_content(self,meta:dict,content:dict,**renderargs):...type=meta['type']if'type'inmetaelse''...returntype.replace(' ').title()if'explanation'intypeelsetype>>>fromgenbaseimportMetaInfo>>>NiceCls(MetaInfo):...def__init__(self,**kwargs):...super().__init__(renderer=CustomRenderer,**kwargs)