Package: corelay
CoRelAy – Composing Relevance AnalysisCoRelAy is a tool to compose small-scale (single-machine) analysis pipelines.
Pipelines are designed with a number of steps (Task) with default operations (Processor).
Any step of the pipeline may then be indiviually changed by assigning a new operator (Processor).
Processors have Params which define their operation.CoRelAy was created to quickly implement pipelines to generate analysis data
which can then be visualized using ViRelAy.If you find CoRelAy useful for your research, why not cite our relatedpaper:@article{anders2021software,
      author  = {Anders, Christopher J. and
                 Neumann, David and
                 Samek, Wojciech and
                 Müller, Klaus-Robert and
                 Lapuschkin, Sebastian},
      title   = {Software for Dataset-wide XAI: From Local Explanations to Global Insights with {Zennit}, {CoRelAy}, and {ViRelAy}},
      journal = {CoRR},
      volume  = {abs/2106.13200},
      year    = {2021},
}DocumentationThe latest documentation is hosted atcorelay.readthedocs.io.InstallCoRelAy may be installed using pip with$pipinstallcorelayTo install optional HDBSCAN and UMAP support, use$pipinstallcorelay[umap,hdbscan]UsageExamples to highlight some features ofCoRelAycan be found inexample/.We mainly use HDF5 files to store results. The structure used byViRelAyis documented in theViRelAyrepository atdocs/database_specification.md. An example to create HDF5 files which can be used withViRelAyis
shown inexample/hdf5_structure.pyTo do a full SpRAy analysis which can be visualized withViRelAy, an advanced script can be found inexample/virelay_analysis.py.The following shows the contents ofexample/memoize_spectral_pipeline.py:'''Example using memoization to store (intermediate) results.'''importtimeimporth5pyimportnumpyasnpfromcorelay.baseimportParamfromcorelay.processor.baseimportProcessorfromcorelay.processor.flowimportSequential,Parallelfromcorelay.pipeline.spectralimportSpectralClusteringfromcorelay.processor.clusteringimportKMeansfromcorelay.processor.embeddingimportTSNEEmbedding,EigenDecompositionfromcorelay.io.storageimportHashedHDF5# custom processors can be implemented by defining a function attributeclassFlatten(Processor):deffunction(self,data):returndata.reshape(data.shape[0],np.prod(data.shape[1:]))classSumChannel(Processor):# parameters can be assigned by defining a class-owned Param instanceaxis=Param(int,1)deffunction(self,data):returndata.sum(1)classNormalize(Processor):deffunction(self,data):data=data/data.sum((1,2),keepdims=True)returndatadefmain():np.random.seed(0xDEADBEEF)fpath='test.analysis.h5'withh5py.File(fpath,'a')asfd:# HashedHDF5 is an io-object that stores outputs of Processors based on hashes in hdf5iobj=HashedHDF5(fd.require_group('proc_data'))# generate some exemplary datadata=np.random.normal(size=(64,3,32,32))n_clusters=range(2,20)# SpectralClustering is an Example for a pre-defined Pipelinepipeline=SpectralClustering(# processors, such as EigenDecomposition, can be assigned to pre-defined tasksembedding=EigenDecomposition(n_eigval=8,io=iobj),# flow-based Processors, such as Parallel, can combine multiple Processors# broadcast=True copies the input as many times as there are Processors# broadcast=False instead attempts to match each input to a Processorclustering=Parallel([Parallel([KMeans(n_clusters=k,io=iobj)forkinn_clusters],broadcast=True),# io-objects will be used during computation when supplied to Processors# if a corresponding output value (here identified by hashes) already exists,# the value is not computed again but instead loaded from the io objectTSNEEmbedding(io=iobj)],broadcast=True,is_output=True))# Processors (and Params) can be updated by simply assigning corresponding attributespipeline.preprocessing=Sequential([SumChannel(),Normalize(),Flatten()])start_time=time.perf_counter()# Processors flagged with "is_output=True" will be accumulated in the output# the output will be a tree of tuples, with the same hierachy as the pipeline# (i.e. clusterings here contains a tuple of the k-means outputs)clusterings,tsne=pipeline(data)# since we memoize our results in a hdf5 file, subsequent calls will not compute# the values (for the same inputs), but rather load them from the hdf5 file# try running the script multiple timesduration=time.perf_counter()-start_timeprint(f'Pipeline execution time:{duration:.4f}seconds')if__name__=='__main__':main()