Package: thingsvision
:notebook_with_decorative_cover: Table of ContentsAbout the ProjectFunctionalityModel collectionGetting StartedSetting up your environmentBasic usageContributingLicenseCitationContributions:star2: About the Projectthingsvisionis a Python package that let's you easily extract image representations from many state-of-the-art neural networks for computer vision. In a nutshell, you feedthingsvisionwith a directory of images and tell it which neural network you are interested in.thingsvisionwill then give you the  representation of the indicated neural network for each image so that you will end up with one feature map (vector or matrix, depending on the layer) per image. You can use these features for further analyses. We use the wordfeaturesfor short when we mean "image representation".:rotating_light: Note: some function calls mentioned in thepaperhave been deprecated. To use this package successfully, exclusively follow thisREADMEand thedocumentation. :rotating_light:(back to top):mechanical_arm: FunctionalityWiththingsvision, you can:extract features for any imageset from many popular networks.extract features for any imageset from your custom networks.extract features for >26,000 images from theTHINGS image database.optionally turn off the standard center cropping performed by many networks before extracting features.extract features from HDF5 datasets directly (e.g. NSD stimuli)conduct basic Representational Similarity Analysis (RSA) after feature extraction.perform Centered Kernel Alignment (CKA) to compare image features across model-module combinations.(back to top):file_cabinet: Model collectionNeural networks come from different sources. Withthingsvision, you can extract image representations of all models from:torchvisionKerastimmssl(self-supervised learning models)simclr-rn50,mocov2-rn50,jigsaw-rn50,rotnet-rn50,swav-rn50,pirl-rn50barlowtwins-rn50,vicreg-rn50,dino-vit{s/b}{8/16},dino-xcit-{small/medium}-{12/24}-p{8/16},dino-rn50,dinov2-vit-small-p14,dinov2-vit-base-p14,dinov2-vit-large-p14,dinov2-vit-giant-p14OpenCLIPboth originalCLIPvariants (ViT-B/32andRN50)a few custom models (Alexnet, VGG-16, Resnet50, and Inception_v3) trained onEcosetrather than ImageNet  and one Alexnet pretrained on ImageNet and fine-tuned onSalObjSubeach of the manyCORnetversionsHarmonizationmodels from the officialHarmonization repo. The default variant isViT_B16. Other available models areResNet50,VGG16,EfficientNetB0,tiny_ConvNeXT,tiny_MaxViT, andLeViT_small.DreamSimmodels from the officialDreamSim repo. The default variant isopen_clip_vitb32. Other available models areclip_vitb32. See thedocsfor more information.(back to top):running: Getting Started:computer: Setting up your environmentWorking locally.First, create a newconda environmentwith Python version 3.8, 3.9, or 3.10 e.g. by usingconda:$condacreate-nthingsvisionpython=3.9
$condaactivatethingsvisionThen, activate the environment and simply installthingsvisionvia running the followingpipcommand in your terminal.$pipinstall--upgradethingsvision
$pipinstallgit+https://github.com/openai/CLIP.gitIf you want to extract features forharmonized modelsfrom theHarmonization repo, you have to additionally run the followingpipcommand in yourthingsvisionenvironment (FYI: as of now, this seems to be working smoothly on Ubuntu only but not on macOS),$pipinstallgit+https://github.com/serre-lab/Harmonization.git
$pipinstallkeras-cv-attention-models>=1.3.5If you want to extract features forDreamSimfrom theDreamSim repo, you have to additionally run the followingpipcommand in yourthingsvisionenvironment,$pipinstalldreamsim==0.1.2See thedocsfor whichDreamSimmodels are available inthingsvision.Google Colab.Alternatively, you can use Google Colab to play around withthingsvisionby uploading your image data to Google Drive (via directory mounting).
You can find the jupyter notebook usingPyTorchhereand theTensorFlowexamplehere.(back to top):mag: Basic usageCommand Line Interface (CLI)thingsvisionwas designed to simplify feature extraction. If you have some folder of images (e.g.,./images) and want to extract features for each of these images without opening a Jupyter Notebook instance or writing a Python script, it's probably easiest to use our CLI. The interface includes two options,thingsvision show-modelthingsvision extract-featuresExample calls might look as follows:thingsvisionshow-model--model-name"alexnet"--source"torchvision"thingsvisionextract_features--image-root"./data"--model-name"alexnet"--module-name"features.10"--batch-size32--device"cuda"--source"torchvision"--file-format"npy"--out-path"./features"Seethingsvision show-model -handthingsvision extract-features -hfor a list of all possible arguments. Note that the CLI provides just the basic extraction functionalities but is probably enough for most users that don't want to dive too deep into various models and modules. If you need more fine-grained control over the extraction itself, we recommend to use the python package directly and write your own Python script.Python commandsTo do this start by importing all the necessary components and instantiating athingsvisionextractor. Here we're usingCLIPfrom the official clip repo as the model to extract features from and also load the model to GPU for faster inference,importtorchfromthingsvisionimportget_extractorfromthingsvision.utils.storingimportsave_featuresfromthingsvision.utils.dataimportImageDataset,DataLoadermodel_name='clip'source='custom'device='cuda'iftorch.cuda.is_available()else'cpu'model_parameters={'variant':'ViT-B/32'}extractor=get_extractor(model_name=model_name,source=source,device=device,pretrained=True,model_parameters=model_parameters,)As a next step, create both dataset and dataloader for your images. We assume that all of your images are in a singlerootdirectory which can contain subfolders (e.g., for individual classes). Therefore, we leverage theImageDatasetclass.root='path/to/root/image/directory'# (e.g., './images/)batch_size=32dataset=ImageDataset(root=root,out_path='path/to/features',backend=extractor.get_backend(),# backend framework of modeltransforms=extractor.get_transformations(resize_dim=256,crop_dim=224)# set the input dimensionality to whichever values are required for your pretrained model)batches=DataLoader(dataset=dataset,batch_size=batch_size,backend=extractor.get_backend()# backend framework of model)Now all that is left is to extract the image features and store them on disk! Here we're extracting features from the image encoder module of CLIP (visual), but if you don't know which modules are available for a given model, just callextractor.show_model()to print all the modules.module_name='visual'features=extractor.extract_features(batches=batches,module_name=module_name,flatten_acts=True,output_type="ndarray",# or "tensor" (only applicable to PyTorch models of which CLIP is one!))save_features(features,out_path='path/to/features',file_format='npy')# file_format can be set to "npy", "txt", "mat", "pt", or "hdf5"For more examples on the many models available inthingsvisionand explanations of additional functionality like how to optionally turn off center cropping, how to use HDF5 datasets (e.g. NSD stimuli), how to perform RSA or CKA, or how to easily extract features for theTHINGS image database, please refer to theDocumentation.(back to top):wave: How to contributeIf you come across problems or have suggestions please submit an issue!(back to top):warning: LicenseThis GitHub repository is licensed under the MIT License - see theLICENSE.mdfile for details.(back to top):page_with_curl: CitationIf you use this GitHub repository (or any modules associated with it), please cite ourpaperfor the initial version ofthingsvisionas follows:@article{Muttenthaler_2021,
	author ={Muttenthaler, Lukas and Hebart, Martin N.},
	title ={THINGSvision: A Python Toolbox for Streamlining the Extraction of Activations From Deep Neural Networks},
	journal ={Frontiers in Neuroinformatics},
	volume ={15},
	pages ={45},
	year ={2021},
	url ={https://www.frontiersin.org/article/10.3389/fninf.2021.679838},
	doi ={10.3389/fninf.2021.679838},
	issn ={1662-5196},}(back to top):gem: ContributionsThis library is based on the groundwork laid byLukas MuttenthalerandMartin N. Hebart, who are both still actively involved, but has been extended and refined into its current form with the help of our many contributors,Alex Murphy(software dev.)Florian Mahner(software dev.)Hannes Hansen(software dev.)Johannes Roth(software dev., design, docs)Jonas Dippel(software dev.)Lukas Muttenthaler(software dev., design, docs, general responsibility)Martin N. Hebart(design)Philipp Kaniuth(design, docs)Roman Leipe(sofware dev., docs),sorted alphabetically.This is a joint open-source project between the Max Planck Institute for Human Cognitive and Brain Sciences, Leipzig, and the Machine Learning Group at Technische Universtität Berlin. Correspondence and requests for contributing should be adressed toLukas Muttenthaler. Feel free to contact us if you want to become a contributor or have any suggestions/feedback. For the latter, you could also just post an issue or engange in discussions. We'll try to respond as fast as we can.(back to top)