Package: celebalucid
CelebA & LucidThis package is a quick access tool of a research project of Alfred Renyi Institute of Mathematics. In our project we use lucid visualizations to get a better understanding how transfer learning manipulates a neural network. As a baseline we took Clarity's InceptionV1 network trained on imagenet and we further trained the network on the CelebA dataset.In our project we also revisit and rethink the comparison of activations and features.NetworksWe've trained numerous neural networks. At the first stage, we trained three: one learning 100 epochs with SGD optimizer, one learning 10 epochs with Adam and one learning 100 epochs with Adam. We reference these trainings later assgd, adam and adam_overfit, while the baseline will be referenced asimagenet. Then, we took a permuation of some sampling parameters. We defined 3 different gradient noise (the order of batches the neural network is fed) and 3+1 networks. The 3 networks were initialized with 3 different, randomly sampled weights, while the +1 is the network which were pretrained on imagenet.GradNoise1 (gn1)GradNoise2 (gn2)GradNoise3 (gn3)Imagenet (img)☑☑☑InitNoise1 (in1)☑☑☑InitNoise2 (in2)☑☑☑InitNoise3 (in3)☑☑☑Details of the trainings:NameEpochsTrain lossTrain AccVal lossVal accsgd1000.1692.7%0.1891.9%adam100.1692.9%0.1891.9%adam_overfit1000.0199.3%0.9390.8%img-gn1200.1194.90%0.2191.63%img-gn2200.1194.89%0.2191.60%img-gn3200.1194.88%0.2191.49%in1-gn1200.1792.33%0.1891.70%in1-gn2200.1792.32%0.1991.56%in1-gn3200.1792.29%0.1991.53%in2-gn1200.1792.36%0.1891.73%in2-gn2200.1792.35%0.1991.60%in2-gn3200.1792.33%0.1991.56%in3-gn1200.1792.32%0.1891.69%in3-gn2200.1792.32%0.1991.59%in3-gn3200.1792.32%0.1991.60%SetupRequirementsPython 3.6+InstallpipinstallcelebalucidUsageLoadFirst, load either of the networks asfromcelebalucidimportload_model# Choose from# 'imagenet', 'sgd', 'adam', 'adam_overfit'# 'img-gn1', 'img-gn2', 'img-gn3'# 'in1-gn1', 'in1-gn2', 'in1-gn3'# 'in2-gn1', 'in2-gn2', 'in2-gn3'# 'in3-gn1', 'in3-gn2', 'in3-gn3'model=load_model('imagenet')Change weightsIn order to avoid the redefinition of models and unnecessary memory usage, you can update the state dictionary of a model.model.switch_to('sgd')Layer informationGet accessable layers asmodel.layer_infoCalling this you will see the list of available layers along with the available number of channels (neurons).Out:[['conv2d0',64],['conv2d1',64],['conv2d2',192],['mixed3a_1x1',64],['mixed3a_3x3_bottleneck',96],...]LucidVisualize a neuron with lucid as# Default args: size=224, thresholds=[512], progress=Falseimg=model.lucid('mixed5a_5x5:7',progress=True)plt.imshow(img)WeightsYou can access the weights of a network as# Method 1weights=model.weights.mixed3a_3x3_bottleneck.wbias=model.weights.mixed3a_3x3_bottleneck.b# Method 2weights=model.weights['mixed3a_3x3_bottleneck'].wbias=model.weights['mixed3a_3x3_bottleneck'].bActivationsIn order to see activations, you first need to download the test data (~300MB)fromcelebalucidimportbuild_generator# The test data will be downloaded to this folderDOWNLOAD_DIR='res/images'# Choose dataset 'celeba' or 'imagenet'DATASET='imagenet'# This is a torch.utils.data.DataLoader object# If you already have the dataset downloaded# to this folder, it will skip the downloadgenerator=build_generator(DOWNLOAD_DIR,DATASET)Now, utilize the data generator to load images from disk, feed the input with the neural network and then you're free to analyse the activations (aka. neurons.)forx,yingenerator:model.stream(x)# Now access neurons the same way as on weightssome_operations(model.neurons.conv2d0)CKA similarityThe code provides a centered kernel alignment (CKA) comparison between models. In each epoch, the script runs the models onn_iters * batch_sizeimages and then calculates the CKA value for a given layer. It does the same for each epoch and it returns with the mean cka over the epochs.Define your CKA analyser asfromcelebalucidimportCKA# Args: working_dir, dataset, models_to_compare# Default kwargs: n_epochs=10, n_iters=64, batch_size=32cka=CKA('res/images','celeba',['imagenet','sgd'])Then run your comparison on a specific layer# Optional: set verbose=False for no progress barcka('mixed4a_3x3')Out:0.95517987If you want to choose models, instead of creating a new CKA object set new models in order to be GPU RAM efficient:cka.set_models(['imagenet','adam'])Weight manipulationYou have the possibility to override a network's weights with constant or another model's weights.imagenet=load_model('imagenet')sgd=load_model('sgd')# Change an entire layer to imagenetsgd.set_weights('mixed4a_3x3',imagenet)# Change a single neuronsgd.set_weights('mixed4a_3x3:0',imagenet)# Change to a constantsgd.set_weights('mixed4a_3x3',0)It's good to know that by default both weights an biases change. If you know you need only one of these, you can save a bit of computation.# Default behavioursgd.set_weights('mixed4a_3x3',0,mode='both')# Change weights onlysgd.set_weights('mixed4a_3x3',0,mode='weight')# Change bias onlysgd.set_weights('mixed4a_3x3',0,mode='bias')Creditshttps://github.com/tensorflow/lucid
https://github.com/greentfrapp/lucentReferences@inproceedings{pmlr-v97-kornblith19a,
  title = {Similarity of Neural Network Representations Revisited},
  author = {Kornblith, Simon and Norouzi, Mohammad and Lee, Honglak and Hinton, Geoffrey},
  booktitle = {Proceedings of the 36th International Conference on Machine Learning},
  pages = {3519--3529},
  year = {2019},
  volume = {97},
  month = {09--15 Jun},
  publisher = {PMLR}
}