Add Google's Trillson Audio Classification Model
Author: vumichien
Pull request: yes
Labels: name: WIP

# What does this PR do?
Add Google's Trillson Audio Classification Model #17339 

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?
@patrickvonplaten



Comments:
Still WIP (maybe we can link the google colab here if you want @vumichien :-) )

Thanks @patrickvonplaten, This is the current link of colab notebook I am working on https://colab.research.google.com/drive/1lFGDVNgtXXyuvM4J-pBmsPj43LW9VlpQ#scrollTo=t-icrfOL60dT to load all TF weights of trillsson3 into Efficientnetv2bS PyTorch model

Current process:
- Check the output of each layer between the original TF and PT model


Hey @vumichien,

Sorry to have dropped the ball here. I will be off for the two weeks ahead, but will take a look at the PR once I'm back!

Hi @patrickvonplaten,
I am swamped finishing my job this month before the summer holiday. 
I just want to update the current process, I have successfully loaded the TF trillson weight to PyTorch model (you can check in the notebook here https://colab.research.google.com/drive/1lFGDVNgtXXyuvM4J-pBmsPj43LW9VlpQ#scrollTo=M4k3zhZcypUH)
In the next step, I think I will write the script in the correct format of the Transformers library. However if possible, could you describe in detail or give necessary notes for the next step?

Very cool @vumichien ! 
I'd suggest to copy-paste the PyTorch model and preprocessing as defined in your colab to `modeling_trillson_efficient.py` in Transformers and in a first step add a couple slow integration tests to make sure that when refactoring your code the outputs stay correct. 
More specifically, it'd be very useful to add a couple of those tests: https://github.com/huggingface/transformers/blob/e54a1b49aa6268c484625c6374f952f318914743/tests/models/bert/test_modeling_bert.py#L585 (maybe 2 or 3 given that the model is relatively complex)
Once this works we have a safety mechanism that we can always fall back to that ensures the model works correctly.

Then the refactor starts. 
- 1) Move the pre-processing out into a new file `feature_extraction_trillson_efficient.py` and make the model code pretty.
Making the code pretty means that it should adhere to Transformers style (e.g. the model should inherit from a `PretrainedModel` class, the naming should be similar to, e.g. BERT.) You can find more information about this here: https://huggingface.co/docs/transformers/add_new_model#stepbystep-recipe-to-add-a-model-to-transformers
- 2) Then the next step is to improve the weight names and to write a conversion script that automatically converts the old checkpoint names to the new PyTorch ones (Point 6. here: https://huggingface.co/docs/transformers/add_new_model#stepbystep-recipe-to-add-a-model-to-transformers) 
- 3) Once that works it's again a bit more refactoring and then we're good to go #18511 
- 4) If you want I can also allocate you with a GPU to run some fine-tuning experiments

Overall, really amazing work & really sorry that I was so unavailable to help you. But now that the most difficult part is solved I think the rest is easy :-) 
Generally, I'd suggest to just adapt this PR to include the newest version of your model + tests and then I'm more than happy to also directly comment in the PR



Hi @patrickvonplaten, I just upload the new version of Trillsson model, would you mind checking on it?

> Hi @patrickvonplaten, I just upload the new version of Trillsson model, would you mind checking on it?

Very cool! Does it work just like the original model? In this case should we try fine-tuning it on emotion recognition? 

I have checked the embedding outputs, they are the same as the TF model in the original repo. I think we could try fine-tuning with emotion recognition by adding one dense classifier layer on top of embedding outputs. Do you have any suggestions which dataset we could use for fine-tuning? 
Can I leverage the Trainer for the fine-tuning? I don't know whether the current version code of Trillsson_efficient is good enough to use with Trainer.

Hey @vumichien,

That's a very nice idea! Could we try to set up this fine-tuning example for key word spotting with trillson?
https://github.com/huggingface/transformers/tree/main/examples/pytorch/audio-classification#single-gpu

I think you only need to add one head layer for it. 
Let me know if you need access to a GPU (think google colab pro should be enough though) or help with how to start training! Think you can just follow the example :-) 

I have started fine-tuning the classifier Trillsson model, it works on the CPU. However, when I tried to fine-tune on GPU, it threw the error `RuntimeError: CUDA error: unspecified launch failure`.  I have checked the whole process, I think it's because of the data loader but it's hard to debug. Do you have any experience in dealing with this error? This is the script I used to fine-tune https://github.com/vumichien/transformers/blob/add_trillson_effecient/src/transformers/models/trillsson_efficient/run_audio_classification.py

Hmm usually datasets should not be related in any way to the GPU. Are you fine-tuning on Google Colab? We could try to debug together on Google colab :-) 

@patrickvonplaten 
I made the notebook in Colab for fine-tuning Trillsson models here https://colab.research.google.com/drive/1q51cxmpa_MtCd6LG6Jj4rLrvFIyj9uDq#scrollTo=8qcyuhPkShQd. It would be great if you have time to check it.

Hey @vumichien,

The notebook looks very nice! Does it work? Can you train it in a google colab? Otherwise happy to give you access to a GPU for a week if you'd like :-) 

Sorry, It still doesn't work on GPU. When I try to fine-tune the model on GPU it throws the error `RuntimeError: CUDA error: unspecified launch failure`. It works on CPU and shows the warning `E tensorflow/stream_executor/cuda/cuda_driver.cc:271] failed call to cuInit: CUDA_ERROR_NOT_INITIALIZED: initialization error`  but I didn't figure out where it came from yet.

This is the result after training 5 epochs on CPU, the eval loss reduces but eval accuracy doesn't change as much. I think we should do more experiments with different hyperparameters like learning_rate and batch_size https://huggingface.co/vumichien/trillsson3-ft-keyword-spotting

Hi @patrickvonplaten 
I think I can solve the problem of training with GPU. The problem is with the dataloader multiprocess (the error is CUDA cannot be initialized), CUDA is initialized beforehand so the dataloader cannot reinitialize CUDA in multiprocess (I think because the TensorFlow code part did it) so I have to set the dataloader_num_workers to 0 to solve this problem. 
The result is this (https://huggingface.co/vumichien/trillsson3-ft-keyword-spotting-6/tensorboard). The eval accuracy is low (0.62) and unchanged after training with 20 epochs. I think training with Colab is good enough and I will try with other learning rates and max_length_seconds.
Do you have any suggestions? 

Very cool that training now works! Hmm, not 100% sure what could help the training here - the training curves looks very smooth. Gently nudging/pinging the author here (cc @joel-shor) - any ideas what could help the training? Do the training curves look reasonable?

I have found and fixed the bug when transposing the shape of the input audio array after preprocessing. Now the results look very good, we could achieve around 91% accuracy after 5 epochs. I have done several training experiments, and the results are here https://wandb.ai/chienvu/trillsson-finetune-emotion. 

Hey @vumichien,

Great job - that's very cool! I think we can now just make the tests green and merge the model then :-) 
Also cc @ArthurZucker @sanchit-gandhi could you maybe help @vumichien with the final steps to merge this PR? 

If possible it would be great if we could add an example code to https://github.com/huggingface/transformers/blob/main/examples/pytorch/audio-classification/README.md that showcases how to train the trillson audio classification model for emotion recognition! >90% is def better than any model we have so far so we should feature this prominently :-) 

Hey @sanchit-gandhi @ArthurZucker. Thank you very much for helping me.
However, when I try to use the feature extractor code from Whisper, the results are not the same.

The first one is mel_filter from [Whisper](https://github.com/huggingface/transformers/blob/0d4c45c585fadd0d9339061feda0f22fce04c57d/src/transformers/models/whisper/feature_extraction_whisper.py#L86)
and [Trillsson](https://github.com/vumichien/transformers/blob/caa21806add107b6e8acab737a8d35ce74a6e0cf/src/transformers/models/trillsson_efficient/feature_extraction_trillsson_efficient.py#L166-L172) are not the same. I have tried with sr=16000, n_mels=80, max_mel=7500.0, and min_mel=125.0. I think I should implement the original code from Tensorflow in Numpy (pls correct me if my assumption is wrong)

https://github.com/tensorflow/tensorflow/blob/359c3cdfc5fabac82b3c70b3b6de2b0a8c16874f/tensorflow/python/ops/signal/mel_ops.py#L89-L215

The second one is the STFT features from [Whisper](https://github.com/huggingface/transformers/blob/0d4c45c585fadd0d9339061feda0f22fce04c57d/src/transformers/models/whisper/feature_extraction_whisper.py#L207) and from [Trillsson](https://github.com/vumichien/transformers/blob/caa21806add107b6e8acab737a8d35ce74a6e0cf/src/transformers/models/trillsson_efficient/feature_extraction_trillsson_efficient.py#L157) are also not the same. I have tried with n_fft = 512, sampling_rate = 16000, n_mels=80, and window_length = 400, hop_length =160, the shape output from Whisper is (400, 257) compared to TF is (398, 257). 



Hey @vumichien! If the implementations of the Mel-filter and STFT are _inherently_ different between the NumPy Whisper code and the Tensorflow Trillsson code, then you are entirely correct in that we should re-implement it. Have you managed to determine _where_ in the code the two implementations deviate? Perhaps if you pin it down we could assert whether it's a configuration issue, or inherent implementation difference. If it's the former, we won't have to re-implement. If it's the latter, we will!

Maybe one other thing you could try is using the PyTorch implementation of the log-Mel feature extractor from Speech2Text: https://github.com/huggingface/transformers/blob/c3a93d8d821bc1df3601ba858e7385eada8db3a5/src/transformers/models/speech_to_text/feature_extraction_speech_to_text.py#L80
If we've got a PyTorch model, having a PyTorch dependency in the feature extractor is ok IMO. However, we'll have to re-implement it in NumPy though should someone wish to add the TF/JAX port of Trillsson later down the line.

BTW since I took care of the Mel and STFT with whisper, I will try having a look 🤗

Maybe @ArthurZucker you could share the reasoning behind why you opted for a **NumPy** feature extractor for Whisper rather than a **PyTorch** one as we have in Speech2Text? We can then decide whether we need a NumPy / PyTorch one for Trillsson. IMO a NumPy one is more "future proof" should we wish to add TF / JAX implementations of Trillsson, and consequently this would be my preference.

Hey @ArthurZucker! If you get the chance it would be awesome to hear your thoughts with regards to a PyTorch feature extractor (see above 👆)

Hey! Sorry for the late reply! 
It's mostly to remove the dependency on `PyTorch` as we were sure that the `tf` model would be implemented. 

I guess that it's not that hard of a constraint. I'll have a look ! Sorry for the long wait 🙇

Great! So I guess it boils down to the question of whether we anticipate this model will be added in TF or Flax? I probably stand-by my previously stated preference from NumPy in this regard - given the performance of this model for audio classification, I don't see any reason why this model won't be added in either of the other frameworks in due course!

Hey @vumichien,

I'm going back on my previous comment! My advice would be to try using the PyTorch feature extractor from speech-to-text:

https://github.com/huggingface/transformers/blob/cbbeca3d1733aa7c9b443af5ff231a5affcd8a1e/src/transformers/models/speech_to_text/feature_extraction_speech_to_text.py#L33

If this PyTorch feature extractor matches the current TF one, we can just use a PyTorch feature extractor for purpose of this PR. If in due course we add the TF or Flax model, we can switch it out for a NumPy one. This should be much faster for you than implementing in NumPy from scratch.

If, however, the PyTorch feature extractor does not match the current TF one, we can implement the feature extractor in NumPy from the get-go with the help of @ArthurZucker!

You can check quickly if the implementations match just by importing the Speech2TextFeatureExtractor:
```python
from transformers import Speech2TextFeatureExtractor

feature_extractor = Speech2TextFeatureExtractor.from_pretrained("facebook/s2t-small-librispeech-asr")

inputs = ...

input_features = feature_extractor(inputs).input_features[0]
```
then check `input_features` against your current TF implementation

Hi @sanchit-gandhi, thank you for your advice.
I will test PyTorch feature extractor from speech-to-text

Hey @sanchit-gandhi @ArthurZucker 
I have tested the Speech2TextFeatureExtractor and the output is not the same as the Trillsson TF feature extractor (it's much closer to WhisperFeatureExtractor but we need to modify it a bit)
In order to implement the Trillsson TF feature extractor in NumPy, we need to implement two functions: [tf.signal.linear_to_mel_weight_matrix](https://www.tensorflow.org/api_docs/python/tf/signal/linear_to_mel_weight_matrix) (This function follows the [Hidden Markov Model Toolkit (HTK)](http://htk.eng.cam.ac.uk/) convention) and [tf.signal.stft](https://www.tensorflow.org/api_docs/python/tf/signal/stft). It's not a hard task but it takes time to check everything carefully.

Okay! Tell me if I can be of help or if you are stuck! 

Sounds good! Keep us posted with progress / questions, more than happy to help 🤗

Hey @sanchit-gandhi  @ArthurZucker. I have implemented the function to compute STFT from Tensorflow into Numpy. I also checked the training result after changing the feature extractor and the [result](https://huggingface.co/vumichien/trillsson3-ft-keyword-spotting-15/tensorboard) was almost the same as before. But the CircleCI tests are still red, would you mind helping me?

Hey! So first thing first, you should run `make fix-copies`. Then try solving whatever error you get when running `make fixup`. This should solve at least a few issues and clean a bit. 
Seems like you have naming errors regarding : 
```python 
AttributeError: module transformers.models.trillsson_efficient has no attribute Trillsson_efficientFeatureExtractor
```
So `make fix-copies` will help a lot. 

Hi @ArthurZucker @sanchit-gandhi, could you help me fix this CI/CD test.
I can run the test file on a local machine by this syntax 
```python
RUN_SLOW=1 pytest -sv tests/models/trillsson_efficient/test_modeling_trillsson_efficient.py
```` 
and all the test cases were passed. But the CI/CD test show that's error 

```python
AttributeError: module transformers.models.trillsson_efficient has no attribute Trillsson_efficientFeatureExtractor
```
However, the test_torch in CI/CD was passed. So in my case, how could I fix it?

Hey, let me have a look ! I think it should be a typo or small nit we forgot! 

@ArthurZucker I have applied your suggestion but It still didn't pass the test :smiling_face_with_tear: 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_17387). All of your documentation changes will be reflected on that endpoint.

Thanks, @ArthurZucker. I think it's fine now. Do you need me to revise or do something before merging?

@ArthurZucker Thank you for your review. I have made some changes as per your kind suggestion. Could you please check again and let me know if I need to revise something?

Hey! Will have a look again 😉 You can also try to pull from main to fix the two `README` issues !

Hi @ArthurZucker, I have solved these conflict README files. However, I want to ask you about the [post init here](https://github.com/vumichien/transformers/blob/2f056b886bf43febc3615fb8988eab68cc0f7b66/src/transformers/models/trillsson_efficientnet/modeling_trillsson_efficientnet.py#L611), when I try to init weight of base model of TrillssonEfficientNetForSequenceClassification like this
```python
from transformers import (
    AutoConfig,
    TrillssonEfficientNetForSequenceClassification,
)
config = AutoConfig.from_pretrained(
    "vumichien/nonsemantic-speech-trillsson3", finetuning_task="audio-classification", use_auth_token=False
)

model = TrillssonEfficientNetForSequenceClassification.from_pretrained(
    "vumichien/nonsemantic-speech-trillsson3",
    from_tf=False,
    config=config,
    use_auth_token=True,
)
```

It fails and shows the error message
```python
Some weights of the model checkpoint at vumichien/nonsemantic-speech-trillsson3 were not used when initializing TrillssonEfficientNetForSequenceClassification: ['block.10.expand_bn.weight', 'block.33.expand_bn.weight', 'block.29.project_bn.bias', 'block.23.se.se_expand.bias', 'block.5.project_conv.weight', 'block.39.expand_bn.weight', 'block.39.expand_bn.num_batches_tracked', 'block.33.bn.bias', 'block.13.se.se_expand.bias', 'block.26.expand_bn.weight', 'block.18.se.se_expand.bias', 'block.36.project_bn.running_var', 'block.33.project_bn.running_var', 'block.11.bn.num_batches_tracked', 'block.22.bn.running_var', 'block.35.project_bn.running_mean', 'block.10.expand_conv.weight', 'block.11.project_bn.num_batches_tracked', 'block.21.bn.running_var', 'block.31.project_bn.running_mean', 'block.16.bn.bias', 'block.11.bn.running_var', 'block.2.expand_conv.weight', 'block.26.expand_bn.running_mean', 'block.38.expand_bn.weight', 'block.20.dwconv2.weight', 'block.14.expand_conv.weight', 'block.27.project_bn.bias', 'block.20.bn.num_batches_tracked', 'block.34.project_conv.weight', 'block.12.se.se_expand.bias', 'block.32.expand_bn.running_mean', 'block.37.bn.weight', 'block.6.expand_bn.bias', 'block.23.expand_bn.num_batches_tracked', 'block.26.expand_conv.weight', 'block.13.project_bn.weight', 'top.top_bn.num_batches_tracked', 'block.24.project_bn.running_mean', 'block.28.project_bn.bias', 'block.16.se.se_reduce.bias', 'block.6.project_bn.weight', 'block.16.project_bn.running_mean', 'block.28.bn.running_mean', 'block.10.project_bn.bias', 'block.16.expand_bn.weight', 'block.17.bn.weight', 'block.1.project_bn.running_mean', 'block.11.expand_bn.weight', 'block.9.expand_bn.bias', 'block.22.dwconv2.weight', 'block.16.bn.weight', 'block.37.se.se_expand.bias', 'block.16.project_bn.weight', 'block.20.se.se_expand.weight', 'block.23.bn.bias', 'block.22.expand_bn.running_mean', 'block.32.expand_conv.weight', 'block.14.project_conv.weight', 'block.13.project_bn.bias', 'block.14.project_bn.running_mean', 'block.31.expand_bn.weight', 'block.25.bn.weight', 'block.33.project_conv.weight', 'block.29.expand_bn.num_batches_tracked', 'block.11.expand_bn.running_mean', 'block.39.bn.running_var', 'block.15.bn.weight', 'block.33.se.se_reduce.weight', 'block.36.expand_bn.bias', 'block.14.expand_bn.weight', 'block.13.se.se_expand.weight', 'block.21.project_bn.running_var', 'block.32.dwconv2.weight', 'block.30.bn.running_var', 'block.25.project_bn.num_batches_tracked', 'block.26.bn.num_batches_tracked', 'block.30.expand_bn.weight', 'block.33.expand_bn.bias', 'block.25.project_bn.running_var', 'block.36.bn.num_batches_tracked', 'block.7.project_conv.weight', 'block.33.bn.num_batches_tracked', 'block.3.project_bn.weight', 'block.14.project_bn.running_var', 'block.38.se.se_expand.bias', 'block.13.expand_bn.bias', 'block.27.expand_bn.num_batches_tracked', 'block.14.se.se_reduce.weight', 'block.32.project_bn.bias', 'block.22.expand_conv.weight', 'block.27.dwconv2.weight', 'block.13.project_bn.running_mean', 'block.16.bn.running_mean', 'block.23.se.se_reduce.bias', 'block.29.se.se_expand.weight', 'block.39.se.se_reduce.weight', 'block.29.project_bn.running_mean', 'block.33.expand_bn.num_batches_tracked', 'block.9.project_bn.running_var', 'block.38.bn.bias', 'block.21.project_bn.num_batches_tracked', 'block.21.bn.running_mean', 'block.21.dwconv2.weight', 'block.39.bn.running_mean', 'block.10.expand_bn.num_batches_tracked', 'block.19.project_bn.running_mean', 'block.28.project_bn.num_batches_tracked', 'block.37.se.se_expand.weight', 'block.20.se.se_reduce.bias', 'block.20.bn.running_var', 'block.18.expand_conv.weight', 'block.25.project_bn.running_mean', 'block.28.project_bn.running_var', 'block.28.bn.num_batches_tracked', 'block.17.bn.running_var', 'block.8.project_bn.bias', 'block.27.bn.running_mean', 'block.26.project_bn.num_batches_tracked', 'block.10.expand_bn.running_mean', 'block.24.bn.num_batches_tracked', 'block.26.se.se_expand.weight', 'block.35.expand_bn.num_batches_tracked', 'block.17.se.se_reduce.bias', 'block.23.project_conv.weight', 'block.7.project_bn.running_mean', 'block.17.project_bn.num_batches_tracked', 'block.31.project_bn.bias', 'block.19.dwconv2.weight', 'block.32.se.se_reduce.weight', 'block.19.expand_conv.weight', 'block.29.project_bn.running_var', 'block.29.expand_bn.running_var', 'top.top_bn.running_var', 'block.27.project_bn.weight', 'block.20.se.se_expand.bias', 'block.28.se.se_expand.weight', 'block.37.expand_conv.weight', 'block.25.expand_bn.running_mean', 'block.28.bn.running_var', 'block.7.project_bn.running_var', 'block.19.expand_bn.weight', 'block.28.expand_bn.bias', 'block.9.expand_conv.weight', 'block.18.project_bn.running_mean', 'block.22.expand_bn.weight', 'block.23.bn.weight', 'block.39.expand_bn.running_var', 'block.21.project_bn.weight', 'block.32.project_bn.num_batches_tracked', 'block.11.expand_conv.weight', 'block.31.se.se_expand.bias', 'block.13.project_bn.running_var', 'block.14.bn.num_batches_tracked', 'block.14.expand_bn.num_batches_tracked', 'block.15.project_bn.bias', 'block.26.se.se_expand.bias', 'block.15.project_bn.running_var', 'block.26.bn.bias', 'block.35.dwconv2.weight', 'block.28.project_bn.running_mean', 'block.32.expand_bn.running_var', 'block.36.project_bn.running_mean', 'stem.stem_bn.running_mean', 'block.22.se.se_reduce.weight', 'block.15.project_conv.weight', 'block.18.bn.weight', 'block.30.se.se_reduce.weight', 'block.6.project_bn.running_var', 'block.19.expand_bn.running_mean', 'block.8.expand_bn.bias', 'block.26.project_bn.running_mean', 'block.33.expand_conv.weight', 'block.10.expand_bn.running_var', 'block.22.expand_bn.bias', 'block.21.project_bn.running_mean', 'block.25.bn.num_batches_tracked', 'block.35.expand_bn.bias', 'block.4.project_bn.running_mean', 'block.11.project_bn.running_var', 'block.38.project_bn.weight', 'block.39.se.se_reduce.bias', 'block.10.project_conv.weight', 'block.5.project_bn.running_var', 'top.top_bn.bias', 'block.21.expand_bn.weight', 'block.5.expand_bn.num_batches_tracked', 'block.6.expand_bn.num_batches_tracked', 'block.18.bn.bias', 'block.28.expand_bn.running_mean', 'block.13.bn.num_batches_tracked', 'block.36.expand_bn.num_batches_tracked', 'block.37.project_bn.running_var', 'block.27.se.se_expand.weight', 'block.13.bn.running_var', 'block.33.se.se_expand.bias', 'block.5.expand_bn.running_mean', 'block.36.project_conv.weight', 'block.31.bn.running_mean', 'block.12.expand_bn.running_var', 'block.39.bn.bias', 'block.15.se.se_expand.weight', 'top.top_conv.weight', 'block.3.project_conv.weight', 'block.25.se.se_reduce.bias', 'block.32.expand_bn.num_batches_tracked', 'block.12.project_bn.bias', 'block.34.project_bn.num_batches_tracked', 'block.34.expand_conv.weight', 'block.16.expand_bn.bias', 'block.15.bn.num_batches_tracked', 'block.8.expand_conv.weight', 'block.24.se.se_expand.bias', 'stem.stem_conv.weight', 'block.23.bn.num_batches_tracked', 'block.16.se.se_reduce.weight', 'block.5.project_bn.weight', 'block.9.project_bn.weight', 'block.13.expand_conv.weight', 'block.3.expand_bn.num_batches_tracked', 'block.31.se.se_expand.weight', 'block.38.dwconv2.weight', 'block.7.expand_bn.num_batches_tracked', 'block.11.dwconv2.weight', 'block.20.bn.weight', 'block.34.expand_bn.bias', 'block.16.project_bn.bias', 'block.29.expand_conv.weight', 'block.12.bn.num_batches_tracked', 'block.13.expand_bn.running_var', 'block.29.expand_bn.weight', 'block.12.expand_bn.num_batches_tracked', 'block.19.bn.weight', 'block.39.project_bn.weight', 'block.18.project_bn.running_var', 'block.13.expand_bn.running_mean', 'block.32.project_bn.weight', 'block.12.bn.running_mean', 'block.33.se.se_expand.weight', 'block.12.expand_conv.weight', 'block.25.bn.running_var', 'block.3.project_bn.running_mean', 'block.2.expand_bn.weight', 'block.12.bn.weight', 'block.18.expand_bn.num_batches_tracked', 'block.21.expand_bn.num_batches_tracked', 'block.36.expand_conv.weight', 'block.11.expand_bn.num_batches_tracked', 'block.36.se.se_expand.weight', 'block.2.project_bn.bias', 'block.28.project_conv.weight', 'block.9.expand_bn.running_var', 'block.18.se.se_expand.weight', 'block.15.project_bn.weight', 'block.15.expand_conv.weight', 'block.5.project_bn.bias', 'block.37.expand_bn.num_batches_tracked', 'block.1.project_bn.num_batches_tracked', 'block.36.project_bn.num_batches_tracked', 'block.31.project_bn.num_batches_tracked', 'block.36.se.se_reduce.bias', 'block.26.bn.running_var', 'block.24.bn.weight', 'block.9.project_bn.running_mean', 'block.5.expand_bn.weight', 'block.38.project_bn.running_var', 'block.22.expand_bn.num_batches_tracked', 'block.18.expand_bn.running_mean', 'block.39.bn.num_batches_tracked', 'block.4.expand_bn.running_var', 'block.37.bn.num_batches_tracked', 'block.30.bn.bias', 'block.31.project_conv.weight', 'block.17.expand_bn.running_var', 'block.35.project_bn.bias', 'block.39.project_bn.num_batches_tracked', 'block.22.project_conv.weight', 'block.35.expand_conv.weight', 'block.11.bn.running_mean', 'block.16.project_bn.num_batches_tracked', 'block.24.expand_bn.running_mean', 'block.31.bn.bias', 'block.12.project_bn.weight', 'block.18.project_bn.weight', 'block.3.expand_bn.running_var', 'block.4.project_bn.weight', 'block.6.project_bn.bias', 'block.16.se.se_expand.bias', 'block.20.expand_bn.running_var', 'block.2.expand_bn.running_mean', 'block.37.dwconv2.weight', 'block.39.project_bn.bias', 'block.32.bn.running_var', 'block.36.expand_bn.weight', 'block.16.project_bn.running_var', 'stem.stem_bn.running_var', 'block.6.expand_bn.weight', 'block.15.se.se_reduce.bias', 'block.10.bn.weight', 'block.17.expand_bn.num_batches_tracked', 'block.18.bn.running_var', 'block.20.project_bn.running_var', 'block.15.expand_bn.bias', 'block.25.dwconv2.weight', 'block.7.project_bn.weight', 'block.14.bn.running_var', 'block.17.expand_bn.running_mean', 'block.28.project_bn.weight', 'block.33.bn.running_mean', 'block.22.bn.bias', 'block.28.expand_bn.running_var', 'block.28.dwconv2.weight', 'block.30.expand_bn.running_var', 'block.37.bn.running_var', 'block.17.expand_bn.bias', 'block.2.expand_bn.num_batches_tracked', 'block.19.expand_bn.bias', 'block.25.expand_bn.bias', 'block.34.expand_bn.running_mean', 'block.3.project_bn.running_var', 'block.38.bn.weight', 'block.37.expand_bn.weight', 'block.15.project_bn.num_batches_tracked', 'block.20.project_bn.weight', 'block.29.expand_bn.running_mean', 'block.24.project_bn.num_batches_tracked', 'block.25.project_bn.bias', 'block.32.project_conv.weight', 'block.39.expand_bn.bias', 'block.12.se.se_reduce.weight', 'block.20.project_bn.bias', 'block.33.bn.weight', 'block.10.project_bn.running_mean', 'block.34.bn.num_batches_tracked', 'block.10.se.se_expand.bias', 'block.22.bn.num_batches_tracked', 'block.15.expand_bn.weight', 'block.38.expand_bn.bias', 'block.16.expand_conv.weight', 'block.4.project_bn.bias', 'block.8.project_bn.num_batches_tracked', 'block.34.se.se_reduce.weight', 'block.24.expand_bn.running_var', 'block.4.project_bn.running_var', 'block.10.project_bn.running_var', 'block.25.expand_bn.num_batches_tracked', 'block.26.project_bn.bias', 'block.37.expand_bn.running_var', 'block.22.bn.running_mean', 'block.33.project_bn.running_mean', 'block.15.dwconv2.weight', 'block.39.se.se_expand.weight', 'block.13.expand_bn.weight', 'block.8.expand_bn.running_mean', 'block.2.expand_bn.bias', 'block.33.expand_bn.running_var', 'block.27.bn.num_batches_tracked', 'block.35.bn.num_batches_tracked', 'block.32.se.se_expand.weight', 'block.31.se.se_reduce.bias', 'block.7.expand_bn.weight', 'block.18.bn.running_mean', 'block.34.bn.running_mean', 'block.32.project_bn.running_var', 'block.5.project_bn.num_batches_tracked', 'block.31.project_bn.weight', 'block.25.bn.running_mean', 'block.27.se.se_expand.bias', 'block.38.bn.running_mean', 'block.30.expand_bn.running_mean', 'block.29.se.se_reduce.bias', 'block.20.expand_conv.weight', 'block.8.project_bn.weight', 'block.0.project_bn.bias', 'block.18.project_conv.weight', 'block.7.expand_bn.running_mean', 'block.26.project_bn.running_var', 'block.38.project_bn.num_batches_tracked', 'block.12.project_conv.weight', 'block.12.project_bn.running_mean', 'block.33.dwconv2.weight', 'block.38.bn.running_var', 'block.12.project_bn.running_var', 'block.21.se.se_reduce.bias', 'block.15.project_bn.running_mean', 'block.19.project_bn.running_var', 'block.34.project_bn.running_mean', 'block.24.se.se_expand.weight', 'block.36.bn.weight', 'block.35.project_bn.num_batches_tracked', 'block.26.se.se_reduce.bias', 'block.29.bn.bias', 'block.3.expand_bn.running_mean', 'block.24.se.se_reduce.bias', 'block.27.expand_bn.bias', 'block.27.se.se_reduce.weight', 'block.31.expand_bn.bias', 'block.21.se.se_expand.bias', 'block.37.se.se_reduce.bias', 'block.19.se.se_expand.weight', 'block.11.se.se_expand.bias', 'block.14.bn.bias', 'block.37.project_bn.running_mean', 'block.14.project_bn.bias', 'block.21.se.se_expand.weight', 'block.32.bn.bias', 'block.12.dwconv2.weight', 'block.23.project_bn.bias', 'block.19.project_bn.bias', 'block.30.bn.num_batches_tracked', 'block.27.project_conv.weight', 'block.12.expand_bn.weight', 'block.35.expand_bn.running_mean', 'block.34.bn.running_var', 'block.24.expand_conv.weight', 'block.36.bn.running_var', 'block.34.expand_bn.weight', 'block.29.expand_bn.bias', 'block.10.se.se_expand.weight', 'block.35.project_conv.weight', 'block.35.se.se_reduce.bias', 'block.14.se.se_expand.bias', 'block.17.se.se_expand.weight', 'block.30.project_bn.weight', 'block.24.bn.running_mean', 'block.20.expand_bn.num_batches_tracked', 'block.18.se.se_reduce.weight', 'block.16.expand_bn.running_mean', 'block.35.expand_bn.running_var', 'block.8.project_bn.running_var', 'block.36.bn.bias', 'block.32.expand_bn.weight', 'block.3.expand_bn.weight', 'block.23.project_bn.running_mean', 'block.22.se.se_expand.weight', 'block.13.bn.running_mean', 'block.31.expand_bn.running_mean', 'block.14.expand_bn.running_var', 'block.23.expand_bn.bias', 'block.24.project_bn.weight', 'block.29.dwconv2.weight', 'block.15.se.se_reduce.weight', 'block.23.expand_bn.running_mean', 'block.11.se.se_expand.weight', 'block.30.se.se_expand.weight', 'block.34.expand_bn.num_batches_tracked', 'block.38.se.se_expand.weight', 'block.39.project_bn.running_var', 'block.37.bn.running_mean', 'block.22.expand_bn.running_var', 'block.9.expand_bn.running_mean', 'block.33.project_bn.num_batches_tracked', 'block.10.project_bn.num_batches_tracked', 'block.19.project_bn.num_batches_tracked', 'block.1.project_conv.weight', 'block.19.expand_bn.running_var', 'block.21.expand_conv.weight', 'block.24.expand_bn.num_batches_tracked', 'block.29.se.se_expand.bias', 'block.8.expand_bn.weight', 'block.36.project_bn.weight', 'block.38.se.se_reduce.bias', 'block.6.project_conv.weight', 'block.36.dwconv2.weight', 'block.0.project_conv.weight', 'block.32.project_bn.running_mean', 'top.top_bn.weight', 'block.31.dwconv2.weight', 'block.36.expand_bn.running_mean', 'block.19.bn.num_batches_tracked', 'block.20.project_conv.weight', 'block.8.expand_bn.running_var', 'block.19.se.se_expand.bias', 'block.5.project_bn.running_mean', 'block.28.se.se_reduce.bias', 'block.29.project_bn.num_batches_tracked', 'block.2.project_bn.running_var', 'block.11.se.se_reduce.bias', 'block.23.bn.running_mean', 'block.10.se.se_reduce.weight', 'block.31.se.se_reduce.weight', 'block.30.project_bn.num_batches_tracked', 'block.7.expand_bn.bias', 'block.38.project_bn.bias', 'block.22.project_bn.bias', 'block.32.bn.running_mean', 'block.28.bn.bias', 'block.15.se.se_expand.bias', 'block.25.se.se_reduce.weight', 'block.17.bn.running_mean', 'block.8.project_conv.weight', 'block.35.project_bn.weight', 'block.11.project_bn.running_mean', 'block.4.project_bn.num_batches_tracked', 'block.6.expand_conv.weight', 'block.18.project_bn.bias', 'block.35.se.se_expand.weight', 'block.18.bn.num_batches_tracked', 'block.28.se.se_reduce.weight', 'block.18.dwconv2.weight', 'block.17.project_conv.weight', 'block.4.expand_bn.bias', 'block.22.project_bn.num_batches_tracked', 'block.30.expand_conv.weight', 'top.top_bn.running_mean', 'block.2.project_bn.weight', 'block.30.project_bn.running_var', 'block.14.se.se_reduce.bias', 'block.35.bn.running_var', 'block.5.expand_conv.weight', 'block.19.bn.running_mean', 'block.31.bn.weight', 'block.6.expand_bn.running_mean', 'block.32.bn.num_batches_tracked', 'block.34.se.se_expand.bias', 'block.16.expand_bn.num_batches_tracked', 'block.3.expand_bn.bias', 'block.25.se.se_expand.bias', 'block.0.project_bn.weight', 'block.8.project_bn.running_mean', 'block.36.se.se_reduce.weight', 'block.20.se.se_reduce.weight', 'block.24.project_bn.bias', 'block.39.se.se_expand.bias', 'block.11.bn.bias', 'block.14.project_bn.weight', 'block.9.expand_bn.num_batches_tracked', 'block.5.expand_bn.running_var', 'block.19.project_conv.weight', 'block.26.expand_bn.bias', 'block.34.se.se_expand.weight', 'block.15.expand_bn.num_batches_tracked', 'block.29.bn.weight', 'block.26.project_bn.weight', 'block.38.expand_bn.running_mean', 'block.39.bn.weight', 'block.19.bn.bias', 'block.15.bn.running_mean', 'block.21.project_bn.bias', 'block.34.bn.weight', 'block.29.bn.running_var', 'block.16.bn.num_batches_tracked', 'block.36.se.se_expand.bias', 'block.39.project_conv.weight', 'block.2.expand_bn.running_var', 'block.22.project_bn.weight', 'block.11.expand_bn.running_var', 'block.34.dwconv2.weight', 'block.23.dwconv2.weight', 'block.27.bn.weight', 'block.4.project_conv.weight', 'block.21.expand_bn.running_mean', 'block.25.se.se_expand.weight', 'block.38.project_conv.weight', 'block.12.expand_bn.bias', 'block.27.project_bn.running_var', 'block.13.se.se_reduce.bias', 'block.30.expand_bn.bias', 'block.27.expand_bn.running_mean', 'block.31.expand_conv.weight', 'block.6.project_bn.running_mean', 'block.4.expand_bn.num_batches_tracked', 'block.20.bn.running_mean', 'block.30.dwconv2.weight', 'block.17.project_bn.weight', 'block.17.expand_bn.weight', 'block.17.expand_conv.weight', 'block.35.project_bn.running_var', 'block.16.dwconv2.weight', 'block.32.se.se_expand.bias', 'block.21.bn.bias', 'block.34.se.se_reduce.bias', 'block.28.bn.weight', 'block.7.project_bn.bias', 'block.29.bn.running_mean', 'block.29.se.se_reduce.weight', 'block.11.project_conv.weight', 'block.38.expand_bn.running_var', 'block.10.bn.running_var', 'block.3.project_bn.num_batches_tracked', 'block.30.bn.weight', 'block.37.expand_bn.running_mean', 'block.1.project_bn.bias', 'block.26.se.se_reduce.weight', 'block.24.project_conv.weight', 'block.19.se.se_reduce.bias', 'block.33.bn.running_var', 'block.14.project_bn.num_batches_tracked', 'block.19.se.se_reduce.weight', 'block.23.expand_conv.weight', 'block.35.expand_bn.weight', 'block.32.expand_bn.bias', 'block.17.se.se_expand.bias', 'block.25.bn.bias', 'block.17.bn.bias', 'block.30.se.se_reduce.bias', 'block.31.bn.num_batches_tracked', 'block.21.expand_bn.running_var', 'block.26.bn.running_mean', 'block.27.expand_bn.weight', 'block.25.expand_bn.running_var', 'block.22.se.se_expand.bias', 'block.18.expand_bn.bias', 'block.12.se.se_reduce.bias', 'block.22.se.se_reduce.bias', 'block.5.expand_bn.bias', 'block.21.expand_bn.bias', 'block.1.project_bn.weight', 'block.17.project_bn.running_mean', 'block.25.project_conv.weight', 'block.29.project_conv.weight', 'block.30.project_bn.bias', 'block.13.expand_bn.num_batches_tracked', 'block.12.bn.running_var', 'block.22.project_bn.running_mean', 'block.23.se.se_expand.weight', 'block.29.project_bn.weight', 'block.23.expand_bn.running_var', 'block.28.se.se_expand.bias', 'block.25.project_bn.weight', 'block.17.project_bn.bias', 'block.33.project_bn.weight', 'block.20.project_bn.num_batches_tracked', 'block.28.expand_bn.weight', 'block.15.bn.bias', 'block.13.se.se_reduce.weight', 'block.11.se.se_reduce.weight', 'block.24.project_bn.running_var', 'block.27.expand_conv.weight', 'block.26.dwconv2.weight', 'block.3.project_bn.bias', 'block.18.expand_bn.running_var', 'block.38.bn.num_batches_tracked', 'block.24.dwconv2.weight', 'block.20.project_bn.running_mean', 'block.30.se.se_expand.bias', 'block.37.project_bn.num_batches_tracked', 'block.31.expand_bn.num_batches_tracked', 'block.4.expand_bn.weight', 'block.26.expand_bn.num_batches_tracked', 'block.15.bn.running_var', 'block.38.project_bn.running_mean', 'block.28.expand_bn.num_batches_tracked', 'block.30.bn.running_mean', 'block.13.dwconv2.weight', 'block.22.bn.weight', 'dense.weight', 'block.16.project_conv.weight', 'block.19.bn.running_var', 'block.27.expand_bn.running_var', 'block.21.se.se_reduce.weight', 'block.27.bn.bias', 'block.11.expand_bn.bias', 'block.23.project_bn.num_batches_tracked', 'block.37.se.se_reduce.weight', 'block.25.expand_conv.weight', 'block.26.expand_bn.running_var', 'block.7.project_bn.num_batches_tracked', 'block.14.se.se_expand.weight', 'block.39.dwconv2.weight', 'block.0.project_bn.running_var', 'block.21.bn.num_batches_tracked', 'block.2.project_bn.running_mean', 'block.37.project_bn.weight', 'block.4.expand_conv.weight', 'block.38.expand_conv.weight', 'block.14.dwconv2.weight', 'block.37.expand_bn.bias', 'block.11.project_bn.weight', 'block.29.bn.num_batches_tracked', 'block.23.bn.running_var', 'block.9.expand_bn.weight', 'block.24.bn.bias', 'block.32.se.se_reduce.bias', 'block.37.project_conv.weight', 'block.23.se.se_reduce.weight', 'block.14.bn.weight', 'block.14.bn.running_mean', 'stem.stem_bn.num_batches_tracked', 'block.14.expand_bn.bias', 'block.39.expand_conv.weight', 'block.24.bn.running_var', 'block.1.project_bn.running_var', 'block.3.expand_conv.weight', 'block.12.se.se_expand.weight', 'block.27.bn.running_var', 'block.23.expand_bn.weight', 'block.13.project_bn.num_batches_tracked', 'block.24.expand_bn.weight', 'block.27.se.se_reduce.bias', 'block.35.bn.weight', 'block.12.bn.bias', 'block.4.expand_bn.running_mean', 'block.34.project_bn.bias', 'block.16.bn.running_var', 'block.8.expand_bn.num_batches_tracked', 'block.17.bn.num_batches_tracked', 'block.16.expand_bn.running_var', 'block.6.project_bn.num_batches_tracked', 'block.6.expand_bn.running_var', 'block.26.project_conv.weight', 'block.20.bn.bias', 'block.13.bn.weight', 'dense.bias', 'block.12.project_bn.num_batches_tracked', 'block.33.project_bn.bias', 'block.11.bn.weight', 'block.15.expand_bn.running_mean', 'block.0.project_bn.running_mean', 'block.10.se.se_reduce.bias', 'block.17.project_bn.running_var', 'block.27.project_bn.running_mean', 'block.9.project_bn.num_batches_tracked', 'block.14.expand_bn.running_mean', 'block.25.expand_bn.weight', 'block.18.expand_bn.weight', 'block.34.project_bn.running_var', 'block.15.expand_bn.running_var', 'block.7.expand_bn.running_var', 'block.27.project_bn.num_batches_tracked', 'block.20.expand_bn.running_mean', 'block.35.se.se_reduce.weight', 'block.7.expand_conv.weight', 'block.19.project_bn.weight', 'block.18.se.se_reduce.bias', 'block.23.project_bn.running_var', 'block.13.project_conv.weight', 'block.10.bn.running_mean', 'block.9.project_conv.weight', 'block.17.dwconv2.weight', 'block.18.project_bn.num_batches_tracked', 'block.17.se.se_reduce.weight', 'block.22.project_bn.running_var', 'block.37.project_bn.bias', 'block.20.expand_bn.bias', 'block.21.bn.weight', 'block.24.se.se_reduce.weight', 'block.10.bn.num_batches_tracked', 'block.16.se.se_expand.weight', 'stem.stem_bn.weight', 'block.9.project_bn.bias', 'block.10.project_bn.weight', 'block.24.expand_bn.bias', 'block.26.bn.weight', 'block.2.project_conv.weight', 'block.0.project_bn.num_batches_tracked', 'block.30.expand_bn.num_batches_tracked', 'block.35.bn.bias', 'block.35.se.se_expand.bias', 'block.31.bn.running_var', 'block.31.expand_bn.running_var', 'block.30.project_conv.weight', 'block.13.bn.bias', 'block.2.project_bn.num_batches_tracked', 'block.12.expand_bn.running_mean', 'block.10.dwconv2.weight', 'block.34.expand_bn.running_var', 'block.28.expand_conv.weight', 'block.30.project_bn.running_mean', 'block.31.project_bn.running_var', 'block.33.se.se_reduce.bias', 'block.21.project_conv.weight', 'block.35.bn.running_mean', 'block.38.expand_bn.num_batches_tracked', 'block.39.project_bn.running_mean', 'block.20.expand_bn.weight', 'stem.stem_bn.bias', 'block.34.bn.bias', 'block.36.bn.running_mean', 'block.23.project_bn.weight', 'block.10.expand_bn.bias', 'block.33.expand_bn.running_mean', 'block.32.bn.weight', 'block.38.se.se_reduce.weight', 'block.39.expand_bn.running_mean', 'block.37.bn.bias', 'block.36.project_bn.bias', 'block.36.expand_bn.running_var', 'block.19.expand_bn.num_batches_tracked', 'block.34.project_bn.weight', 'block.11.project_bn.bias', 'block.10.bn.bias']
- This IS expected if you are initializing TrillssonEfficientNetForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing TrillssonEfficientNetForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Some weights of TrillssonEfficientNetForSequenceClassification were not initialized from the model checkpoint at vumichien/nonsemantic-speech-trillsson3 and are newly initialized: ['trillsson.block.39.expand_conv.weight', 'trillsson.block.37.project_conv.weight', 'trillsson.block.11.se.se_expand.bias', 'trillsson.block.34.expand_bn.num_batches_tracked', 'trillsson.block.18.expand_bn.bias', 'trillsson.block.8.expand_bn.weight', 'trillsson.block.31.bn.running_var', 'trillsson.block.5.expand_conv.weight', 'trillsson.block.36.expand_bn.running_mean', 'trillsson.block.18.se.se_reduce.weight', 'trillsson.block.28.se.se_reduce.weight', 'trillsson.block.24.expand_bn.bias', 'trillsson.block.29.se.se_reduce.bias', 'trillsson.block.16.bn.bias', 'trillsson.block.37.se.se_reduce.weight', 'trillsson.block.37.bn.running_mean', 'trillsson.block.33.bn.weight', 'trillsson.block.19.se.se_expand.weight', 'trillsson.block.36.bn.num_batches_tracked', 'classifier.bias', 'trillsson.block.35.project_bn.weight', 'trillsson.block.28.bn.num_batches_tracked', 'trillsson.block.37.se.se_expand.bias', 'trillsson.block.37.project_bn.running_mean', 'trillsson.block.19.se.se_reduce.weight', 'trillsson.block.26.bn.running_mean', 'trillsson.block.22.dwconv2.weight', 'trillsson.block.23.bn.num_batches_tracked', 'trillsson.block.26.project_bn.running_mean', 'trillsson.block.18.project_conv.weight', 'trillsson.block.18.bn.bias', 'trillsson.block.6.project_bn.bias', 'trillsson.block.3.project_bn.running_mean', 'trillsson.block.25.project_bn.weight', 'trillsson.block.19.project_conv.weight', 'trillsson.block.29.expand_bn.running_var', 'trillsson.block.29.project_bn.running_var', 'trillsson.block.2.expand_bn.weight', 'trillsson.block.16.se.se_reduce.bias', 'trillsson.block.31.bn.bias', 'trillsson.block.10.bn.running_var', 'trillsson.block.17.se.se_reduce.bias', 'trillsson.block.38.bn.running_var', 'trillsson.block.20.se.se_expand.weight', 'trillsson.block.14.bn.running_var', 'trillsson.block.20.bn.running_var', 'trillsson.block.13.project_bn.num_batches_tracked', 'trillsson.block.16.project_conv.weight', 'trillsson.block.16.bn.running_mean', 'trillsson.block.37.bn.running_var', 'trillsson.block.21.project_bn.num_batches_tracked', 'trillsson.block.13.bn.running_mean', 'trillsson.block.16.dwconv2.weight', 'trillsson.block.26.bn.num_batches_tracked', 'trillsson.block.15.expand_conv.weight', 'trillsson.block.10.bn.num_batches_tracked', 'trillsson.block.22.project_bn.num_batches_tracked', 'trillsson.block.32.bn.weight', 'trillsson.block.27.se.se_expand.bias', 'trillsson.block.26.se.se_expand.bias', 'trillsson.block.10.project_bn.bias', 'trillsson.block.20.project_bn.num_batches_tracked', 'trillsson.block.29.expand_bn.weight', 'trillsson.block.36.project_conv.weight', 'trillsson.block.25.expand_bn.weight', 'trillsson.block.14.bn.weight', 'trillsson.block.13.bn.weight', 'trillsson.block.0.project_conv.weight', 'trillsson.block.2.expand_bn.running_var', 'trillsson.block.27.bn.running_var', 'trillsson.block.28.se.se_reduce.bias', 'trillsson.block.11.expand_conv.weight', 'trillsson.block.28.bn.running_mean', 'trillsson.block.20.expand_bn.weight', 'trillsson.block.26.se.se_reduce.weight', 'trillsson.block.21.se.se_expand.bias', 'trillsson.block.13.expand_bn.running_var', 'trillsson.block.9.project_bn.bias', 'trillsson.block.21.expand_bn.running_var', 'trillsson.block.10.project_bn.num_batches_tracked', 'trillsson.block.15.project_bn.num_batches_tracked', 'trillsson.block.38.bn.bias', 'trillsson.block.15.project_conv.weight', 'trillsson.block.9.expand_bn.weight', 'trillsson.block.14.bn.num_batches_tracked', 'trillsson.block.14.se.se_expand.bias', 'trillsson.block.39.project_bn.num_batches_tracked', 'trillsson.block.9.project_bn.running_mean', 'trillsson.block.10.expand_bn.running_mean', 'trillsson.block.20.project_bn.bias', 'trillsson.block.19.expand_bn.running_var', 'trillsson.block.12.expand_conv.weight', 'trillsson.block.33.se.se_expand.bias', 'trillsson.block.37.expand_bn.running_mean', 'trillsson.block.31.project_bn.num_batches_tracked', 'trillsson.block.26.project_conv.weight', 'trillsson.block.38.bn.running_mean', 'trillsson.block.21.project_bn.weight', 'trillsson.block.19.bn.running_var', 'trillsson.block.24.project_bn.running_mean', 'trillsson.block.38.expand_bn.bias', 'trillsson.block.4.expand_bn.bias', 'trillsson.block.20.project_bn.running_var', 'trillsson.block.10.project_conv.weight', 'trillsson.block.0.project_bn.num_batches_tracked', 'trillsson.block.26.dwconv2.weight', 'trillsson.block.22.expand_bn.weight', 'trillsson.block.34.bn.weight', 'trillsson.block.6.expand_conv.weight', 'trillsson.top.top_bn.running_mean', 'trillsson.block.11.bn.weight', 'trillsson.block.10.se.se_expand.bias', 'trillsson.block.23.project_bn.running_mean', 'trillsson.block.15.bn.bias', 'trillsson.block.25.project_bn.bias', 'trillsson.block.14.project_bn.running_mean', 'trillsson.block.8.project_bn.num_batches_tracked', 'trillsson.block.39.bn.weight', 'trillsson.block.28.expand_bn.bias', 'trillsson.block.4.project_bn.num_batches_tracked', 'trillsson.block.18.dwconv2.weight', 'trillsson.block.32.expand_bn.weight', 'trillsson.block.22.se.se_expand.weight', 'trillsson.block.1.project_bn.num_batches_tracked', 'trillsson.block.30.project_bn.running_mean', 'trillsson.block.19.bn.num_batches_tracked', 'trillsson.block.5.project_bn.bias', 'trillsson.block.2.expand_bn.bias', 'trillsson.block.36.expand_conv.weight', 'trillsson.block.21.bn.num_batches_tracked', 'trillsson.block.32.project_conv.weight', 'trillsson.block.9.expand_bn.running_mean', 'trillsson.block.18.expand_bn.weight', 'trillsson.block.24.expand_bn.weight', 'trillsson.block.7.project_bn.bias', 'trillsson.block.19.bn.bias', 'trillsson.block.17.bn.running_var', 'trillsson.block.4.expand_bn.weight', 'trillsson.block.15.project_bn.weight', 'trillsson.block.21.bn.bias', 'trillsson.block.32.expand_bn.running_mean', 'trillsson.block.8.expand_bn.bias', 'trillsson.block.26.expand_bn.running_mean', 'trillsson.block.10.project_bn.running_var', 'trillsson.block.29.dwconv2.weight', 'trillsson.block.22.project_bn.weight', 'trillsson.block.34.project_bn.num_batches_tracked', 'trillsson.block.39.expand_bn.bias', 'trillsson.block.16.se.se_expand.weight', 'trillsson.block.17.dwconv2.weight', 'trillsson.block.26.expand_bn.weight', 'trillsson.block.22.expand_bn.num_batches_tracked', 'trillsson.block.17.se.se_expand.bias', 'trillsson.block.35.se.se_expand.weight', 'trillsson.block.16.project_bn.weight', 'trillsson.block.31.se.se_reduce.bias', 'trillsson.block.29.expand_conv.weight', 'trillsson.block.38.project_bn.num_batches_tracked', 'trillsson.block.37.bn.weight', 'trillsson.block.3.expand_bn.running_mean', 'trillsson.block.27.project_bn.num_batches_tracked', 'trillsson.block.25.expand_bn.num_batches_tracked', 'trillsson.block.10.se.se_reduce.weight', 'trillsson.block.12.project_conv.weight', 'trillsson.block.23.expand_bn.num_batches_tracked', 'trillsson.block.7.project_bn.running_var', 'trillsson.block.28.project_bn.bias', 'trillsson.block.39.se.se_expand.weight', 'trillsson.block.21.expand_bn.bias', 'trillsson.block.38.expand_bn.weight', 'trillsson.block.38.dwconv2.weight', 'trillsson.block.8.expand_conv.weight', 'trillsson.block.21.expand_bn.num_batches_tracked', 'trillsson.block.24.project_bn.running_var', 'trillsson.block.35.se.se_expand.bias', 'trillsson.block.11.project_bn.weight', 'trillsson.block.16.project_bn.running_mean', 'trillsson.block.17.expand_conv.weight', 'trillsson.block.34.bn.running_var', 'trillsson.block.36.bn.weight', 'trillsson.block.20.se.se_reduce.bias', 'trillsson.block.7.expand_bn.running_mean', 'trillsson.block.7.project_bn.weight', 'trillsson.block.31.bn.num_batches_tracked', 'trillsson.block.12.se.se_expand.bias', 'trillsson.block.25.bn.weight', 'trillsson.block.4.project_bn.running_mean', 'trillsson.block.28.project_bn.running_mean', 'trillsson.block.28.project_bn.num_batches_tracked', 'trillsson.block.20.project_bn.running_mean', 'trillsson.block.26.project_bn.num_batches_tracked', 'trillsson.block.9.expand_bn.bias', 'trillsson.block.31.dwconv2.weight', 'trillsson.block.6.expand_bn.running_var', 'trillsson.block.4.expand_conv.weight', 'trillsson.block.25.se.se_reduce.weight', 'trillsson.block.12.expand_bn.running_mean', 'trillsson.block.36.bn.bias', 'trillsson.block.22.bn.bias', 'trillsson.block.35.expand_bn.weight', 'trillsson.block.29.se.se_expand.weight', 'trillsson.block.39.bn.running_mean', 'trillsson.block.5.project_conv.weight', 'trillsson.block.37.bn.num_batches_tracked', 'trillsson.block.33.dwconv2.weight', 'trillsson.block.7.expand_bn.weight', 'trillsson.block.18.bn.weight', 'trillsson.block.29.project_bn.weight', 'trillsson.block.6.project_bn.running_var', 'trillsson.block.33.bn.running_var', 'trillsson.block.6.project_bn.num_batches_tracked', 'trillsson.block.13.project_bn.running_mean', 'trillsson.block.25.project_conv.weight', 'trillsson.block.30.expand_bn.running_mean', 'trillsson.block.32.bn.bias', 'trillsson.block.35.bn.weight', 'trillsson.block.12.project_bn.running_mean', 'trillsson.block.20.expand_bn.running_var', 'trillsson.block.22.project_bn.running_var', 'trillsson.block.17.expand_bn.num_batches_tracked', 'trillsson.block.27.expand_bn.weight', 'trillsson.block.17.se.se_expand.weight', 'trillsson.block.28.bn.running_var', 'trillsson.block.11.bn.num_batches_tracked', 'trillsson.block.23.expand_conv.weight', 'trillsson.block.36.se.se_expand.weight', 'trillsson.block.37.project_bn.running_var', 'trillsson.block.9.expand_bn.num_batches_tracked', 'trillsson.block.23.se.se_expand.bias', 'trillsson.block.5.expand_bn.weight', 'trillsson.block.20.bn.bias', 'trillsson.block.30.project_bn.running_var', 'trillsson.block.14.expand_bn.bias', 'trillsson.block.19.project_bn.running_var', 'trillsson.block.18.se.se_expand.weight', 'trillsson.block.2.project_bn.bias', 'trillsson.block.15.se.se_reduce.bias', 'trillsson.block.25.expand_bn.bias', 'trillsson.block.25.project_bn.running_var', 'trillsson.block.31.project_conv.weight', 'trillsson.block.34.dwconv2.weight', 'trillsson.block.25.se.se_expand.bias', 'trillsson.block.11.project_bn.running_mean', 'trillsson.block.39.se.se_reduce.weight', 'trillsson.block.29.project_conv.weight', 'trillsson.block.28.dwconv2.weight', 'trillsson.block.19.bn.running_mean', 'trillsson.block.11.project_bn.num_batches_tracked', 'trillsson.block.38.project_bn.running_mean', 'trillsson.block.26.project_bn.running_var', 'trillsson.block.6.expand_bn.bias', 'trillsson.block.22.expand_bn.running_var', 'trillsson.block.12.bn.num_batches_tracked', 'trillsson.block.32.expand_bn.bias', 'trillsson.block.25.bn.running_mean', 'trillsson.block.37.expand_bn.running_var', 'trillsson.block.27.bn.weight', 'trillsson.block.23.bn.weight', 'trillsson.block.39.expand_bn.running_mean', 'trillsson.block.26.expand_bn.running_var', 'trillsson.block.17.expand_bn.running_var', 'trillsson.block.29.project_bn.running_mean', 'trillsson.block.38.project_bn.running_var', 'trillsson.block.36.bn.running_mean', 'trillsson.block.13.bn.num_batches_tracked', 'trillsson.block.14.dwconv2.weight', 'trillsson.block.38.se.se_reduce.bias', 'trillsson.stem.stem_bn.num_batches_tracked', 'trillsson.block.2.project_bn.running_mean', 'trillsson.block.32.dwconv2.weight', 'trillsson.block.36.expand_bn.bias', 'trillsson.block.33.expand_bn.num_batches_tracked', 'trillsson.block.27.project_bn.running_mean', 'trillsson.block.19.project_bn.bias', 'trillsson.block.5.project_bn.weight', 'trillsson.block.17.bn.num_batches_tracked', 'trillsson.block.31.project_bn.bias', 'trillsson.block.36.se.se_reduce.weight', 'trillsson.block.23.dwconv2.weight', 'trillsson.stem.stem_conv.weight', 'trillsson.block.13.expand_bn.num_batches_tracked', 'trillsson.block.20.expand_conv.weight', 'trillsson.block.35.se.se_reduce.weight', 'trillsson.block.18.expand_bn.num_batches_tracked', 'trillsson.block.33.expand_bn.bias', 'trillsson.block.10.expand_bn.num_batches_tracked', 'trillsson.block.25.expand_conv.weight', 'trillsson.block.32.expand_bn.num_batches_tracked', 'trillsson.block.39.expand_bn.running_var', 'trillsson.block.2.project_bn.running_var', 'trillsson.block.8.expand_bn.running_var', 'trillsson.block.16.se.se_reduce.weight', 'trillsson.block.33.project_bn.weight', 'trillsson.block.29.expand_bn.num_batches_tracked', 'trillsson.block.29.bn.bias', 'trillsson.block.39.project_bn.running_var', 'trillsson.block.3.project_bn.num_batches_tracked', 'trillsson.block.17.bn.bias', 'trillsson.block.3.expand_bn.weight', 'trillsson.block.27.bn.bias', 'trillsson.block.27.bn.num_batches_tracked', 'trillsson.block.6.expand_bn.running_mean', 'trillsson.block.11.dwconv2.weight', 'trillsson.block.22.se.se_reduce.bias', 'trillsson.block.34.expand_conv.weight', 'trillsson.block.12.bn.bias', 'trillsson.block.13.se.se_expand.weight', 'trillsson.block.8.project_conv.weight', 'trillsson.dense.bias', 'trillsson.block.24.bn.bias', 'trillsson.block.35.expand_bn.running_mean', 'trillsson.block.23.expand_bn.bias', 'trillsson.block.11.project_conv.weight', 'trillsson.block.38.bn.weight', 'trillsson.block.29.project_bn.bias', 'trillsson.block.5.expand_bn.running_var', 'trillsson.block.20.expand_bn.num_batches_tracked', 'trillsson.block.33.project_conv.weight', 'trillsson.block.27.bn.running_mean', 'trillsson.block.2.expand_bn.running_mean', 'trillsson.block.18.expand_bn.running_mean', 'trillsson.block.32.bn.running_mean', 'trillsson.block.33.bn.num_batches_tracked', 'trillsson.block.24.project_bn.weight', 'trillsson.block.34.expand_bn.weight', 'trillsson.block.9.expand_bn.running_var', 'trillsson.block.1.project_bn.running_mean', 'trillsson.block.4.project_bn.bias', 'trillsson.block.32.se.se_reduce.weight', 'trillsson.block.10.expand_bn.bias', 'trillsson.block.37.expand_bn.weight', 'trillsson.block.23.bn.bias', 'trillsson.block.26.se.se_reduce.bias', 'trillsson.block.19.project_bn.num_batches_tracked', 'trillsson.block.10.dwconv2.weight', 'trillsson.block.16.expand_bn.weight', 'trillsson.block.36.expand_bn.num_batches_tracked', 'trillsson.block.16.expand_bn.running_var', 'trillsson.block.7.expand_conv.weight', 'trillsson.block.26.bn.weight', 'trillsson.block.28.bn.weight', 'classifier.weight', 'trillsson.block.28.expand_bn.weight', 'trillsson.block.15.expand_bn.running_var', 'trillsson.block.25.bn.bias', 'trillsson.block.13.project_bn.weight', 'trillsson.block.10.project_bn.weight', 'trillsson.block.15.bn.running_mean', 'trillsson.block.19.se.se_expand.bias', 'trillsson.block.24.se.se_reduce.bias', 'trillsson.block.3.project_conv.weight', 'trillsson.block.35.bn.running_mean', 'trillsson.block.11.expand_bn.weight', 'trillsson.block.0.project_bn.weight', 'trillsson.block.4.expand_bn.running_var', 'trillsson.block.15.dwconv2.weight', 'trillsson.block.22.expand_bn.running_mean', 'trillsson.block.31.expand_bn.num_batches_tracked', 'trillsson.block.16.se.se_expand.bias', 'trillsson.block.35.project_bn.running_var', 'trillsson.block.14.bn.running_mean', 'trillsson.block.4.expand_bn.num_batches_tracked', 'trillsson.block.1.project_bn.running_var', 'trillsson.block.23.project_bn.num_batches_tracked', 'trillsson.block.20.expand_bn.bias', 'trillsson.block.8.project_bn.running_mean', 'trillsson.block.12.se.se_reduce.bias', 'trillsson.block.36.project_bn.running_var', 'trillsson.block.1.project_conv.weight', 'trillsson.block.31.se.se_expand.bias', 'trillsson.block.18.project_bn.bias', 'trillsson.block.22.bn.running_mean', 'trillsson.block.5.project_bn.num_batches_tracked', 'trillsson.block.36.project_bn.num_batches_tracked', 'trillsson.block.24.se.se_reduce.weight', 'trillsson.block.29.project_bn.num_batches_tracked', 'trillsson.block.16.expand_bn.bias', 'trillsson.block.18.project_bn.running_mean', 'trillsson.block.21.se.se_expand.weight', 'trillsson.block.17.bn.weight', 'trillsson.block.27.project_conv.weight', 'trillsson.block.33.se.se_expand.weight', 'trillsson.block.30.bn.num_batches_tracked', 'trillsson.block.21.expand_bn.weight', 'trillsson.block.21.project_bn.running_var', 'trillsson.block.30.bn.running_mean', 'trillsson.block.30.project_bn.num_batches_tracked', 'trillsson.block.26.bn.running_var', 'trillsson.block.31.expand_bn.weight', 'trillsson.block.25.expand_bn.running_var', 'trillsson.block.14.project_bn.num_batches_tracked', 'trillsson.block.17.project_bn.running_mean', 'trillsson.block.26.expand_bn.bias', 'trillsson.block.24.expand_bn.num_batches_tracked', 'trillsson.block.34.se.se_reduce.bias', 'trillsson.block.10.se.se_expand.weight', 'trillsson.block.27.dwconv2.weight', 'trillsson.block.37.se.se_reduce.bias', 'trillsson.block.20.se.se_expand.bias', 'trillsson.block.29.bn.running_var', 'trillsson.block.28.project_bn.running_var', 'trillsson.block.39.bn.num_batches_tracked', 'trillsson.block.31.project_bn.running_var', 'trillsson.block.13.expand_bn.running_mean', 'trillsson.block.14.expand_bn.weight', 'trillsson.block.27.project_bn.bias', 'trillsson.block.38.bn.num_batches_tracked', 'trillsson.block.15.expand_bn.num_batches_tracked', 'trillsson.block.39.project_bn.bias', 'trillsson.block.12.bn.weight', 'trillsson.block.36.project_bn.bias', 'trillsson.block.11.expand_bn.bias', 'trillsson.block.30.se.se_reduce.bias', 'trillsson.block.15.expand_bn.bias', 'trillsson.stem.stem_bn.running_var', 'trillsson.block.27.project_bn.running_var', 'trillsson.block.4.project_bn.weight', 'trillsson.block.31.se.se_reduce.weight', 'trillsson.block.17.project_bn.running_var', 'trillsson.block.15.se.se_reduce.weight', 'trillsson.block.16.project_bn.running_var', 'trillsson.block.17.expand_bn.weight', 'trillsson.block.23.expand_bn.weight', 'trillsson.block.6.project_bn.weight', 'trillsson.block.21.bn.weight', 'trillsson.block.8.expand_bn.num_batches_tracked', 'trillsson.block.31.expand_bn.bias', 'trillsson.block.35.se.se_reduce.bias', 'trillsson.stem.stem_bn.weight', 'trillsson.block.22.se.se_reduce.weight', 'trillsson.block.27.expand_bn.running_mean', 'trillsson.block.3.project_bn.running_var', 'trillsson.block.36.se.se_expand.bias', 'trillsson.block.13.se.se_expand.bias', 'trillsson.block.38.se.se_expand.weight', 'trillsson.block.35.project_bn.bias', 'trillsson.block.39.bn.running_var', 'trillsson.block.29.expand_bn.bias', 'trillsson.block.34.project_bn.bias', 'trillsson.block.16.expand_conv.weight', 'trillsson.block.32.se.se_reduce.bias', 'trillsson.block.32.project_bn.running_mean', 'trillsson.block.9.expand_conv.weight', 'trillsson.block.39.expand_bn.num_batches_tracked', 'trillsson.block.30.se.se_reduce.weight', 'trillsson.block.5.expand_bn.bias', 'trillsson.block.11.bn.bias', 'trillsson.block.18.project_bn.running_var', 'trillsson.block.39.se.se_reduce.bias', 'trillsson.block.33.bn.bias', 'trillsson.block.10.bn.running_mean', 'trillsson.block.33.expand_bn.running_var', 'trillsson.block.10.expand_bn.weight', 'trillsson.block.28.project_conv.weight', 'trillsson.block.12.expand_bn.weight', 'trillsson.block.3.project_bn.weight', 'trillsson.block.25.project_bn.num_batches_tracked', 'trillsson.block.10.bn.weight', 'trillsson.block.23.project_bn.bias', 'trillsson.block.32.project_bn.num_batches_tracked', 'trillsson.block.11.bn.running_var', 'trillsson.block.31.bn.weight', 'trillsson.block.38.expand_conv.weight', 'trillsson.block.18.project_bn.num_batches_tracked', 'trillsson.stem.stem_bn.bias', 'trillsson.block.15.se.se_expand.weight', 'trillsson.block.15.bn.num_batches_tracked', 'trillsson.block.24.expand_bn.running_mean', 'trillsson.block.36.dwconv2.weight', 'trillsson.block.33.bn.running_mean', 'trillsson.block.33.se.se_reduce.weight', 'trillsson.block.27.expand_conv.weight', 'trillsson.block.3.expand_bn.bias', 'trillsson.block.30.expand_bn.bias', 'trillsson.block.37.expand_conv.weight', 'trillsson.block.22.project_bn.bias', 'trillsson.block.2.project_conv.weight', 'trillsson.block.38.se.se_expand.bias', 'trillsson.block.32.project_bn.bias', 'trillsson.block.17.se.se_reduce.weight', 'trillsson.block.18.bn.running_mean', 'trillsson.block.18.expand_conv.weight', 'trillsson.block.38.project_conv.weight', 'trillsson.block.25.project_bn.running_mean', 'trillsson.block.16.bn.num_batches_tracked', 'trillsson.block.29.bn.running_mean', 'trillsson.block.30.project_conv.weight', 'trillsson.block.20.bn.num_batches_tracked', 'trillsson.block.0.project_bn.running_var', 'trillsson.block.9.project_bn.num_batches_tracked', 'trillsson.block.4.expand_bn.running_mean', 'trillsson.block.24.bn.running_var', 'trillsson.block.7.project_conv.weight', 'trillsson.block.35.project_conv.weight', 'trillsson.block.20.dwconv2.weight', 'trillsson.block.18.bn.num_batches_tracked', 'trillsson.block.30.expand_bn.running_var', 'trillsson.block.37.bn.bias', 'trillsson.block.34.bn.bias', 'trillsson.block.2.project_bn.num_batches_tracked', 'trillsson.top.top_conv.weight', 'trillsson.block.0.project_bn.bias', 'trillsson.block.32.project_bn.running_var', 'trillsson.block.15.project_bn.running_var', 'trillsson.block.2.expand_bn.num_batches_tracked', 'trillsson.block.13.dwconv2.weight', 'trillsson.block.26.project_bn.bias', 'trillsson.block.35.expand_bn.bias', 'trillsson.block.8.expand_bn.running_mean', 'trillsson.block.35.project_bn.num_batches_tracked', 'trillsson.block.2.expand_conv.weight', 'trillsson.block.17.project_bn.weight', 'trillsson.block.22.bn.num_batches_tracked', 'trillsson.block.24.bn.running_mean', 'trillsson.block.3.expand_bn.running_var', 'trillsson.block.34.expand_bn.bias', 'trillsson.block.15.project_bn.bias', 'trillsson.block.5.project_bn.running_mean', 'trillsson.block.28.expand_bn.num_batches_tracked', 'trillsson.block.21.expand_bn.running_mean', 'trillsson.block.26.bn.bias', 'trillsson.block.30.project_bn.bias', 'trillsson.block.35.expand_bn.running_var', 'trillsson.block.27.project_bn.weight', 'trillsson.block.12.project_bn.num_batches_tracked', 'trillsson.block.23.project_conv.weight', 'trillsson.block.34.bn.running_mean', 'trillsson.block.17.project_bn.num_batches_tracked', 'trillsson.block.12.expand_bn.bias', 'trillsson.block.17.project_conv.weight', 'trillsson.block.39.expand_bn.weight', 'trillsson.block.16.project_bn.num_batches_tracked', 'trillsson.block.33.expand_bn.weight', 'trillsson.block.12.project_bn.weight', 'trillsson.block.26.expand_bn.num_batches_tracked', 'trillsson.block.39.dwconv2.weight', 'trillsson.block.39.project_conv.weight', 'trillsson.block.14.project_bn.bias', 'trillsson.block.31.project_bn.weight', 'trillsson.block.33.project_bn.bias', 'trillsson.block.27.se.se_reduce.bias', 'trillsson.block.14.bn.bias', 'trillsson.block.28.expand_bn.running_mean', 'trillsson.block.32.bn.num_batches_tracked', 'trillsson.block.2.project_bn.weight', 'trillsson.block.11.bn.running_mean', 'trillsson.block.23.bn.running_var', 'trillsson.block.37.expand_bn.bias', 'trillsson.block.33.project_bn.num_batches_tracked', 'trillsson.block.36.project_bn.weight', 'trillsson.block.38.project_bn.weight', 'trillsson.block.24.expand_bn.running_var', 'trillsson.block.36.expand_bn.weight', 'trillsson.block.13.bn.bias', 'trillsson.block.5.project_bn.running_var', 'trillsson.block.28.bn.bias', 'trillsson.block.16.bn.running_var', 'trillsson.block.30.expand_bn.num_batches_tracked', 'trillsson.block.28.project_bn.weight', 'trillsson.block.3.expand_conv.weight', 'trillsson.block.39.project_bn.running_mean', 'trillsson.block.9.project_conv.weight', 'trillsson.block.38.expand_bn.running_var', 'trillsson.block.8.project_bn.bias', 'trillsson.block.17.expand_bn.bias', 'trillsson.block.23.se.se_reduce.weight', 'trillsson.block.24.bn.num_batches_tracked', 'trillsson.block.13.expand_bn.weight', 'trillsson.block.19.se.se_reduce.bias', 'trillsson.block.27.se.se_reduce.weight', 'trillsson.block.6.project_bn.running_mean', 'trillsson.block.21.expand_conv.weight', 'trillsson.block.39.se.se_expand.bias', 'trillsson.block.34.project_bn.running_mean', 'trillsson.block.29.bn.num_batches_tracked', 'trillsson.block.7.expand_bn.num_batches_tracked', 'trillsson.block.21.bn.running_var', 'trillsson.block.13.se.se_reduce.weight', 'trillsson.block.35.expand_conv.weight', 'trillsson.block.14.se.se_expand.weight', 'trillsson.block.20.bn.weight', 'trillsson.block.21.project_bn.running_mean', 'trillsson.block.4.project_conv.weight', 'trillsson.block.26.se.se_expand.weight', 'trillsson.block.16.project_bn.bias', 'trillsson.top.top_bn.bias', 'trillsson.block.39.bn.bias', 'trillsson.block.11.expand_bn.num_batches_tracked', 'trillsson.block.38.project_bn.bias', 'trillsson.block.7.expand_bn.bias', 'trillsson.block.33.project_bn.running_var', 'trillsson.block.4.project_bn.running_var', 'trillsson.block.20.bn.running_mean', 'trillsson.block.30.dwconv2.weight', 'trillsson.block.3.expand_bn.num_batches_tracked', 'trillsson.block.21.dwconv2.weight', 'trillsson.block.22.se.se_expand.bias', 'trillsson.block.11.se.se_reduce.weight', 'trillsson.block.37.project_bn.weight', 'trillsson.block.30.se.se_expand.weight', 'trillsson.block.33.project_bn.running_mean', 'trillsson.block.15.se.se_expand.bias', 'trillsson.block.34.se.se_reduce.weight', 'trillsson.block.13.project_bn.bias', 'trillsson.block.22.project_bn.running_mean', 'trillsson.block.31.project_bn.running_mean', 'trillsson.block.5.expand_bn.running_mean', 'trillsson.block.32.se.se_expand.bias', 'trillsson.block.34.project_conv.weight', 'trillsson.block.13.bn.running_var', 'trillsson.block.24.se.se_expand.bias', 'trillsson.block.25.expand_bn.running_mean', 'trillsson.block.27.expand_bn.num_batches_tracked', 'trillsson.block.12.project_bn.bias', 'trillsson.block.12.bn.running_mean', 'trillsson.block.3.project_bn.bias', 'trillsson.block.11.expand_bn.running_var', 'trillsson.block.11.se.se_expand.weight', 'trillsson.block.12.se.se_reduce.weight', 'trillsson.block.28.expand_conv.weight', 'trillsson.block.30.bn.running_var', 'trillsson.block.18.project_bn.weight', 'trillsson.block.10.se.se_reduce.bias', 'trillsson.block.7.project_bn.running_mean', 'trillsson.block.29.bn.weight', 'trillsson.block.34.project_bn.weight', 'trillsson.block.16.expand_bn.num_batches_tracked', 'trillsson.block.33.se.se_reduce.bias', 'trillsson.block.12.expand_bn.running_var', 'trillsson.block.23.project_bn.running_var', 'trillsson.block.24.project_bn.bias', 'trillsson.block.20.project_conv.weight', 'trillsson.block.15.project_bn.running_mean', 'trillsson.block.11.expand_bn.running_mean', 'trillsson.block.15.bn.running_var', 'trillsson.block.12.dwconv2.weight', 'trillsson.block.39.project_bn.weight', 'trillsson.block.27.expand_bn.running_var', 'trillsson.block.6.expand_bn.num_batches_tracked', 'trillsson.block.16.bn.weight', 'trillsson.block.22.bn.running_var', 'trillsson.block.22.bn.weight', 'trillsson.block.34.expand_bn.running_mean', 'trillsson.block.22.expand_bn.bias', 'trillsson.block.20.se.se_reduce.weight', 'trillsson.block.21.bn.running_mean', 'trillsson.block.14.project_bn.running_var', 'trillsson.block.19.dwconv2.weight', 'trillsson.block.24.expand_conv.weight', 'trillsson.block.28.expand_bn.running_var', 'trillsson.dense.weight', 'trillsson.block.9.project_bn.weight', 'trillsson.block.19.expand_bn.bias', 'trillsson.block.12.project_bn.running_var', 'trillsson.block.21.se.se_reduce.bias', 'trillsson.block.23.bn.running_mean', 'trillsson.block.11.project_bn.running_var', 'trillsson.block.25.se.se_expand.weight', 'trillsson.block.14.se.se_reduce.weight', 'trillsson.block.14.expand_bn.running_mean', 'trillsson.block.34.se.se_expand.weight', 'trillsson.block.17.bn.running_mean', 'trillsson.block.31.expand_conv.weight', 'trillsson.block.36.project_bn.running_mean', 'trillsson.block.14.expand_conv.weight', 'trillsson.block.27.expand_bn.bias', 'trillsson.block.6.expand_bn.weight', 'trillsson.block.37.project_bn.bias', 'trillsson.block.27.se.se_expand.weight', 'trillsson.block.30.project_bn.weight', 'trillsson.block.6.project_conv.weight', 'trillsson.block.8.project_bn.weight', 'trillsson.block.22.project_conv.weight', 'trillsson.block.11.se.se_reduce.bias', 'trillsson.block.17.expand_bn.running_mean', 'trillsson.block.17.project_bn.bias', 'trillsson.stem.stem_bn.running_mean', 'trillsson.block.29.se.se_reduce.weight', 'trillsson.block.37.se.se_expand.weight', 'trillsson.block.32.project_bn.weight', 'trillsson.block.8.project_bn.running_var', 'trillsson.block.37.dwconv2.weight', 'trillsson.block.19.project_bn.weight', 'trillsson.block.30.expand_conv.weight', 'trillsson.block.19.bn.weight', 'trillsson.block.13.expand_conv.weight', 'trillsson.block.30.expand_bn.weight', 'trillsson.block.36.expand_bn.running_var', 'trillsson.block.37.expand_bn.num_batches_tracked', 'trillsson.block.21.se.se_reduce.weight', 'trillsson.block.19.project_bn.running_mean', 'trillsson.block.29.expand_bn.running_mean', 'trillsson.block.13.expand_bn.bias', 'trillsson.block.14.project_conv.weight', 'trillsson.block.31.se.se_expand.weight', 'trillsson.top.top_bn.running_var', 'trillsson.block.19.expand_bn.running_mean', 'trillsson.block.30.bn.bias', 'trillsson.block.16.expand_bn.running_mean', 'trillsson.block.35.dwconv2.weight', 'trillsson.block.28.se.se_expand.weight', 'trillsson.block.15.expand_bn.running_mean', 'trillsson.block.7.expand_bn.running_var', 'trillsson.block.26.expand_conv.weight', 'trillsson.block.14.expand_bn.num_batches_tracked', 'trillsson.block.24.bn.weight', 'trillsson.block.19.expand_bn.weight', 'trillsson.top.top_bn.num_batches_tracked', 'trillsson.block.21.project_bn.bias', 'trillsson.block.23.se.se_expand.weight', 'trillsson.block.18.bn.running_var', 'trillsson.block.10.expand_conv.weight', 'trillsson.block.12.bn.running_var', 'trillsson.block.25.se.se_reduce.bias', 'trillsson.block.24.project_conv.weight', 'trillsson.block.38.expand_bn.running_mean', 'trillsson.block.23.project_bn.weight', 'trillsson.block.13.project_conv.weight', 'trillsson.block.13.se.se_reduce.bias', 'trillsson.block.31.expand_bn.running_mean', 'trillsson.block.24.se.se_expand.weight', 'trillsson.block.19.expand_bn.num_batches_tracked', 'trillsson.block.23.se.se_reduce.bias', 'trillsson.block.15.bn.weight', 'trillsson.block.24.project_bn.num_batches_tracked', 'trillsson.block.38.se.se_reduce.weight', 'trillsson.block.25.bn.num_batches_tracked', 'trillsson.block.10.expand_bn.running_var', 'trillsson.block.5.expand_bn.num_batches_tracked', 'trillsson.block.20.expand_bn.running_mean', 'trillsson.block.24.dwconv2.weight', 'trillsson.block.34.se.se_expand.bias', 'trillsson.block.7.project_bn.num_batches_tracked', 'trillsson.block.35.bn.running_var', 'trillsson.block.10.bn.bias', 'trillsson.block.30.bn.weight', 'trillsson.block.12.se.se_expand.weight', 'trillsson.block.20.project_bn.weight', 'trillsson.top.top_bn.weight', 'trillsson.block.1.project_bn.bias', 'trillsson.block.14.project_bn.weight', 'trillsson.block.9.project_bn.running_var', 'trillsson.block.34.bn.num_batches_tracked', 'trillsson.block.13.project_bn.running_var', 'trillsson.block.29.se.se_expand.bias', 'trillsson.block.32.bn.running_var', 'trillsson.block.1.project_bn.weight', 'trillsson.block.33.expand_bn.running_mean', 'trillsson.block.34.expand_bn.running_var', 'trillsson.block.0.project_bn.running_mean', 'trillsson.block.35.expand_bn.num_batches_tracked', 'trillsson.block.12.expand_bn.num_batches_tracked', 'trillsson.block.32.expand_conv.weight', 'trillsson.block.14.expand_bn.running_var', 'trillsson.block.35.bn.num_batches_tracked', 'trillsson.block.31.expand_bn.running_var', 'trillsson.block.25.bn.running_var', 'trillsson.block.23.expand_bn.running_mean', 'trillsson.block.32.se.se_expand.weight', 'trillsson.block.18.expand_bn.running_var', 'trillsson.block.33.expand_conv.weight', 'trillsson.block.18.se.se_expand.bias', 'trillsson.block.10.project_bn.running_mean', 'trillsson.block.14.se.se_reduce.bias', 'trillsson.block.21.project_conv.weight', 'trillsson.block.22.expand_conv.weight', 'trillsson.block.32.expand_bn.running_var', 'trillsson.block.36.se.se_reduce.bias', 'trillsson.block.35.bn.bias', 'trillsson.block.26.project_bn.weight', 'trillsson.block.25.dwconv2.weight', 'trillsson.block.36.bn.running_var', 'trillsson.block.19.expand_conv.weight', 'trillsson.block.23.expand_bn.running_var', 'trillsson.block.11.project_bn.bias', 'trillsson.block.30.se.se_expand.bias', 'trillsson.block.35.project_bn.running_mean', 'trillsson.block.37.project_bn.num_batches_tracked', 'trillsson.block.15.expand_bn.weight', 'trillsson.block.28.se.se_expand.bias', 'trillsson.block.31.bn.running_mean', 'trillsson.block.18.se.se_reduce.bias', 'trillsson.block.34.project_bn.running_var', 'trillsson.block.38.expand_bn.num_batches_tracked']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference. 
```
Do you have any ideas how to fix this problem?

OKay! Will have a look at that with my review 😉 

Also I think we should probably specify which architecture is actually being implemented here as TRILLsson introduced 3 student networks, and I am mostly seing 2 variations in architectures here. (so which one gives Resnetish or efficient net). Correct me if I am wrong! 
- Audio Spectrogram Transformer (AST) is already included in HF, but we could probably convert the checkpoints of this model and add it to the hub easily ! (the config should be enough to change the depths and widths) 


Thank you very much for reviewing my code :heart: . I will check all of them

Sorry for the late response and thanks again for reviewing my code @ArthurZucker. I revised it as per your suggestion. Regarding your question, Trillsson has 3 different student architectures but here I just implement only one, efficientnetv2. After that, we can expand to other variants in the future.

Feel free to ping me again for a review

@ArthurZucker can you review it again 

Update: I am really sorry, but EfficientNet has been merged to transformers! SOrry because it means some of your hard work goes to the drain (a little bit)! But should ease the addition of your models! See this [PR](https://github.com/huggingface/transformers/pull/21563/). You can use this implementation by relying on `# copied from` to get the efficien net code in your code!  

@ArthurZucker Thank you for letting me know. I think I need a bit of time to investigate the structure of merged EfficientNet to load weight correctly.