Model to translate between Norwegian Bokmål and Norwegian Nynorsk

Question:
Norwegian has two different written standards or variants, Bokmål and Nynorsk. Some mean they should be considered two different languages, but with extensive lexical as well as grammatical overlap. In many cases a word-by-word translation suffices, but there are for instance also some auxiliary and voice constructions that are different in addition to differing pronoun systems. Students and civil servants are expected to be fluent in both languages and you have as a Norwegian citizen the right to receive public documents in your preferred language. Nynorsk is by far the minority, used in less than 20% of the cases.
Translation between Bokmål and Nynorsk seems like a low hanging machine translation task, especially since the loss can be calculated more effectively. Even not translating at all, will probably give you a BLEU score in the area of 20-30 because of the overlaps.
There are no recent machine translation-models trying to do this translation task, most likely due to the lack of available corpora. We are however able to make such a dataset available for this workshop.

Language
We aim at using as much of the available Flax code as possible.

Model
To be decided by the team. To our knowledge, both BERT-infused NML-models, T5-like models and BART might be suitable for such a task. Model choice might be influenced based on what modules are available in FLAX and performance.

Datasets
For this workshop we are able to make publicly available two pre-prepared datasets collected though other projects:

A balanced (5+5=10GB) corpus in Bokmål and Nynorsk. Collected through various sources, MC4, Oscar, Wikipedia, public documents, pdf-scans. There is even more text available in Bokmål here, but we feel that a balanced set would be best for this task.
A high quality parallel corpus of 100.000 sentence-pairs in Bokmål and Nynorsk.

The sets are already thoroughly cleaned, and we will make sure they are ready to import by the start of the workshop.

Training scripts
The goal is to use the current scripts as much as possible due to the short timespan.

Challenges
The first challenge is to find a model architecture that can give a reasonable result in this timeframe. It is however reasonable to assume that this will be a much easier task than translating from for instance English to German.
For the project to be realistic, we need team members with previous experience with machine translation. The proposers have solid experience in training large language models, but lack experience with translation. Since the timeframe for the workshop is short, we should find suitable models already implemented in Flax.
No knowledge of Norwegian is necessary.

Desired project outcome
It is a tech demo, hoping to show that it is doable to build a mainly unsupervised machine learning algorithm with Flax in a week.

Context: This is an interesting idea, but I wonder if we’d had enough data to test a simple fine-tuning first.


Question:
Description: We will pretrain a large BART model on The Pile, and measure a performance increase downstream. Potentially we could also add rotary embeddings?
Model: BART (1b+)
Dataset: The Pile
Training scripts: Training scripts will be written as part of the project. Data processing scripts can be taken from GPT-J 6b
Expected result: An adaptable JAX pipeline for training seq2seq models like BART on the pile.

Context: Im Interested in this project, Any reasons why you’d aim for BART and not any model with some architectural improvements?


Question:
Although JAX is faster than the others via jit compiling, another strong point of JAX is intuitive autograd.
This project will try to exploit an advantage of JAX autograd while reproducing Neural ODE and Neural SDE which is suitable for this purpose.

This project will use Imagenet and CIFAR-10 dataset.
The models to reproduce can be found in below papers,
– Neural ODE https://arxiv.org/pdf/1806.07366.pdf 24
– Score-Based Generative Modeling through Stochastic Differential Equations | OpenReview 24

Please feel free to suggest any opinion about this project
Some of you might think this project requires mathematical understanding and harder than the others.
However, there are plenty of materials we can help you understand its math background such as

implicit-layers-tutorial at ICML20’
so do not hesitate

Current team members:
sw32-seo
@wgchang
@lenscloth
Thanks,
Seung-woo Seo

Context:
Awesome! the intuitive autograd of JAX would be very useful while implementing Neural ODE or SDE.
I would like to contribute the project


Question:
PreTrain Wav2Vec2 in Spanish
There is currently only a multilingually pretrained model for Spanish Wav2Vec2. Let’s make a Wav2Vec2 only pretrained on Spanish.

Model
A randomly initialized Wav2Vec2 model.

Datasets
We can use the Spanish portion of Common Voice 2. The dataset is available through the datasets library here: common_voice · Datasets at Hugging Face 1.

Available training scripts
FlaxWav2Vec2 will be merged soon: [Flax] Add wav2vec2 by patrickvonplaten · Pull Request #12271 · huggingface/transformers · GitHub  3 and a pretraining script should be relatively easy to be merged.

(Optional) Desired project outcome
The best Spanish ASR model.

(Optional) Challenges
It would be nice to use more data than just the Common Voice dataset.

Context:
I am in!
cc: @patrickvonplaten @valhalla


Question:
GPT2 for Russian
Currently, there is no GPT2 model that was trained from scratch for Turkish on the hub: Hugging Face – The AI community building the future. 3 . For this project, the goal is to create a strong language generation model for Russian.

Model
A randomly initialized GPT2 model

Datasets
One can make use OSCAR 2 the dataset is also available through the datasets library here: oscar · Datasets at Hugging Face 1.

Available training scripts
A causal language modeling script for Flax is available here 2. It can be used pretty much without any required code changes.

(Optional) Desired project outcome
The desired project output is a GPT2 model that is able to generate Russian language. A nice generation demo can be created for this.

(Optional) Challenges
The dataset on OSCAR is very large: > 300GB. One might want to explore dataset streaming techniques here. Dataset streaming will be merged to datasets in a couple of days. See PR here 2 and the docs here 3.

(Optional) Links to read upon
The most important read would be the following colab:

Google Colaboratory 4

Context: I’m interested.


Question:
GPT2 for Spanish
Currently, there is no GPT2 model that was trained from scratch for Spanish on the hub: Hugging Face – The AI community building the future. 3 . For this project, the goal is to create a strong language generation model for Spanish.

Model
A randomly initialized GPT2 model

Datasets
One can make use OSCAR 4 the dataset is also available through the datasets library here: oscar · Datasets at Hugging Face 3.

Available training scripts
A causal language modeling script for Flax is available here 8. It can be used pretty much without any required code changes.

(Optional) Desired project outcome
The desired project output is a GPT2 model that is able to generate Spanish language.

(Optional) Challenges
The dataset on OSCAR is quite large → around 150 GB. One might want to try out dataset streaming to solve this problem. Dataset streaming will be merged to datasets in a couple of days. See PR here 2 and the docs here 4.

(Optional) Links to read upon
The most important read would be the following colab:

Google Colaboratory 6

Other reads that might be interesting include:

GPT-2 no habla español: Artificial Intelligence, Anglocentrism, and the non-human side of DH | | Vanderbilt University 4
https://towardsdatascience.com/train-gpt-2-in-your-own-language-fc6ad4d60171 6

Context: I’m definitely in! Check this @mrm8488 it’s exactly what we talked about


Question:
Hello all,
I have tried my best to organize FLAX/JAX learning resources. I hope it will be useful for all of us.
If you have more learning resources, drop a comment & I will add them here…
JAX 3 is a numerical computation library that exposes a NumPy-like API with tracing capabilities.
Flax is the most widely used JAX library, with 129 dependent projects 9 as of May 2021.
Flax builds on top of JAX with an ergonomic module abstraction using Python dataclasses that leads to concise and explicit code.
Flax’s “lifted” JAX transformations (e.g. vmap , remat ) allow you to nest JAX transformation and modules in any way you wish.  It is also the library underlying all of the official Cloud TPU JAX examples.
Github links :
JAX Projects 5
FLAX examples 28
Colab links:
Causal Language Modelling on OSCAR 3
Masked Language Modelling on OSCAR  7
Text Classification on GLUE 5
Docs:
JAX 101 Tutorial 21
Flax Basics Colab 17
Flax examples 18
More useful links:
Guide to run JAX on Google Cloud TPU 4.
Load Dataset in streaming mode 9
TALKS Date & Time:

30.6.2021  :  9am - 11am PST  Zoom link 6

1.7.2021    :  8:30am - 10am PST   Zoom link 3

2.7.2021    :  8am - 10am PST  Zoom link

Context: Great summary - thanks a lot @srisweet !


Question:
Segmentation of COVID-19 Chest CT Scans using TransUNet
Transformers have shown great performance in the field of Image Segmentation. It would be amazing to train them using JAX/Flax. The goal of this project is automatic COVID-19 lesion segmentation from chest CT.

2. Language
The model will be trained in English.

3. Model
The TransUNet model will be used for segmentation.

4. Datasets
Possible links to publicly available datasets include:

dataset-01: https://doi.org/10.5281/zenodo.3757476 4

dataset-02: COVID-19 - Medical segmentation 4

5. Training scripts
Are there publicly available training scripts that can be used/tweaked for the project?
Answer: Unfortunately, No.

6. (Optional) Challenges

Build TransUnet model in JAX/Flax
Implement Augmentation for Segmentation.

7. (Optional) Desired project outcome

This project will help clinicians segment COVID-19 lesions which will help clinicians identify potential COVID-19 cases.

TransUNet can be later on other segmentation tasks.

Context:
I’d love to join this project!
I’ve written many UNets and data augmentations in both TF and PyTorch as part of my daily job and could bring that experience to the team, helping with the model or data augmentation as needed.
On the other hand I haven’t seen much Jax, dealt with many medical datasets, nor combined transformers with UNet architectures – all of which I’d love to correct. The project outcome is also clearly beneficial, although one should point out that recent analyses have found much Covid-related CV research quite problematic (Nature article 2).
I live in the London timezone (BST, GMT+1).


Question: Potentially try out a number of model architectures (t5/roberta/gpt2/bigbird…) using datasets such as Oscar, mc5, gdelt,… & test with a fine-tuned task

Context: I really like this idea, especially since models on Swahili are quite sparse! Do you think we could settle on one model? It might make the project much easier


Question:
GPT2 for Polish
Currently, there is no GPT2 model that was trained from scratch for Polosh on the hub. For this project, the goal is to create a strong language generation model for Polish.

Model
A randomly initialized GPT2 model

Datasets
Pre-training corpora that was used to train Polish Bert by Darek Kłeczek.
Uncased
Polish subset of Open Subtitles
Polish subset of ParaCrawl
Polish Parliamentary Corpus
Polish Wikipedia
Cased
Polish subset of Open Subtitles
Polish subset of ParaCrawl
Polish Parliamentary Corpus
Polish Wikipedia - Feb 2020
One can make use OSCAR the dataset is also available through the datasets library - Datasets at Hugging Face.

Available training scripts
A causal language modeling script for Flax is available here 6. It can be used pretty much without any required code changes.

(Optional) Desired project outcome
The desired project output is a GPT2 model that is able to generate Polish language. A nice generation demo can be created for this.

(Optional) Challenges
It might be possible that there is not enough data for the model to perform reasonably well on text generation. In this case, one would have to look at other datasets as well, like mc4 .

(Optional) Links to read upon
The most important read would be the following colab:

Google Colaboratory 7

Polish BERT by Darek Kłeczek

Context: Noting this down since Darek will be joining the project!


IndoClip : Pre Training Clip for Indonesian dataset

Question:
CLIP like contrastive vision-language models for Indonesian with pre-trained text and vision models
For this project, a pre-trained image model like ViT and a pre-trained text model like BERT can be used as an image encoder and text encoder respectively.

Model
Pre-trained ViT, BERT models can be found on the model hub. We could also use multi-lingual BERT/RoBERTa models for the Indonesian language.

Datasets
The WIT 3 dataset can be used for this task.
The GEM 3 dataset can also be used for the task

Available training scripts
A training script for this will be provided soon. (see PR 3)

(Optional) Desired project outcome
The desired outcome is to train a CLIP-like model for Indonesian language. This can be showcased with a streamlit or gradio app.

(Optional) Challenges
This model will require some modifications to the existing models. Specifically, we will need to add projection layers in both the text and image encoder models.

Context: Hello @munggok .Are you still looking for team members for this project? I’m very interested with this project


Train hybrid FNet for generation (FNet: Mixing Tokens with Fourier Transforms))

Question:
Background
The FNet is a efficient transformer model that uses Fourier Transform to replace self-attention.
The paper compare it with bert, and showed it “trains nearly seven times faster on GPUs and twice as fast on TPUs”, it’s also more memory efficient and faster for inference.
The authors released their code and pre-trained weights recently: (It uses flax!)

      github.com

    google-research/google-research 3

  master/f_net

  Google Research. Contribute to google-research/google-research development by creating an account on GitHub.

However, it’s not yet capable of generation.
Here is the last paragraph of the FNet paper.

Throughout this work we have restricted our attention to encoders. FNet decoders can be designed
using causally masked discrete Fourier Transform matrices. However, in settings where the FFT is preferable, a lower level implementation would be required to introduce causal masking. Lastly, designing the equivalent of encoder-decoder cross-attention remains an avenue for future work, as
evidence suggests that it is crucial to decoder performance (You et al., 2020
[2105.03824] FNet: Mixing Tokens with Fourier Transforms 1

Proposal
Here is the main thing I want to try:

Use a hybrid encoder-decoder where the encoder is FNet, and the decoder can be a normal transformer with attention. The motivation is to take advantage of the more efficient architecture and augment it with generation capability.

Details

Language: monolingual, english
Model: encoder-decoder, replaces encoder self-attention with Fourier Transform.
Data: c4 or the pile
Training scripts: available in the repo above.
What are possible challenges? The code is encoder only, we need to add the decoder part.
Desired project outcome: We will train an efficient model for generation tasks.

Discussion
Some more details for discussion:

The ideal loss could be prefix-LM or bart-like, so it can generate longer text better. We can compare the prefix-LM loss/inference cost with GPT2. (Similar to making GPT2 faster by encoding prefix with FNet.)
We can first train the hybrid model and initialize the encoder with pre-trained weight and make sure our code is correct (we need to write some modeling code).
Later we can train the model from scratch, which we can follow T5 model sizes, and see how it compares. (should be more efficient, we can try to scale it slightly different to match the compute cost/performance)

Other easier things we can do:

Reproduce it.
Train it on the Pile (the original one is trained on C4).

Note

The model is not yet on huggingface/transformers, but I would like to help adding it. 🌟 New model addition: FNet · Issue #12411 · huggingface/transformers · GitHub 1

Context:
Hi @ttj
Very interesting idea. I am planning to implement the FNet in PyTorch. I made a PR a few days back and I’m yet to update here: [WIP] FNet by gchhablani · Pull Request #12335 · huggingface/transformers · GitHub 4. Not sure if Flax would be easier.


Question:
After some brainstorming, we came with the following project ideas. Would love some feedback/opinion on the same.

ViT + mBART - Multilingual Image Captioning (WIT pre-train)
ViT + mBERT - Multilingual Visual Question Answering (WIT/COCO pre-train, test on VQA/GQA)
Use CLIP/VQGAN for Image Synthesis. A project has been proposed GIF generation. We can train on a different dataset/domain.

Possible modifications of CLIP/VQGAN:

Multilingual CLIP model for image/sentence matching + Image Generation using VQGAN for this dataset.
FashionCLIP- Train a model for Fashion image-text matching and VQGAN, trained to generate dress/shirt/glasses based on description. Dataset is hard to find, I guess.
Scene generation - CLIP + VQGAN, trained on text-scene dataset for scene (movie/landscape) generation.

Alternatively, can do similar things for video+text, following the example of VideoBERT and other transformers, maybe?
Please comment here if you are interested in collaboration. We are a team of 5 as of now (timezone GMT+5:30). Would also love any suggestions to improve these ideas.
Team:

@abheesht
@bhavitvyamalik
@Divyadixit
@gchhablani
@harsh68

Thanks,
Gunjan

Context: @gchhablani very interesting ideas, I have experience in CV and NLP and would like to contribute in this project.


Question:
KoCLIP
Building on top of Korean Language models that are publicly available, we want to train a multimodal generative system. Specifically, we train CLIP on Korean datasets using KoBERT 10 and ViT as backbones.

Model
ViT, KoBERT (or any other Korean encoder LM)

Datasets
KETI has released a Korean image captioning dataset, available on AI Hub 11. We can also utilize the WIT dataset 13, which is a multilingual dataset scraped from Wikipedia.

Training Script
The training script is (almost) already available, see here 34.

Challenges

KoBERT likely performs slightly worse than the default English BERT. While this could be a bottleneck, we can always write the script in such a way that it is easy to plug-and-play different LMs. This way, when a better Korean LM is released, it can easily be used.
There might be minor architectural adjustments we have to make (e.g. adding projection layers).
Attaining fluency in JAX will take time and effort.

Desired Outcomes
The final deliverable of this project will most likely be an open source repository, accompanying documentation, model weights, and potentially a demo Streamlit app.

Context:
Awesome Idea! I’m also very interested in Korean NLP:)
Count me in!


Question:
PreTrain ProteinBERT from scratch
ProteinBERT is a universal deep-learning model of protein sequence and function based on the BERT architecture. The goal of this project is to pretrain the ProteinBERT model in JAX/Flax for downstream finetuning tasks like predicting protein structure, post translational modifications and/or biophysical attributes. The model itself can be seen as an extension to the Protein Transformer models trained by the Rostlab 2.

2. Language
The model will be trained on protein sequences and gene ontology (GO) annotations.

3. Model
The classic Transformer/BERT architecture, with some additions.

4. Datasets
ProteinBERT is pretrained on a dataset derived from UniRef90 which consists of ~106M protein sequences. Protein Gene Ontology (GO) annotations are used as additional inputs (to help the model infer about the function of the input protein and update its internal representations and outputs accordingly).
Possible links to publicly available datasets include:

GO annotations from CAFA
UniRef90 XML
UniRef90 FASTA

5. Training scripts
Data preprocessing scripts are available on the Official Github Repo 7. The training script can be adapted from run_mlm_flax.py.

6. Challenges

The data of protein sequences and GO annotations require ~1 TB of scratch disk space.
The original paper states a pretraining time of ~28 days on a single GPU (Nvidia Quadro RTX 5000).
Besides NLP skills there may be some bioinformatics skills required

7. Desired project outcome
The model that can be further finetuned to predict protein structure, post translational modifications, and/or biophysical attributes.

8. Reads
The following links can be useful to better understand the project and
what has previously been done.

ProteinBERT article 3
Official Github Repo 7
Work in progress porting ProteinBERT to JAX by Saurav Maheshkar 3

Context:
Hi @timothyman
I find the idea very intriguing and will like to contribute to it. I have previously studied and explored the ProteinBERT architecture and definitely want to be a part of it.


Question:
Pretrain GPT2 from scratch in Romanian
Train a Romanian GPT2 on a 24 GB Romanian dataset

2. Language
Currently, there is no GPT2 model that was trained from scratch for Romanian on the hub.
For this project, the goal is to create a strong language generation model for Romanian.

3. Model
A randomly initialized GPT2 model

4. Datasets
We created a dataset from OSCAR, text from books and web crawled datasets.
The entire dataset has 24 GB.

5. Training scripts
A causal language modeling script for FLAX is available here:

      github.com

    huggingface/transformers/blob/master/examples/flax/language-modeling/run_clm_flax.py

    #!/usr/bin/env python
# coding=utf-8
# Copyright 2021 The HuggingFace Team All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""
Pre-training/Fine-tuning the library models for causal language modeling (GPT, GPT-2, CTRL, ...) on a text file or a dataset.

Here is the full list of checkpoints on the hub that can be fine-tuned by this script:
https://huggingface.co/models?filter=causal-lm

  This file has been truncated. show original

6. (Optional) Challenges
<(Optionally) FILL ME: 6. What are possible challenges?>

7. (Optional) Desired project outcome
There are no causal language models for Romanian language. This will be the first causal model for Romanian language

8. (Optional) Reads
The following links can be useful to better understand the project and
what has previously been done.

      colab.research.google.com

Google Colaboratory 6

Context: This is interesting, as it would be the first Romanian model that can be used for text generation. I highly support this project!


Question:
CLIP like contrastive vision-language models for Spanish with pre-trained text and vision models
For this project, a pre-trained image model like ViT and a pre-trained text model like BERT can be used as an image encoder and text encoder respectively.

Model
Pre-trained ViT 3, BERT 1 models can be found on the model hub. We could also use multi-lingual BERT/ROBERTa models for the Spanish language.

Datasets
The WIT 5 dataset can be used for this task.

Available training scripts
A training script for this will be provided soon. (see PR 5)

(Optional) Desired project outcome
The desired outcome is to train a CLIP-like model for Spanish language. This can be showcased with a streamlit or gradio app.

(Optional) Challenges
This model will require some modifications to the existing models. Specifically, we will need to add projection layers in both the text and image encoder models.

Context: Count me in!


Question:
Image captioning for Indonesia with pre-trained vision and text model
For this project, a pre-trained image model like ViT can be used as an encoder, and a pre-trained text model like BERT and/or GPT2 can be used as a decoder.

Model
Pre-trained ViT 2, BERT models can be found on the model hub. We could use multi-lingual BERT/ROERTa models for the Indonesian language.

Datasets
The WIT  1 dataset can be used for this task. It has almost over 200K image-text pairs for Indonesia.
The GEM 3 dataset can also be used for the task

Available training scripts
As this will be a Seq2Seq model, the run_summarization_flax.py 4 script can be used for training this model with some modifications.

(Optional) Desired project outcome
The desired outcome is to see if pre-trained vision and text models can be leveraged for image captioning and also train captioning models in for the Indonesia language. This can be showcased with a streamlit or gradio app.

(Optional) Challenges
-This model will require some modifications to the existing text models.
-Data Processing

Context: Count me in!


Train GPT2/3 on social media posts and comments (reddit/Facebook etc)

Question: This project idea comes in my mind, we all are commenting to a post in any social media platforms. Just Imagine when GPT2/3 trained on all or any of the social media posts and its comments data, after it had been trained, it will reply/comment naturally in any of the posts by itself as…woow moment to see this happening

Context: I like this one  Would it be in English?


Question:
GPT2 for Punjabi
Pretrain GPT2 on Punjabi language to create a strong language generation model for Punjbai

Model
A randomly initialized GPT2 model

Datasets
One can make use of Kaggle Wikipedia Punjabi Articles dataset - Punjabi Wikipedia Articles | Kaggle 2

Available training scripts
A causal language modeling script for Flax is available here 1.

(Optional) Desired project outcome
The desired project output is a GPT2 model that is able to generate Punjabi language.

Context:
Hey @hgarg!
Great to see someone working on Punjabi. There is definitly a lack of models for Punjabi in the HuggingFace Hub, so this can definitly supplement that.
Great idea and all the best!


Question:
T5 for Bengali
Currently, there is no T5 model that was trained from scratch for Bengali on the hub. For this project, the goal is to create a strong language generation model for Bengali using T5 Model.

2. Language
Bengali.

3. Model
A randomly initialized T5 model.

4. Datasets
One can make use of OSCAR the dataset is also available through the datasets library here: oscar · Datasets at Hugging Face. The total Bengali resource in OSCAR is 11 GB.
Another source can be the mC4 dataset which is available in AllenAI 1. The resource size is 29GB.

5. Training scripts
A causal language modeling script for Flax is available here 4. It can be tweaked for training T5.

6. Challenges

Adapt the training script to T5
Fix a good tokenizer that covers Bengali vocabulary properly and make sure that the LM doesn’t become character-level LM.

7. Desired project outcome
The desired project output is a T5 model that is able to generate Bengali language.

8. Reads
The most important read would be the following colab:

Google Colaboratory  3

Apart from that we may need to look at the seqio library and source code of T5 here,

seqio 2
T5 github 2

Context: I am also a Bengali speaker. I am in!


Question:
PreTrain Wav2Vec2 in Indonesian
There is currently only a multilingually pre-trained model for Indonesian Wav2Vec2. Therefore we would like to pre-train Wav2Vec2 with only Indonesian datasets.

Model
A randomly initialized Wav2Vec2 model (if possible the large model)

Datasets
In addition to the Indonesian Common Voice (18h), we have also collected the following Indonesian speech datasets:

Wavenet Synthetic Voice (>400h)
TIML-IDN (14.5h)
Bible.is (40h)
Podcast (>10kh)

Available training scripts
FlaxWav2Vec2 will be merged soon: [Flax] Add wav2vec2 by patrickvonplaten · Pull Request #12271 · huggingface/transformers · GitHub 8 and a pretraining script should be relatively easy to be merged.

(Optional) Desired project outcome
The best Indonesian ASR model

Team
We have a team from the last wav2vec2 event:

@cahya
@Galuh
@Wikidepia
@ayameRushia
@munggok
@yasirabd

Context:
Wuhuuu - super excited about this! I’ll make sure a pretraining script is ready until Thursday.
Finalizing it!


RobIt : PreTrain RoBERTa-base from scratch in Italian

Question:
1. RobIt
The idea is to use the Italian portion of mC4 (which is about  590 GB) and OSCAR dataset(which is about 137 GB) to pre-train a RoBERTa-base model.

2. Language
The model will be trained in Italian

3. Model used
ROBERTa-base

4. Datasets
OSCAR-Italian, Italian Portion of multilingual C4 dataset

5.Training Scripts:
There are already Flax scripts to pre-train RoBERTa that we can easily use:

      github.com

    huggingface/transformers

  master/examples/flax/language-modeling

  🤗Transformers: State-of-the-art Natural Language Processing for Pytorch, TensorFlow, and JAX.

Google Colaboratory

6. Challenges:

It is too much data. We need a way to reduce the amount of data to finish on time so we will use random sampling.
Achieving SOTA.

7. Desired Possible Outcome:

A Italian monolingual well performing model. We could optionally test the validity of our results at a later stage by fine-tuning the model on some downstream tasks.

8. Reads

https://arxiv.org/pdf/1907.11692.pdf
We found a model trained on RoBERTa  1 but we plan on increasing the dataset with a different tokenisation.

Context: Count me in !!!


Pretrain RoBERTa-large from scratch in Finnish

Question:
Finnish RoBERTA-large
The project idea is somewhat identical to the one for Pretraining Roberta in Spanish but instead using the Finnish datasets
The idea is to use the Finnish portion of mC4  (which roughly amounts for 100GB of uncompressed text) to pre-train a RoBERTa-large model, first on 256 sequence length and then on 512.  Also we might try few other datasets (OSCAR, STT, Yle news)

2. Language
The model will be trained in Finnish.

3. Model
RoBERTa-large
(Maybe also some other models if time is left)

4. Datasets
Finnish portion of mC4 of about 100gb
Yle news dataset
STT news

5. Training scripts
There are already Flax scripts to pre-train RoBERTa that we can easily use:
https://github.com/huggingface/transformers/tree/master/examples/flax/language-modeling)

6. Challenges
Will the data be enough to be able to train a good model?

7. Desired project outcome
A Finnish monolingual well performing model on the usual benchmarks. We hope to beat SOTA model for this task. Current SOTA is TurkuNLP/finbert

8. Reads
- https://arxiv.org/pdf/1907.11692.pdf
- https://arxiv.org/pdf/1911.00359.pdf
- https://www.aclweb.org/anthology/W11-2123.pdf

Context: Count me in! Would be exciting to get new pretrained Finnish models in addition to the current one and only Finnish BERT.


PReTrain RoBERTa from scratch in Norwegian

Question:
RoBERTa/BERT for Norwegian
Currently, there is only a very limited amount of BERT-like models for Thai on the hub: Hugging Face – The AI community building the future. . For this project, the goal is to create a RoBERTa/BERT model for just the Norwegian language.

Model
A randomly initialized RoBERTa/BERT model

Datasets
One can make use OSCAR 3 the dataset is also available through the datasets library here: oscar · Datasets at Hugging Face 3.

Available training scripts
A masked language modeling script for Flax is available here 9. It can be used pretty much without any required code changes.

(Optional) Desired project outcome
The desired project output is a strong RoBERTa/BERT model in Norwegian.

(Optional) Challenges
The OSCAR dataset might be too small (it has < 5GB of data for Thai). Also it might be important
to find datasets the BERT-like model can be evaluated on after pretraining in Norwegian. Having found a dataset to fine-tune the pretrained BERT-like model on, one can make use of the text-classification script here 9

(Optional) Links to read upon
The most important read would be the following colab:

Google Colaboratory 14

Context:
I am also really interested in this one, or maybe even ALBERT. But the available dataset on OSCAR might not be enough. Using mC4 might add almost 100GB of clean-ish text. I have worked on Norwegian BERT-based models before 3, and we (the National Library of Norway) have a big corpus that unfortunately we are not allowed to share. We could maybe share parts of it to complement mC4/OSCAR, but not all it. Not sure if there is a way to train on a private corpus in this event, I guess not, but worth asking.
There are also datasets for extrinsic evaluation on Norwegian for POS, NER, and sentiment (the sentiment one I feel was kinda designed for word2vec-like models) from the UiO LTG group, another on political speech party affiliation we built from the Talks of Norway, and a recent one on hate speech we could use. Not sure if evaluating on WikiANN could be an option.


PreTrain RoBERTa from scratch in Indonesian

Question:
RoBERTa in Indonesian
There are currently only a few 1 Indonesian RoBERTa models trained from scratch. We aim to create a strong Indonesian RoBERTa language model, to be used for further fine-tuning.

Language
The model will be trained in Indonesian (Bahasa Indonesia).

Model
A randomly initialized RoBERTa model.

Datasets
There is an Indonesian OSCAR 1 dataset, which is also available in Hugging Face’s Datasets.
Alternatively, there are publicly available datasets such as:

Indo4B by IndoBenchmark

Training scripts
We can make use of the Flax MLM script from Hugging Face 1 to train the model.

(Optional) Challenges
The deduplicated Indonesian OSCAR dataset is of the size 16GB, which may not be the best option available. The Indo4B dataset, on the other hand, sums up a total of 23GB.

(Optional) Desired project outcome
The desired project output is a strong RoBERTa model in Indonesian. To benchmark the results, we can try to fine-tune the output model to downstream Indonesian LM tasks provided by IndoBenchmark.

(Optional) Reads
The following links can be useful to better understand the project and what has previously been done.

Google Colaboratory 1

Context: Great Idea, would be great to expand the Indonesian NLP community. Count me in.


Question:
Fine-Tuning a Seq2Seq model for sentence fusion in English.
Sentence fusion is the task of joining several independent sentences into a single coherent text. E.g.: the sentences:
Hillary goes to school. Hillary meets her friends at school.

could be fused to
Hillary goes to school and meets her friends.

Currently there is only one model on the hub for sentence fusion as can be seen on the dataset site: discofuse · Datasets at Hugging Face 9 .
The goal of this project is to have the best sentence fusion model for English on the hub.

Model
One use one or multiple of the pretrained T5 models:

google/byt5-base · Hugging Face
google/t5-v1_1-base · Hugging Face 1
t5-base · Hugging Face 2

Datasets
The Discofuse dataset can be used:   discofuse · Datasets at Hugging Face 9

Available training scripts
As this will be a Seq2Seq model, the run_summarization_flax.py 14 script can be used for training.

(Optional) Desired project outcome
The desired outcome is to have a sentence fusion model for the English language. This can be showcased directly on the hub or with a streamlit  or gradio app.

(Optional) Challenges
Beating the existing model will be the most challenging part.

(Optional) Links to read upon

[1907.12461] Leveraging Pre-trained Checkpoints for Sequence Generation Tasks 1
google/roberta2roberta_L-24_discofuse · Hugging Face 2
discofuse · Datasets at Hugging Face 9

Context: Interested in this project.


Question: Topics related to either pytorch/vision or vision research related topics

Context:
hallo, I am new to pytorch, but I am enjoying this framework. I am having diffeculty in figuring out this message.can anyone help please !!
RuntimeError: The size of tensor a (10) must match the size of tensor b (32) at non-singleton dimension 0
I did a bit of debugging and I found that there is a collapse of labels happening in labels tensors
[[[-1.0000, -1.0000, -1.0000,  …, -1.0000, -1.0000, -1.0000],
[-1.0000, -1.0000, -1.0000,  …, -1.0000, -1.0000, -1.0000],
[-1.0000, -1.0000, -1.0000,  …, -1.0000, -1.0000, -1.0000],
…,
[-1.0000, -1.0000, -1.0000,  …, -1.0000, -1.0000, -1.0000],
[-1.0000, -1.0000, -1.0000,  …, -1.0000, -1.0000, -1.0000],
[-1.0000, -1.0000, -1.0000,  …, -1.0000, -1.0000, -1.0000]]]]) tensor([9, 2, 1, 1, 6, 1, 4, 6, 5, 7, 4, 5, 7, 3, 4, 1, 2, 4, 8, 0, 2, 5, 7, 9,
1, 4, 6, 0, 9, 3, 8, 8])
I created my model from scratch . I tried with batch_size 32 , it threw me this error. I tried with batch_size 1 , I got 320% accuracy (something wrong !). The error went away. so there is some dimensional mismatch  some where, I am lost! thanks a million !!! for your help and advise


Question:
I have a problem where I want to create a network that classify a set of images, however the number of images are not fixed, How do I come up with a solution such that the network deals with this dynamic behavior of set of inputs.
Thank you.

Context:
You can train for a large set and pad shorter ones (that’s the typical approach when working with sequences)
Another idea is to compute N features for N images and then doing an average (which is somehow a good option)


Predicted labels stuck at 1 for test set where class 0 is 20% of data

Question:
So, my class 0 is 20% and class 1 is 80% of the entire data and I have stratified 5 fold train/val/test with division of 60/20/20.
However, my predicted labels for test set are stuck at 1.
Any thoughts how to fix this? I used the weighted loss suggested in

    Unclear about Weighted BCE Loss

    If you would like to use the weight as a class weight, I assume you have a binary target, i.e. it contains only zeros and ones.
In that case I would create a weight tensor and just multiply it with your unreduces loss. Here is a small example:
weight = torch.tensor([0.1, 0.9])
weight_ = weight[y.data.view(-1).long()].view_as(y)
criterion = nn.BCELoss(reduce=False)
loss = criterion(output, y)
loss_class_weighted = loss * weight_
loss_class_weighted = loss_class_weighted.mean()

Let me know, if …

except I am not sure if I used it correctly.
X here is an intermediate representation of NxD where N is number of patches in my images and D is resnet18 last layer feature, 512 1D tensor. My batch size is 16.
I am also unsure if pred_labels = out.argmax(dim=1) is correct or not? Also, couldn’t directly use BCELoss since my out looks like something like:
*one instance from val phase:
x is:  tensor([[ 0.0154,  0.0957],
        [-0.0446,  0.0787],
        [-0.0272,  0.0265],
        [ 0.1656, -0.1316],
        [ 0.0931,  0.0134],
        [ 0.2454,  0.2079],
        [ 0.5408, -0.0578],
        [ 0.2008,  0.1331],
        [ 0.1812,  0.0924],
        [ 0.1018,  0.0468],
        [ 0.1389,  0.2432],
        [-0.3082, -0.2986],
        [ 0.1031,  0.0219],
        [ 0.0910, -0.1239],
        [ 0.4165,  0.0890],
        [ 0.2290, -0.0611]], device='cuda:0')
loss:  tensor([0.6538, 0.6334, 0.7203, 0.5555, 0.7338, 0.7120, 1.0366, 0.7276, 0.7385,
        0.7210, 0.7466, 0.6884, 0.7346, 0.8064, 0.8703, 0.5585],
       device='cuda:0')
weight_:  tensor([0.9000, 0.9000, 0.1000, 0.1000, 0.9000, 0.9000, 0.9000, 0.9000, 0.9000,
        0.9000, 0.1000, 0.9000, 0.9000, 0.9000, 0.9000, 0.1000])
val pred:  tensor([1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0], device='cuda:0')
val label:  tensor([1, 1, 0, 0, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 0])

self.criterion = nn.CrossEntropyLoss(reduce=False)
#self.criterion = nn.BCELoss(reduce=False)
def forward(self, X, labels):

    stacked_X = torch.stack(X)
    out = self.transformer(stacked_X)

    with torch.autocast('cuda'):
        # https://discuss.pytorch.org/t/unclear-about-weighted-bce-loss/21486/2
        labels = torch.tensor(labels)
        weight = torch.tensor([0.1, 0.9]) # how to exactly decide on this weights?
        weight_ = weight[labels.data.view(-1).long()].view_as(labels)
        loss = self.criterion(out, torch.tensor(labels).cuda())
        print('loss: ', loss)
        print('weight_: ', weight_)
        loss_class_weighted = loss * weight_.cuda()
        loss_class_weighted = loss_class_weighted.mean()

    #pred = out.data.max(1)[1]

    pred_labels = out.argmax(dim=1)

    return pred_labels, labels, loss_class_weighted

If I use BCELoss with reduce=False instead, I get this error:
  loss = self.criterion(out, torch.tensor(labels).cuda())
Traceback (most recent call last):
  File "main_classifier.py", line 250, in <module>
    pred,label,loss = trainer.train(sample_batched, model)
    pred, labels, loss = model.forward(feats, labels)
    return self.module(*inputs[0], **kwargs[0])
  File "/home/jalal/research/venv/dpcc/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
    loss = self.criterion(out, torch.tensor(labels).cuda())
  File "/home/jalal/research/venv/dpcc/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/jalal/research/venv/dpcc/lib/python3.8/site-packages/torch/nn/modules/loss.py", line 603, in forward
    return F.binary_cross_entropy(input, target, weight=self.weight, reduction=self.reduction)
  File "/home/jalal/research/venv/dpcc/lib/python3.8/site-packages/torch/nn/functional.py", line 2906, in binary_cross_entropy
    raise ValueError(
ValueError: Using a target size (torch.Size([16])) that is different to the input size (torch.Size([16, 2])) is deprecated. Please ensure they have the same size.

Context:
unlike corssentropy which gets model score  (-∞,+∞), BCELoss accept output of sigmoid layer as input.
if you want you can use
m = nn.Sigmoid()
loss = nn.BCELoss()
input = torch.randn(3, requires_grad=True)
target = torch.empty(3).random_(2)
output = loss(m(input[:,1]-input[:,0]), target)

with the same model or you can chage your model output shape to match the BCELoss.


How to resize and pad in a torchvision.transforms.Compose()?

Question:
I’m creating a torchvision.datasets.ImageFolder() data loader, adding torchvision.transforms steps for preprocessing each image inside my training/validation datasets.
My main issue is that each image from training/validation has a different size (i.e.: 224x400, 150x300, 300x150, 224x224 etc). Since the classification model I’m training is very sensitive to the shape of the object in the image, I can’t make a simple torchvision.transforms.Resize(), I need to use padding to maintain the proportion of the objects.
Is there a simple way to add a padding step into a torchvision.transforms.Compose() pipeline (ensuring that every image is 224x224, without cropping the image, only doing a resize and padding)? *each image has a different original shape/size
data_transforms = {
    'train': transforms.Compose([
        transforms.Resize((224,224)),
        transforms.Grayscale(num_output_channels=3),
        transforms.RandomHorizontalFlip(),
        transforms.ToTensor(),
        transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
    ]),
    'val': transforms.Compose([
        transforms.Resize((224,224)),
        transforms.Grayscale(num_output_channels=3),
        transforms.ToTensor(),
        transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
    ]),
}

data_dir = "./my_data_dir/"
image_datasets = {x: datasets.ImageFolder(os.path.join(data_dir, x),
                                          data_transforms[x])
                  for x in ['train', 'val']}

dataloaders = {x: torch.utils.data.DataLoader(image_datasets[x], batch_size=16,
                                             shuffle=True, num_workers=8)
              for x in ['train', 'val']}

Reading the torchvision.transforms.Pad() documentation, I’ve understood that I need to know the size of the padding beforehand, and know if it will be applied on left/right or top/bottom, before using this transform.
Is there a simple way to add a step on this transforms.Compose() to infer the image size, so I can get the parameters that I need to configure my torchvision.transforms.Pad()?

Context:
I think I did something similar where I kept all ratios by making the each making all the images the width, and height of the biggest image. Then set the image in the center and pad the empty spaces. I don’t know if there’s an function that does this automatically but I did it myself. but I made functions like this in my data class, which works for me. After this you should be able resize and keep the ratios with the padding.
def get_padding(image):
    max_w = 1203
    max_h = 1479

    imsize = image.size
    h_padding = (max_w - imsize[0]) / 2
    v_padding = (max_h - imsize[1]) / 2
    l_pad = h_padding if h_padding % 1 == 0 else h_padding+0.5
    t_pad = v_padding if v_padding % 1 == 0 else v_padding+0.5
    r_pad = h_padding if h_padding % 1 == 0 else h_padding-0.5
    b_pad = v_padding if v_padding % 1 == 0 else v_padding-0.5

    padding = (int(l_pad), int(t_pad), int(r_pad), int(b_pad))

    return padding

def pad_image(self, image):
        padded_im = pad(image, get_padding(image)) # torchvision.transforms.functional.pad
        return padded_im

Note: I just wrote this only to make it work, so it’s not optimal or ‘ideal way’ but it works for me.


Question:
Given a batch of error maps, I would like to mask out error values based on the max of each error map in the batch.
Using torch.max currently gives the single maximum value in the whole batch of error maps. Masking using this single value would result in inconsistent masks with respect to each error map.
Example Input: Loss Map shape: (8, 256, 640) (Batch, Height, Width)
Return: (8, 256, 640) with masked out pixels.
Current Implementation:
bound = torch.max(loss[i]) * 0.9          # Find threshold
mask = torch.zeros_like(loss[i])          # create mask
mask[loss[i] < bound] = 1.0                # set the values i want to keep to 1
mask[loss[i] >= bound] = 0.0             # set others to 0
loss[i] *= mask                                  # Mask!

I am looking for an efficient way to do this in PyTorch instead of a naive approach of using a for loop for each image in a batch.
Thanks!

Context:
One solution, inspired from How to efficiently normalize a batch of tensor to [0, 1] 2, is as follows.
import torch
batch_size, height, width = 2, 2, 2

torch.manual_seed(0)
loss = torch.randn((batch_size, height, width))
print(loss)

# loss.size(0) or batch_size
loss = loss.view(loss.size(0), -1) # tensor of shape (batch_size, height * width)
bound = loss.max(dim = 1, keepdim=True)[0] * 0.9
mask = torch.zeros_like(loss)
mask[loss < bound] = 1.0   # set the values i want to keep to 1
mask[loss >= bound] = 0.0
loss *= mask
loss = loss.view(batch_size, height, width) # tensor of shape (batch_size, height, width)

print(loss)

I’ve set the random seed just to ensure reproducibility, the following code (yours) gives the same result
torch.manual_seed(0)
loss = torch.randn((batch_size, height, width))
print(loss)

for i in range(batch_size) :
    bound = torch.max(loss[i]) * 0.9
    mask = torch.zeros_like(loss[i])
    mask[loss[i] < bound] = 1.0  # set the values i want to keep to 1
    mask[loss[i] >= bound] = 0.0
    loss[i] *= mask

print(loss)


Advantage of creating multiple layers with same arguments

Question:
Case 1:
class _ResGraphConv(nn.Module):
    def __init__(self, adj, input_dim, output_dim, hid_dim, p_dropout):
        super(_ResGraphConv, self).__init__()

        self.gconv1 = _GraphConv(adj, input_dim, output_dim, p_dropout)
        self.gconv2 = _GraphConv(adj, input_dim, output_dim, p_dropout)

    def forward(self, x):
        residual = x
        out = self.gconv1(x)
        out = self.gconv2(out)
        return residual + out

Case 2:
class _ResGraphConv(nn.Module):
    def __init__(self, adj, input_dim, output_dim, hid_dim, p_dropout):
        super(_ResGraphConv, self).__init__()

        self.gconv1 = _GraphConv(adj, input_dim, output_dim, p_dropout)

    def forward(self, x):
        residual = x
        out = self.gconv1(x)
        out = self.gconv1(out)
        return residual + out

As you can see in case 1, I have created 2 layers and calling them from forward. And in case 2, I’m creating only 1 layer as they take same arguments but calling that from forward twice to decrease the number of parameters. Is there any disadvantage of doing case 2?

Context:
@tanvir_14
In the second case, since your are reusing the same layer, your learnable parameters have been reduced.
Imagine this way, you are using the same filters on the image twice.


Error while applying torch.nn.DataParallel to custom network

Question:
import torch
 import torch.nn as nn
 import torch.nn.functional as F
 from torchvision import datasets, transforms
 from torch.utils.data.dataloader import DataLoader

 import copy

 devices    = [0,1]
 batch_size = 64
 sim_type   = 'snn'
 timesteps  = 100

 device = 'cuda:' + str(devices[0])

 def test(model, epoch, test_loader):
     print('Testing model..')

     total = 0
     with torch.no_grad():
         model.eval()

         for i in range(100):
             # looping over dataset (start)
             for batch_idx, (data, target) in enumerate(test_loader):

                 if sim_type == 'snn':
                     data = data.repeat(timesteps, 1, 1, 1, 1)
                     data = data.swapaxes(0, 1)
                     B,T,C,H,W = data.size()
                 else:
                     B,C,H,W   = data.size()

                 data, target = data.to(device), target.to(device)
                 if batch_idx == 0:
                     print('data: {}'.format(data.size()))

                 output  = model(data)
                 loss    = F.cross_entropy(output, target)
                 pred    = output.max(1,keepdim=True)[1]
                 correct = pred.eq(target.data.view_as(pred)).cpu().sum()
                 total   += B
                 top1    = correct / total

                 print('Batch: {}, Accuracy: {}/{}({:.4f}), Loss: {:.4f}'
                     .format( batch_idx, correct.item(), B, top1, loss,))
             # looping over dataset (end)

 def load_cifar100():
     normalize = transforms.Normalize(mean = [0.5, 0.5, 0.5], std = [0.5, 0.5, 0.5])
     transform = transforms.Compose([transforms.ToTensor(), normalize])
     testset   = datasets.CIFAR100(root='./Datasets/cifar_data', \
             train=False, download=False, transform=transform)
     return testset

 class ANN(nn.Module):
     def __init__(self):
         super().__init__()

         self.ann_dict = nn.ModuleDict({
             'conv1': nn.Conv2d(3, 64, 3, padding=1, stride=1, bias=False),
             'relu1': nn.ReLU(inplace=True),
             'avg1' : nn.AvgPool2d(2, 2),

             'conv2': nn.Conv2d(64, 128, 3, padding=1, stride=1, bias=False),
             'relu2': nn.ReLU(inplace=True),
             'avg2' : nn.AvgPool2d(2, 2),

             'flat3': nn.Flatten(),
             'lin4' : nn.Linear(8192, 100, bias=False),
             })

     def forward(self, x):
         out_prev = x
         for k, v in self.ann_dict.items():
             out_prev = self.ann_dict[k](out_prev)
         return out_prev

 class SNN(nn.Module):
     def __init__(self, ann, device, in_sz):
         super(SNN, self).__init__()

         num_relus = 0
         for k, v in ann.named_modules():
             if isinstance(v, nn.ReLU):
                 num_relus += 1

         self.device   = device
         thresholds    = torch.ones(num_relus, device=device)
         self.snn_cell = SNNCell(ann, thresholds, device=device, in_sz=in_sz)

     def reset_vmems(self):
         self.snn_cell.reset_vmems()

     def get_vmems(self):
         return self.snn_cell.get_vmems()

     def forward(self, x):
         self.reset_vmems()
         vmems = self.get_vmems()
         return self.snn_cell(x, *vmems)

 class SNNCell(nn.Module):
     def __init__(self, ann, thresholds, device, in_sz):
         super(SNNCell, self).__init__()

         self.device   = device
         self.in_sz    = in_sz
         self.snn_dict = self.convert_to_snn(ann, thresholds)

     def convert_to_snn(self, ann, thr):
         snn_dict   = copy.deepcopy(ann.ann_dict)
         relu_idx   = 0
         in_sz      = self.in_sz
         for k, v in snn_dict.items():
             out_sz = v.cpu()(torch.zeros(in_sz)).size()
             if isinstance(v, nn.ReLU):
                 snn_dict[k] = SpikeRelu(thr[relu_idx], sz=in_sz, device=self.device,)
                 relu_idx += 1
             in_sz = out_sz
         return snn_dict

     def reset_vmems(self):
         for k, v in self.snn_dict.items():
             if isinstance(v, SpikeRelu):
                 self.snn_dict[k].reset_vmem()
     def get_vmems(self):
         vmems = []
         for k, v in self.snn_dict.items():
             if isinstance(v, SpikeRelu):
                 vmems.append(v.vmem.clone())
         return vmems

     def forward(self, x, *mem):
         T = x.size(1)

         L       = len(mem)
         mem_in  = list(mem[:])
         mem_out = [None] * L

         for t in range(T):
             spk_layer_idx  = 0
             outprev        = x[:,t]
             for name, layer in self.snn_dict.items():
                 if isinstance(layer, SpikeRelu):
                     outprev, mem_out[spk_layer_idx] = layer(outprev, mem_in[spk_layer_idx])
                     spk_layer_idx += 1
                 else:
                     outprev = layer(outprev)
             mem_in = mem_out[:]
         return outprev

 class SpikeRelu(nn.Module):
     def __init__(self, vth, sz, device='cuda:0'):
          super(SpikeRelu, self).__init__()

          self.threshold = vth
          self.sz        = sz
          self.device    = device
          self.reset_vmem()

     def reset_vmem(self):
         self.vmem = torch.zeros(*self.sz, device=self.device, requires_grad=True)

     def forward(self, x, mem_in):
         mem_thr = (mem_in / self.threshold) - 1.0
         op_spk  = torch.zeros_like(mem_thr)
         op_spk[mem_thr > 0] = 1.0
         rst     = self.threshold * (mem_thr > 0).float()
         mem_out = mem_in + x - rst
         return op_spk, mem_out

     def __repr__(self):
         return 'SpikeRelu(v_th : {:.3f}, vmem : {})'.format(self.threshold, torch.sum(self.vmem))

Hello everyone,
I am trying to apply torch.nn.DataParallel to a custom module (class SNN) whose simplified definition is provided above. I perform inference on this network using the following code:
 test_set    = load_cifar100()
 test_loader = DataLoader(test_set, batch_size=batch_size, shuffle=False)
 model = ANN()

 print('devices ', devices, 'device ', device)
 in_sz = (1,3,32,32)
 if sim_type == 'snn':
     model = SNN(model, device, in_sz, )

 model = torch.nn.DataParallel(model, device_ids=devices)
 model = model.to(device)

 print(model)
 test(model, 0, test_loader)

This keeps giving the following error: RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cuda:1!
I believe the error is due to the way I am initializing my  vmem  in class SpikeRelu but I am not sure how to fix it. Please help!!

Context:
Don’t pass the device argument to SkipeRelu, but use the .device attribute from e.g. the incoming activation tensor, if needed.
E.g. you could use:
     def reset_vmem(self, device):
         self.vmem = torch.zeros(*self.sz, device=device, requires_grad=True)

and call it via self.reset_vmem(x.device) where x would be the input tensor.


Question:
My code performs a loss minimization from the output of a pretrained GAN. Apart from the GAN parameters I am also trying to optimize the latent variable z.
The code is set up as follows
with torch.no_grad():
   gan_out = G(z_initial)

z_initial.retain_grad = True
loss = MSEloss(gan_out, observed_image)

GAN_optim.zero_grad()
z_optim.zero_grad()
loss.backward()
GAN_optim.step()
z_optim.step()

The optimizers are defined as
GAN_optim = optim.RMSprop(G.parameters(), lr = 3e-5)
z_optim = optim.RMSprop([z_initial], lr = 1e-3)

# where z_initial is

z_initial = torch.Variable(z_.clone().to(device), requires_grad = True)

After training, I am visualizing z_initial and the GAN weights. While the GAN weights are updating and changing z_initial is not.
The tensorboard visualization is done for every epoch as follows
writer.add_histogram('Z_prior', z_initial)

How can I make sure z_initial updates?

Context:
How about using nn.Parameter instead of Variable?
z_initial = nn.Parameter(z_).to(device)


Question:
I was just training a model and this warning popped, I’ve never seen it before. I don’t know if it’s code related.
my_project\venv\lib\site-packages\torchvision\io\image.py:11: UserWarning: Failed to load image Python extension: Could not find module 'my_project\venv\Lib\site-packages\torchvision\image.pyd' (or one of its dependencies). Try using the full path with constructor syntax.
  warn(f"Failed to load image Python extension: {e}")

Is it something I should worry about? How can I fix it?

Context: How did you install torchvision? Did you build it from source or did you install the binaries? In any case, did you downgrade it recently?


Question:
I am working on Cardiac Ct images by implementing the 2DCNN model. I am trying to train vgg model. But every time I got this issue. I know this is related to layers as I am new in PyTorch I am not able to understand. If someone can guide me related to it

Here is the error and code. My torch.Size([2, 1, 65, 65]) I am using the dense-net model and it has the same torch.Size([2, 1, 65, 65]) and it’s working fine .

VGG_MODEL

Traceback (most recent call last):
  File "ct_pretrained.py", line 201, in <module>
    loss, metric = train(model, train_loader, optimizer)
  File "ct_pretrained.py", line 59, in train
    output = model(axial, sagittal, coronal, emr)
  File "/root/miniconda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/data/heart_ct/torch/models/vgg.py", line 44, in forward
    sagittal_feature = self.sa_co_model(sagittal)
  File "/root/miniconda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/root/miniconda/lib/python3.8/site-packages/torchvision/models/vgg.py", line 43, in forward
    x = self.features(x)
  File "/root/miniconda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/root/miniconda/lib/python3.8/site-packages/torch/nn/modules/container.py", line 117, in forward
    input = module(input)
  File "/root/miniconda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/root/miniconda/lib/python3.8/site-packages/torch/nn/modules/pooling.py", line 153, in forward
    return F.max_pool2d(input, self.kernel_size, self.stride,
  File "/root/miniconda/lib/python3.8/site-packages/torch/_jit_internal.py", line 267, in fn
    return if_false(*args, **kwargs)
  File "/root/miniconda/lib/python3.8/site-packages/torch/nn/functional.py", line 585, in _max_pool2d
    return torch.max_pool2d(
RuntimeError: Given input size: (512x1x3). Calculated output size: (512x0x1). Output size is too small

import torch

import torch.nn as nn

from torchvision import models

__all__ = ['Vgg']

class Vgg(nn.Module):

    def __init__(self, is_emr=False, mode='sum'):

        super().__init__()

        self.is_emr = is_emr

        self.mode = mode

        in_dim = 45

        self.axial_model = models.vgg13(pretrained=True)

        out_channels = self.axial_model.features[0].out_channels

        self.axial_model.features[0] = nn.Conv2d(1, out_channels, kernel_size=7, stride=1, padding=0, bias=False)

        self.axial_model.features[3] = nn.MaxPool2d(1)

        num_ftrs = self.axial_model.classifier[6].in_features

        self.axial_model.classifier[6] = nn.Linear(num_ftrs, 15)

        self.sa_co_model = models.vgg13(pretrained=True)

        self.sa_co_model.features[0] = nn.Conv2d(1, out_channels, kernel_size=7, stride=1, padding=(3,0), bias=False)

        self.sa_co_model.features[3] = nn.MaxPool2d(1)

        num_ftrs = self.sa_co_model.classifier[6].in_features

        self.sa_co_model.classifier[6] = nn.Linear(num_ftrs, 15)

        if self.is_emr:

            self.emr_model = EMRModel()

            if self.mode == 'concat': in_dim = 90

        self.classifier = Classifier(in_dim)

        #print(self.classifier)

    def forward(self, axial, sagittal, coronal, emr):

        #print(axial.shape)

        axial = axial[:,:,:-3,:-3]

        sagittal = sagittal[:,:,:,:-3]

        coronal = coronal[:,:,:,:-3]

        axial_feature = self.axial_model(axial)

        sagittal_feature = self.sa_co_model(sagittal)

        coronal_feature = self.sa_co_model(coronal)

        out = torch.cat([axial_feature, sagittal_feature, coronal_feature], dim=1)

        out = self.classifier(out)

        if self.is_emr:

            emr_feature = self.emr_model(emr)

            out += emr_feature

        return axial_feature

class EMRModel(nn.Module):

    def __init__(self):

        super().__init__()

        self.layer = nn.Sequential(

            nn.Linear(7, 256),

            nn.BatchNorm1d(256),

            nn.LeakyReLU(negative_slope=0.2),

            nn.Dropout(p=0.2, inplace=True),

            nn.Linear(256, 256),

            nn.BatchNorm1d(256),

            nn.LeakyReLU(negative_slope=0.2),

            nn.Dropout(p=0.2, inplace=True),

            nn.Linear(256, 5),

        )

    def forward(self, x):

        return self.layer(x)

class Classifier(nn.Module):

    def __init__(self, in_dim):

        super().__init__()

        self.layer = nn.Sequential(

            nn.Linear(in_dim, 5)

        )

    def forward(self, x):

        return self.layer(x)

class ConvBN(nn.Module):

    def __init__(self, in_dim, out_dim, **kwargs):

        super().__init__()

        self.layer = nn.Sequential(

            nn.Conv2d(in_dim, out_dim, bias=False, **kwargs),

            nn.BatchNorm2d(out_dim),

            nn.LeakyReLU(negative_slope=0.2))

    def forward(self, x):

        return self.layer(x)

'''

if __name__ == "__main__":

     images = torch.randn(2,1,65,65)

     model = Vgg()

     out = model(images,images,images)

     model = models.vgg16(pretrained=True)

     for k, v in model.state_dict().items():

         print(k)

  '''

Context: The spatial size of the input is too small for the model architecture, as one of the layers would create an empty tensor as given in the error message. Increase the spatial size of the input and it should work.


Torchvision’s ColorJitter support of uint8 images?

Question:
Does ColorJitter transform support torch.uint8 tensor images?
I found a similar question but for torch.int16 tensor images: ColorJitter Transformation for 16-bit images? 2

Context:
Yes, this seems to work:
transform = transforms.ColorJitter(brightness=0.5, contrast=0.5, saturation=0.5, hue=0.5)
x = torch.randint(0, 256, (3, 224, 224), dtype=torch.uint8)
out = transform(x)


How to access to a layer by module name?

Question:
I have a ResNet34 model and I want to find all the ReLU layer. I used named_modules() method to get the layers.
for name, layer in model.named_modules():
    if isinstance(layer, nn.ReLU):
        print(name, layer)

And got out put like
0.2 ReLU(inplace=True)
0.4.0.relu ReLU(inplace=True)
0.4.1.relu ReLU(inplace=True)
0.4.2.relu ReLU(inplace=True)

Is there any way that I can use the name (for example 0.4.0.relu) directly to access to a relu layer? Or I have to index the number in [] like model[0][4][0].relu?

Context: image1746×910 155 KB


Inconsistent behaviour for DataLoader torch ‘1.9.0’

Question:
Hi everyone, I have an issue with the behaviour of the torch.utils.data.DataLoader for a minimal custom torch.utils.data.Dataset.
Here is the dataset
class MyData(Dataset):
  'Characterizes a dataset for PyTorch'
  def __init__(self, data, targets):
        """"Initialization, here display serves to show an example
        if false, it means that we intend to feed the data to a model"""
        self.targets = targets
        self.data = data

  def __len__(self):
        'Denotes the total number of samples'
        return len(self.targets)

  def __getitem__(self, index):
        'Generates one sample of data'
        # Select sample
        X = self.data[index]
        # Load data and get label
        y = self.targets[index]
        return X,y

But now, if I try to use it within a DataLoader, it doesn’t switch the number of channel to the second dimension of the tensor
Here is a minimal exemple:
train_data = datasets.CIFAR10('../data/cifar', train=True, transform=transforms.ToTensor(), download=False)
test_data = datasets.CIFAR10('../data/cifar', train=False, transform=transforms.ToTensor(), download=False)

source_data = MyData(train_data.data, train_data.targets)
source_loader = torch.utils.data.DataLoader(source_data, batch_size=1024)
c,d = next(iter(source_loader))
c.shape

It it will return shape
torch.Size([1024, 32, 32, 3])

Whereas, if I do it directly the cifar10 dataset. For instance
train_loader = DataLoader(train_data, batch_size=1024, shuffle=True)
a,b = next(iter(train_loader))
a.shape

will return
torch.Size([1024, 3, 32, 32])

I am using

torch ‘1.9.0’
torchvision ‘0.10.0’

Context:
Jean_Michel_Sarr:

source_data = MyData(train_data.data, train_data.targets)

Probably you are not applying transform sequence as torchvision's CIFAR10 does.
Remember that only the raw data is passed as input to your newly written MyData class. All the other post-computations (such as Normalization) should be handled in MyData.

      github.com

    pytorch/vision/blob/main/torchvision/datasets/cifar.py#L113-L118

            # doing this so that it is consistent with all other datasets
            # to return a PIL Image
            img = Image.fromarray(img)

if self.transform is not None:
                img = self.transform(img)


Question:
I would like to fine tune a pre-trained GAN available online using my own images. For example, BigGAN, which was trained on ImageNet, can generate realistic images. However, I want to generate medical images. How can I fine tune the pre-train models?
https://github.com/ajbrock/BigGAN-PyTorch 1

Context: The inked repository has a fine tuning section 3 which explains how the code can be used to fine tune a model using a custom dataset. Generally, I would recommend to also take a look at the paper and check how the model was trained at all. Since your images are coming from another domain (medical images) you would have to experiment if the mentioned fine tuning recipe is working or if you would have to train a model from scratch.


Accuracy not increasing loss not decreasing

Question:
Hello,
I am training a pytorch model for sign language classification. There are 29 classes. I am using torchvision augmentation. For weeks I have been trying to train the model. But the loss keeps hovering around the number where it starts, and the accuracy to remains where it started(accuracy is as good as choosing a random label). I first tried smaller models, then bigger models, now pytorch’s inbuilt models, all of which give me the same result. I read online and tried weight decay, different hyperparameters, none of which seem to help.
Here is the code:
!unzip /content/drive/MyDrive/asl.zip -d /content/data

from torch.utils.data import Dataset, DataLoader
import torch
import torchvision.transforms as T
import cv2 as cv
import glob
import numpy as np
import random
import os
from PIL import Image
import torchvision.models as models

os.rename('/content/data/asl_alphabet_train/asl_alphabet_train', '/content/data/train')
os.rename('/content/data/asl_alphabet_test/asl_alphabet_test', '/content/data/test')
os.removedirs('/content/data/asl_alphabet_train')
os.removedirs('/content/data/asl_alphabet_test')

train_transf = T.Compose(
    [
     T.GaussianBlur(9),
     T.RandomRotation((0, 5)),
     T.RandomPerspective(),
     T.RandomHorizontalFlip(),
     T.RandomVerticalFlip(),
     T.ToTensor(),
     T.Normalize(mean=[0.5, 0.5, 0.5], std=[0.5, 0.5, 0.5]),
    ]
)
test_transf = T.Compose(
    [
     T.GaussianBlur(9),
     T.RandomRotation((0, 180)),
     T.RandomHorizontalFlip(),
     T.RandomVerticalFlip(),
     T.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]),
     T.ToTensor()
    ]
)

train_data_path ='/content/data/train'
test_data_path ='/content/data/test'

train_img_paths = []
#classes = []
for path in glob.glob(train_data_path + '/*'):
  #classes.append(path.split('/')[-1])
  for img_path in glob.glob(path + '/*'):
    #new_fn = img_path[:20] + img_path.split('/')[-2] + '/' + ''.join([i for i in img_path.split('/')[-1] if i.isdigit() or i == '.' or i == '/']) + 'jpg'
    train_img_paths.append(img_path)

idx_to_class = ['A', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 'I', 'J', 'K', 'L', 'M',
                'N', 'O', 'P', 'Q', 'R', 'S', 'T', 'U', 'V', 'W', 'X', 'Y', 'Z',
                'del', 'nothing', 'space']
class_to_idx = {value:key for key,value in enumerate(idx_to_class)}

class DS(Dataset):
  def __init__(self, image_paths, transforms):
    self.image_paths = image_paths
    self.transforms = transforms

  def __len__(self):
    return len(self.image_paths)

  def __getitem__(self, idx):
    filename = self.image_paths[idx]
    #image = cv.imread(filename)
    #image = cv.cvtColor(image, cv.COLOR_BGR2RGB)
    image = None
    try:
      image = Image.open(filename)
    except:
      filename = self.image_paths[idx+1]
      image = Image.open(filename)
    label = filename.split('/')[-2]
    label = class_to_idx[label]

    if self.transforms is not None:
      image = self.transforms(image)

    return image, label

import matplotlib.pyplot as plt
import copy
%matplotlib inline

def vis_augment(dataset, index = 0, samples = 10, cols = 5, rand_img = False):
  dataset = copy.deepcopy(dataset)
  dataset.transforms = T.Compose(
    [
     T.GaussianBlur(9),
     T.RandomRotation((0, 10)),
     T.RandomHorizontalFlip(),
     T.RandomVerticalFlip(),
    ]
  )
  rows = samples//cols
  fig, ax = plt.subplots(nrows = rows, ncols = cols, figsize = (12, 8))

  for i in range(samples):
    if rand_img:
      idx = np.random.randint(1,len(train_img_paths))
      image, lab = dataset[idx]
      ax.ravel()[i].imshow(image)
      ax.ravel()[i].set_axis_off()
      ax.ravel()[i].set_title(idx_to_class[lab])
  plt.tight_layout(pad=1)
  plt.show()

vis_augment(train_ds,np.random.randint(1,len(train_img_paths)), rand_img = True)

lr = 0.1
weight_decay = 0.1
batch_size = 1
num_epochs = 5
num_classes = 29

train_dtldr = DataLoader(train_ds, batch_size=batch_size, shuffle=True)

device = torch.device('cuda' if torch.cuda.is_available else 'cpu')

import torch
import torch.nn as nn
import torch.optim as optim
import torch.nn.functional as F

class CNN(nn.Module):
  def __init__(self, input_size = (200, 200), num_classes = 29):
    super(CNN, self).__init__()
    #self.conv1=nn.Conv2d(in_channels=3,out_channels=12,kernel_size=3,stride=1,padding=1)
    #Shape= (256,12,150,150)
    #self.bn1=nn.BatchNorm2d(num_features=12)
    #Shape= (256,12,150,150)
    #self.relu1=nn.ReLU()
    #Shape= (256,12,150,150)
    #self.pool=nn.MaxPool2d(kernel_size=2)
    #Reduce the image size be factor 2
    #Shape= (256,12,75,75)
    #self.conv2=nn.Conv2d(in_channels=12,out_channels=20,kernel_size=3,stride=1,padding=1)
    #Shape= (256,20,75,75)
    #self.relu2=nn.ReLU()
    #Shape= (256,20,75,75)
    #self.conv3=nn.Conv2d(in_channels=20,out_channels=32,kernel_size=3,stride=1,padding=1)
    #Shape= (256,32,75,75)
    #self.bn3=nn.BatchNorm2d(num_features=32)
    #Shape= (256,32,75,75)
    #self.relu3=nn.ReLU()
    #Shape= (256,32,75,75)
    self.model_vgg = models.SqueezeNet(num_classes=29)
    self.model_vgg.classifier[1] = nn.Conv2d(512, num_classes, kernel_size=(1,1), stride=(1,1))

  def forward(self, input):
    #output=self.conv1(input)
    #output=self.bn1(output)
    #output=self.relu1(output)

    #output=self.pool(output)

    #output=self.conv2(output)
    #output=self.relu2(output)

    #output=self.conv3(output)
    #output=self.bn3(output)
    #output=self.relu3(output)

    #output=output.view(-1,320000)
    #output=self.fc(output)
    output = self.model_vgg(input)

    return output

model = CNN((200, 200), 29).to(device)
criterion = nn.CrossEntropyLoss()
adam = optim.Adam(model.model_vgg.parameters(), lr=lr, weight_decay=weight_decay)

total_right=0
step = 1
for i in range(num_epochs):
  for _, (x, y) in enumerate(train_dtldr):
    model.train()
    x = x.to(device)
    y = y.to(device)

    adam.zero_grad()

    scores = model(x).to(device)
    loss = criterion(scores, y).to(device)

    #print(scores.shape, y.shape)

    total_right+=int(torch.sum(torch.argmax(scores)==y.data))

    if step % 100 == 0:
      print(f'epoch:{i}\tstep:{step}')

    if step % 100 == 0:
      print(scores.data, y)
      print(f'-------------\npred = {torch.argmax(scores.data)}\nreal = {y.data}\nloss = {loss} \ntotal_right = {total_right} accuracy = {(total_right/(step+1))*100.00}%')

    loss.backward()

    adam.step()
    step+=1

Context: @XGBoosted_Learner Although, I haven’t gone through the entire code, can you try a small Learning rate, say 1e-3, and see if that solves your issue.


Question:
Greetings, everyone!
I would like to clarify the case with the image classifier, which I can describe in general, and then I will provide an example in Colab.
I have images of 7 classes that are easily predicted by a simple ResNet classifier with 98+% accuracy.  I modified my dataset a bit: I took samples from one class and randomly split them by half - the first half retained the previous class, the other half gained a new class. My goal was to confuse the classifier between two classes in the pair since the sample distribution in both classes was the same.  And I expected that the network would classify the samples from similar classes with equal probability: for example, if a single class from the previous setup was guessed with 98% accuracy, then in the new dataset the correct guesses would be 49% vs 49%, or at least without landslide difference.
But classifier preferred to classify samples from both subsets with one class, like 70% vs 30%.
I understand that I deal with bias but I wonder if this situation is normal?
The reason why I was doing so is my experiments with metadata, which I supply to the semantic segmentation network via separate input and this metadata is a hint about the samples domain.
I’m convinced that metadata becomes important when the network cannot guess from the source sample to which domain it belongs.  I quantify my assumptions on how easily the network can guess the domain with the ResNet classifier.
And I’m a bit uncomfortable that the classifier prefers one class when it cannot distinguish between two (or more…) classes.
I might be wrong and can reconcile with this bias if somebody from this respectable community can confirm this or point to some published discussion on the topic.
I prepared a test case for illustration:

      colab.research.google.com

Google Colaboratory

Here I overrode the CIFAR10 dataloader get_item method and when it takes an image of class 8 (ship), it can randomly make it a new class 11.
Similarly, the sample of class 9 ‘truck’ can be left the same or be assigned a new class 10 ‘truk_m’ (‘modified truck’).
The simple classifier from the PyTorch tutorial after training doesn’t distribute predictions equally between similar classes but usually prefers to predict one class more often (the confusion matrix at the bottom).

Context:
i think that it’s better to train model for more epochs. also you can use larger lr.
your loss is still to high, maybe training for a bit more give you a better result,
you can use weight decay to have more generalization.
i found these topics, i hope it helps you.

Expected calibration error (ECE) is a metric that compares neural network model output pseudo-probabilities to model accuracies . ECE values can be used to calibrate (adjust) a neural network model so that output pseudo-probabilities more closely match actual probabilities of a correct prediction.

Implicit model calibration:

By artificially softening the targets, label smoothing prevents the network from becoming overconfident. But does it improve the calibration of the model by making the confidence of its predictions more accurately represent their accuracy? In this section, we seek to answer this question. Guo et al. [15] have shown that modern neural networks are poorly calibrated and over-confident despite having better performance than better calibrated networks from the past. To measure calibration, the authors computed the estimated expected calibration error (ECE). They demonstrated that a simple post-processing step, temperature scaling, can reduce ECE and calibrate the network. Temperature scaling consists in multiplying the logits by a scalar before applying the softmax operator. Here, we show that label smoothing also reduces ECE and can be used to calibrate a network without the need for temperature scaling.

and i think you should check class probability of each sample not class accuracy


Torchvision example on Mixup and CrossEntropyLoss

Question:
In PyTorch’s recent vision examples here: vision/transforms.py at main · pytorch/vision · GitHub 1
and vision/train.py at main · pytorch/vision · GitHub, it was shown how to use Mixup with the pipeline. I noticed when trying to use their Mixup function on my own that CrossEntropyLoss in general don’t expect targets to be of one-hot encoded, and it threw me a RuntimeError: Expected object of scalar type Long but got scalar type Float for argument #2 'target' in call to _thnn_nll_loss_forward error.
Am I missing something from the documentation
class RandomMixup(torch.nn.Module):
    """Randomly apply Mixup to the provided batch and targets.
    The class implements the data augmentations as described in the paper
    `"mixup: Beyond Empirical Risk Minimization" <https://arxiv.org/abs/1710.09412>`_.
    Args:
        num_classes (int): number of classes used for one-hot encoding.
        p (float): probability of the batch being transformed. Default value is 0.5.
        alpha (float): hyperparameter of the Beta distribution used for mixup.
            Default value is 1.0.
        inplace (bool): boolean to make this transform inplace. Default set to False.
    """

    def __init__(self, num_classes: int, p: float = 0.5, alpha: float = 1.0, inplace: bool = False) -> None:
        super().__init__()
        assert num_classes > 0, "Please provide a valid positive value for the num_classes."
        assert alpha > 0, "Alpha param can't be zero."

        self.num_classes = num_classes
        self.p = p
        self.alpha = alpha
        self.inplace = inplace

    def forward(self, batch: Tensor, target: Tensor) -> Tuple[Tensor, Tensor]:
        """
        Args:
            batch (Tensor): Float tensor of size (B, C, H, W)
            target (Tensor): Integer tensor of size (B, )
        Returns:
            Tensor: Randomly transformed batch.
        """
        if batch.ndim != 4:
            raise ValueError(f"Batch ndim should be 4. Got {batch.ndim}")
        if target.ndim != 1:
            raise ValueError(f"Target ndim should be 1. Got {target.ndim}")
        if not batch.is_floating_point():
            raise TypeError(f"Batch dtype should be a float tensor. Got {batch.dtype}.")
        if target.dtype != torch.int64:
            raise TypeError(f"Target dtype should be torch.int64. Got {target.dtype}")

        if not self.inplace:
            batch = batch.clone()
            target = target.clone()

        if target.ndim == 1:
            target = torch.nn.functional.one_hot(target, num_classes=self.num_classes).to(dtype=batch.dtype)

        if torch.rand(1).item() >= self.p:
            return batch, target

        # It's faster to roll the batch by one instead of shuffling it to create image pairs
        batch_rolled = batch.roll(1, 0)
        target_rolled = target.roll(1, 0)

        # Implemented as on mixup paper, page 3.
        lambda_param = float(torch._sample_dirichlet(torch.tensor([self.alpha, self.alpha]))[0])
        batch_rolled.mul_(1.0 - lambda_param)
        batch.mul_(lambda_param).add_(batch_rolled)

        target_rolled.mul_(1.0 - lambda_param)
        target.mul_(lambda_param).add_(target_rolled)

        return batch, target

where they did one hot encoded the targets.

Context:
Hi Hongnan!

 Hongnan_G:

CrossEntropyLoss in general don’t expect targets to be of one-hot encoded,

Possibly relevant to your issue:
As of version 1.10, CrossEntropyLoss supports probabilistic targets (in
addition to integer-class-label targets).  One-hot-encoded targets can be
understood as a special case of probabilistic targets.  Earlier versions did
not support probabilistic targets (and hence not one-hot-encoded targets).
If you are seeing your issue with an earlier version, you might try upgrading
to the latest stable version, 1.10.1.
Best.
K. Frank


Vanilla vision transformer not returning the binary labels

Question:
I created embeddings for my patches and then feed them to the vanilla vision transformer for binary classification.
Here’s the forward method:
def forward(self, x):
        #x = self.to_patch_embedding(img)
        b, n, _ = x.shape

        cls_tokens = repeat(self.cls_token, '() n d -> b n d', b = b)
        x = torch.cat((cls_tokens, x), dim=1)
        #x += self.pos_embedding[:, :(n + 1)]
        x = self.dropout(x)

        x = self.transformer(x)

        x = x.mean(dim = 1) if self.pool == 'mean' else x[:, 0]

        x = self.to_latent(x)

        x = self.mlp_head(x)

        print('x is: ', x)
        return x

However, I am not getting labels as a result. How should I convert this to binary labels?
I get this error consequently:
x is:  tensor([[-8.7743e-01, -1.1380e-01],
        [-4.8789e-01,  5.5360e-04],
        [-7.1857e-01,  3.6758e-01],
        [-5.9797e-01,  2.3756e-01],
        [-6.1892e-01, -1.2594e-02],
        [-4.2626e-01,  1.5825e-01],
        [-8.1902e-01, -1.5155e-01],
        [-5.5616e-01,  1.7184e-02]], device='cuda:0', grad_fn=<AddmmBackward0>)
out shape is:  torch.Size([8, 2])
out is:  tensor([[-8.7743e-01, -1.1380e-01],
        [-4.8789e-01,  5.5360e-04],
        [-7.1857e-01,  3.6758e-01],
        [-5.9797e-01,  2.3756e-01],
        [-6.1892e-01, -1.2594e-02],
        [-4.2626e-01,  1.5825e-01],
        [-8.1902e-01, -1.5155e-01],
        [-5.5616e-01,  1.7184e-02]], device='cuda:0', grad_fn=<AddmmBackward0>)
labels are:  tensor([1, 1, 1, 1, 1, 1, 1, 1], dtype=torch.int32)

Traceback (most recent call last):
  File "main_classifier.py", line 250, in <module>
    pred,label,loss = trainer.train(sample_batched, model)

    pred, labels, loss = model.forward(feats, labels)
 File "/home/jalal/research/venv/dpcc/lib/python3.8/site-packages/torch/nn/parallel/data_parallel.py", line 166, in forward
    return self.module(*inputs[0], **kwargs[0])
  File "/home/jalal/research/venv/dpcc/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)

    loss = self.criterion(out.cuda(), labels.cuda())

 File "/home/jalal/research/venv/dpcc/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/jalal/research/venv/dpcc/lib/python3.8/site-packages/torch/nn/modules/loss.py", line 1150, in forward
    return F.cross_entropy(input, target, weight=self.weight,
  File "/home/jalal/research/venv/dpcc/lib/python3.8/site-packages/torch/nn/functional.py", line 2846, in cross_entropy
    return torch._C._nn.cross_entropy_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index, label_smoothing)
RuntimeError: "nll_loss_forward_reduce_cuda_kernel_2d_index" not implemented for 'Int'

I also call the transformer like this:
        labels = torch.IntTensor(labels)
        print('labels :', labels)
        stacked_X = torch.stack(X)
        out = self.transformer(stacked_X)
        print('out shape is: ', out.shape)
        print('out is: ', out)
        # loss
        print('labels are: ', labels)
        print(type(labels))
        loss = self.criterion(out.cuda(), labels.cuda())

Here’s the full code of vanilla vision transformer:
import torch
from torch import nn

from einops import rearrange, repeat
from einops.layers.torch import Rearrange

# helpers

def pair(t):
    return t if isinstance(t, tuple) else (t, t)

# classes

class PreNorm(nn.Module):
    def __init__(self, dim, fn):
        super().__init__()
        self.norm = nn.LayerNorm(dim)
        self.fn = fn
    def forward(self, x, **kwargs):
        return self.fn(self.norm(x), **kwargs)

class FeedForward(nn.Module):
    def __init__(self, dim, hidden_dim, dropout = 0.):
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(dim, hidden_dim),
            nn.GELU(),
            nn.Dropout(dropout),
            nn.Linear(hidden_dim, dim),
            nn.Dropout(dropout)
        )
    def forward(self, x):
        return self.net(x)

class Attention(nn.Module):
    def __init__(self, dim, heads = 8, dim_head = 64, dropout = 0.):
        super().__init__()
        inner_dim = dim_head *  heads
        project_out = not (heads == 1 and dim_head == dim)

        self.heads = heads
        self.scale = dim_head ** -0.5

        self.attend = nn.Softmax(dim = -1)
        self.to_qkv = nn.Linear(dim, inner_dim * 3, bias = False)

        self.to_out = nn.Sequential(
            nn.Linear(inner_dim, dim),
            nn.Dropout(dropout)
        ) if project_out else nn.Identity()

    def forward(self, x):
        qkv = self.to_qkv(x).chunk(3, dim = -1)
        q, k, v = map(lambda t: rearrange(t, 'b n (h d) -> b h n d', h = self.heads), qkv)

        dots = torch.matmul(q, k.transpose(-1, -2)) * self.scale

        attn = self.attend(dots)

        out = torch.matmul(attn, v)
        out = rearrange(out, 'b h n d -> b n (h d)')
        return self.to_out(out)

class Transformer(nn.Module):
    def __init__(self, dim, depth, heads, dim_head, mlp_dim, dropout = 0.):
        super().__init__()
        self.layers = nn.ModuleList([])
        for _ in range(depth):
            self.layers.append(nn.ModuleList([
                PreNorm(dim, Attention(dim, heads = heads, dim_head = dim_head, dropout = dropout)),
                PreNorm(dim, FeedForward(dim, mlp_dim, dropout = dropout))
            ]))
    def forward(self, x):
        for attn, ff in self.layers:
            x = attn(x) + x
            x = ff(x) + x
        return x

class VisionTransformer(nn.Module):
    def __init__(self, *, image_size=256, patch_size=16, dim=512, depth=4, heads=12, mlp_dim=256, dropout=0.25, pool = 'cls', channels = 3, dim_head=64, emb_dropout = 0., num_classes=2):
        super().__init__()
        image_height, image_width = pair(image_size)
        patch_height, patch_width = pair(patch_size)

        assert image_height % patch_height == 0 and image_width % patch_width == 0, 'Image dimensions must be divisible by the patch size.'

        num_patches = (image_height // patch_height) * (image_width // patch_width)
        patch_dim = channels * patch_height * patch_width
        assert pool in {'cls', 'mean'}, 'pool type must be either cls (cls token) or mean (mean pooling)'

        self.to_patch_embedding = nn.Sequential(
            Rearrange('b c (h p1) (w p2) -> b (h w) (p1 p2 c)', p1 = patch_height, p2 = patch_width),
            nn.Linear(patch_dim, dim),
        )

        self.pos_embedding = nn.Parameter(torch.randn(1, num_patches + 1, dim))
        self.cls_token = nn.Parameter(torch.randn(1, 1, dim))
        self.dropout = nn.Dropout(emb_dropout)

        self.transformer = Transformer(dim, depth, heads, dim_head, mlp_dim, dropout)

        self.pool = pool
        self.to_latent = nn.Identity()

        self.mlp_head = nn.Sequential(
            nn.LayerNorm(dim),
            nn.Linear(dim, num_classes)
        )

    def forward(self, x):
        #x = self.to_patch_embedding(img)
        b, n, _ = x.shape

        cls_tokens = repeat(self.cls_token, '() n d -> b n d', b = b)
        x = torch.cat((cls_tokens, x), dim=1)
        #x += self.pos_embedding[:, :(n + 1)]
        x = self.dropout(x)

        x = self.transformer(x)

        x = x.mean(dim = 1) if self.pool == 'mean' else x[:, 0]

        x = self.to_latent(x)

        x = self.mlp_head(x)

        print('x is: ', x)
        return x

My batch size here is 8 and labels are 0 and 1.

Context:
The following worked for me:
def forward(self, X, labels, is_print=False):
        stacked_X = torch.stack(X)
        out = self.transformer(stacked_X)
        with torch.autocast('cuda'):
            loss = self.criterion(out, torch.tensor(labels).cuda())
        pred = out.data.max(1)[1]
        return pred, labels, loss


train_loss.backward(retain_graph=True) AttributeError: ‘float’ object has no attribute ‘backward’

Question:
I am doing binary classification with a batch size of 8 and the output of my model is:
out shape is:  torch.Size([8, 2])
out is:  tensor([[-1.2086,  0.8501],
        [-0.8705, -0.2427],
        [-1.0448,  0.5410],
        [-1.2318,  0.2670],
        [-1.0920,  0.0126],
        [-0.9867,  0.1056],
        [-1.2848,  0.9758],
        [-1.2415, -0.2882]], device='cuda:0', grad_fn=<AddmmBackward0>)

and I am getting this error:
train loss:  0.0
Traceback (most recent call last):
  File "main_classifier.py", line 253, in <module>
    train_loss.backward(retain_graph=True)
AttributeError: 'float' object has no attribute 'backward'

How can I solve this error?

Context: Your train_loss seems to be a plain float (i.e. not a tensor) so make sure you are calling backward on the actual loss tensor not the Python float scalar.


Error on torch.load() (PytorchStreamReader failed)

Question:
Hi,
I was trying to load the pytorch model but facing an unexpected error. I do not disturb the folder structure and still getting this error.
>>> torch.load("outputs/test_validation_loss_logging/model_001650.pth", map_location="cpu")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/pritesh/anaconda3/envs/pytorch/lib/python3.7/site-packages/torch/serialization.py", line 586, in load
    with _open_zipfile_reader(f) as opened_zipfile:
  File "/home/pritesh/anaconda3/envs/pytorch/lib/python3.7/site-packages/torch/serialization.py", line 246, in __init__
    super(_open_zipfile_reader, self).__init__(torch._C.PyTorchFileReader(name_or_buffer))
RuntimeError: [enforce fail at inline_container.cc:143] . PytorchStreamReader failed reading zip archive: failed finding central directory

Linux OS
Python3.7
torch version 1.5.0
Check how the model is saved here 176
.

Context: Ok, I’m able to load the model. The problem was with the saved weight file. It wasn’t saved properly and the weight file size was smaller (only 90 MB instead of 200 MB).


Where is the actual implementation/source code for nn.PixelShuffle?

Question:
I can find just a docstring when I go to torch.nn.functional.pixel_shuffle. Where can I find the actual implementation?

      github.com

    pytorch/pytorch/blob/master/torch/nn/modules/pixelshuffle.py

      from .module import Module
from .. import functional as F

from torch import Tensor

class PixelShuffle(Module):
    r"""Rearranges elements in a tensor of shape :math:`(*, C \times r^2, H, W)`
    to a tensor of shape :math:`(*, C, H \times r, W \times r)`, where r is an upscale factor.

    This is useful for implementing efficient sub-pixel convolution
    with a stride of :math:`1/r`.

    See the paper:
    `Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network`_
    by Shi et. al (2016) for more details.

    Args:
        upscale_factor (int): factor to increase spatial resolution by

  This file has been truncated. show original

Context:
You won’t be able to locally change a C++ source file and see the changes without rebuilding it.
To apply changes to the PixelShuffle implementation, use this guide to build PyTorch from source, manipulate the file, and rebuild it.
This Contributing README might also be useful for incremental builds.


Using dataloader RecursionError: maximum recursion depth exceeded while calling a Python object

Question:
Hi, I’m using dataloader with 60060 images, with batch size set to 5, so the length of my dataloader is 12012.
I got the error :
RecursionError: maximum recursion depth exceeded while calling a Python object

What should I do?
I suppose it’s because the image’s number is too big, so how to divide the dataloder or make it work little by little?
Thank you in advance
More detailed traceback:
Traceback (most recent call last):
  File "Training.py", line 141, in <module>
    for i_batch, sample_batched in enumerate(dataloader):
  File "/home/xiaoyu/miniconda3/envs/deep_mol/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 637, in __next__
    return self._process_next_batch(batch)
  File "/home/xiaoyu/miniconda3/envs/deep_mol/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 658, in _process_next_batch
    raise batch.exc_type(batch.exc_msg)
RecursionError: Traceback (most recent call last):
  File "/home/xiaoyu/miniconda3/envs/deep_mol/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 138, in _worker_loop
    samples = collate_fn([dataset[i] for i in batch_indices])
  File "/home/xiaoyu/miniconda3/envs/deep_mol/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 138, in <listcomp>
    samples = collate_fn([dataset[i] for i in batch_indices])
  File "/home/xiaoyu/miniconda3/envs/deep_mol/lib/python3.6/site-packages/torch/utils/data/dataset.py", line 81, in __getitem__
    return self.datasets[dataset_idx][sample_idx]
  File "/home/xiaoyu/miniconda3/envs/deep_mol/lib/python3.6/site-packages/torch/utils/data/dataset.py", line 81, in __getitem__
    return self.datasets[dataset_idx][sample_idx]
  File "/home/xiaoyu/miniconda3/envs/deep_mol/lib/python3.6/site-packages/torch/utils/data/dataset.py", line 81, in __getitem__
    return self.datasets[dataset_idx][sample_idx]
  [Previous line repeated 321 more times]
  File "Training.py", line 67, in __getitem__
    T1a_arr = io.imread(os.path.join(T1a_dir, T1a_str))
  File "/home/xiaoyu/miniconda3/envs/deep_mol/lib/python3.6/site-packages/skimage/io/_io.py", line 61, in imread
    with file_or_url_context(fname) as fname:
  File "/home/xiaoyu/miniconda3/envs/deep_mol/lib/python3.6/contextlib.py", line 159, in helper
    return _GeneratorContextManager(func, args, kwds)
RecursionError: maximum recursion depth exceeded while calling a Python object

Context: Thank you, I set sys.setrecursionlimit(10000) and the traceback disappear


If I have images saved in many folders, is there a way to begin training without loading all the file paths first?

Question:
Hi,
I have around a million images I want to train spread across around 50 folders.  Do I have to load in all the filenames of every image.  Right now, I have my Dataset set up so it does an os.walk across all the directories and retrieves a list of all filenames in the init function.  This takes several hours.  Is there a way to just start on one folder and start loading image paths for training while simultaneously loading the batch size of images and starting training?  Below is my code.
Thank you!

class Custom_Dataset(Dataset):
  def  __init__( self, transform=None, config=None):
        self.samples = []
        for crop_path in config.dataset_crops_path_list:
            for root, subdirs, filenames in os.walk(crop_path):
                file_name_list = [f for f in filenames]    #MAJOR TIME SINK

                for filename in file_name_list:
                        self.samples.append(os.path.join(root, filename))

    self.targets = []
    for ii in tqdm(self.samples):
        image_label = os.path.basename(ii)
        self.targets.append(image_label)

    self.transform = transform
    self.output_shape = config.input_sz

def __len__(self):
    return len(self.samples)

def __getitem__(self, idx):
    if torch.is_tensor(idx):
        idx = idx.tolist()
    image_path = self.samples[idx]
    image = io.imread(image_path)
    image_label = self.targets[idx]

    non_transformed_image = transform.resize(image.copy(), (self.output_shape, self.output_shape), preserve_range=True)

    if self.transform:
        image = self.transform(image)
        return image, image_label
    return non_transformed_image, image_label

Context:
Vivek_Reddy:

Right now, I have my Dataset set up so it does an os.walk across all the directories and retrieves a list of all filenames in the init function. This takes several hours.

This seems quite unexpected. I’ve just tested os.walk on a full ImageNet (~1000 folders) and it finishes in ~2s.