Repository: facebookresearch/fairseq

Issue #3693: Error when using Wav2vec-u to launch GAN training
Created: 2021-07-08T14:22:35Z
❓ Questions and Help
Hi, everyone. I am using the wav2vec Unsupervised by following indicates on README. However, when I executed the below command, it raised the error AttributeError: 'NoneType' object has no attribute 'transpose'.
Commands :
PREFIX=w2v_unsup_gan_xp
TASK_DATA=/workspace/wav2vec_u/fairseq/examples/wav2vec/dataset/prepare_audio/precompute_pca512_cls128_mean_pooled
TEXT_DATA=/workspace/wav2vec_u/fairseq/examples/wav2vec/dataset/data/phones
KENLM_PATH=/workspace/wav2vec_u/fairseq/examples/wav2vec/dataset/data/phones/lm.phones.filtered.04.bin
fairseq-hydra-train \
    -m --config-dir unsupervised/config/gan \
    --config-name w2vu \
    task.data=${TASK_DATA} \
    task.text_data=${TEXT_DATA} \
    task.kenlm_path=${KENLM_PATH} \
    common.user_dir=examples/wav2vec/unsupervised \
    model.code_penalty=2,4 model.gradient_penalty=1.5,2.0 \
    model.smoothness_weight=0.5,0.75,1.0 'common.seed=range(0,5)'
Here is all the log and error information.
[2021-07-08 13:50:57,269][fairseq_cli.train][INFO] - task: UnpairedAudioText
[2021-07-08 13:50:57,269][fairseq_cli.train][INFO] - model: Wav2vec_U
[2021-07-08 13:50:57,269][fairseq_cli.train][INFO] - criterion: ModelCriterion
[2021-07-08 13:50:57,269][fairseq_cli.train][INFO] - num. shared model params: 1,166,337 (num. trained: 1,166,337)
[2021-07-08 13:50:57,269][fairseq_cli.train][INFO] - num. expert model params: 0 (num. trained: 0)
[2021-07-08 13:50:57,276][unsupervised.data.extracted_features_dataset][INFO] - loaded 1680, skipped 0 samples
[2021-07-08 13:50:57,276][unsupervised.tasks.unpaired_audio_text][INFO] - split valid has unpaired text? False
[2021-07-08 13:50:58,923][fairseq.utils][INFO] - ***********************CUDA enviroments for all 1 workers***********************
[2021-07-08 13:50:58,924][fairseq.utils][INFO] - rank   0: capabilities =  6.1  ; total memory = 10.917 GB ; name = GeForce GTX 1080 Ti                     
[2021-07-08 13:50:58,924][fairseq.utils][INFO] - ***********************CUDA enviroments for all 1 workers***********************
[2021-07-08 13:50:58,924][fairseq_cli.train][INFO] - training on 1 devices (GPUs/TPUs)
[2021-07-08 13:50:58,924][fairseq_cli.train][INFO] - max tokens per device = 200000 and max sentences per device = 16
[2021-07-08 13:50:58,925][fairseq.trainer][INFO] - Preparing to load checkpoint ./checkpoint_last.pt
[2021-07-08 13:50:58,926][fairseq.trainer][INFO] - No existing checkpoint found ./checkpoint_last.pt
[2021-07-08 13:50:58,926][fairseq.trainer][INFO] - loading train data for epoch 1
[2021-07-08 13:50:58,942][unsupervised.data.extracted_features_dataset][INFO] - loaded 4620, skipped 0 samples
[2021-07-08 13:50:58,942][unsupervised.tasks.unpaired_audio_text][INFO] - split train has unpaired text? False
[2021-07-08 13:50:59,080][fairseq.optim.adam][INFO] - using FusedAdam
[2021-07-08 13:50:59,146][fairseq.optim.adam][INFO] - using FusedAdam
[2021-07-08 13:50:59,154][fairseq.logging.progress_bar][WARNING] - tensorboard not found, please install with: pip install tensorboard
[2021-07-08 13:50:59,157][fairseq.trainer][INFO] - begin training epoch 1
[2021-07-08 13:50:59,157][fairseq_cli.train][INFO] - Start iterating over samples
Traceback (most recent call last):
  File "/tmp/fairseq/fairseq_cli/hydra_train.py", line 45, in hydra_main
    distributed_utils.call_main(cfg, pre_main)
  File "/tmp/fairseq/fairseq/distributed/utils.py", line 369, in call_main
    main(cfg, **kwargs)
  File "/tmp/fairseq/fairseq_cli/train.py", line 180, in main
    valid_losses, should_stop = train(cfg, trainer, task, epoch_itr)
  File "/usr/lib/python3.6/contextlib.py", line 52, in inner
    return func(*args, **kwds)
  File "/tmp/fairseq/fairseq_cli/train.py", line 291, in train
    log_output = trainer.train_step(samples)
  File "/usr/lib/python3.6/contextlib.py", line 52, in inner
    return func(*args, **kwds)
  File "/tmp/fairseq/fairseq/trainer.py", line 710, in train_step
    ignore_grad=is_dummy_batch,
  File "/tmp/fairseq/fairseq/tasks/fairseq_task.py", line 502, in train_step
    loss, sample_size, logging_output = criterion(model, sample)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/tmp/fairseq/fairseq/criterions/model_criterion.py", line 50, in forward
    net_output = model(**sample["net_input"])
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/tmp/fairseq/fairseq/../examples/wav2vec/unsupervised/models/wav2vec_u.py", line 567, in forward
    token_y = self.discriminator(token_x, token_padding_mask)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/tmp/fairseq/fairseq/../examples/wav2vec/unsupervised/models/wav2vec_u.py", line 266, in forward
    x = x.transpose(1, 2)  # BTC -> BCT
AttributeError: 'NoneType' object has no attribute 'transpose'
Can someone help on this ? Thanks a lot in advance.

Duplicate issue #5283: Cannot run GAN training for wav2vec-U 2.0 
Created: 2023-08-16T07:11:31Z
❓ Questions and Help
Before asking:

search the issues.
search the docs.


What is your question?
I tried to reproduce wav2vec-U 2.0 with python 3.8. When I ran the GAN training script from README, however, I got the following message.
File "/mnt/hdd/esyoon/anaconda3/envs/iclr24_38/lib/python3.8/site-packages/hydra/_interal/core_plugins/importlib_resources_config_source.py", line 72, in available
    ret = resources.is_resource(self.path, "__init__.py") # type: ignore
AttributeError: module 'importlib_resources' has no attribute 'is_resource'

I assumed this was because some kind of version mismatch inside hydra and python, so I changed my environment to python 3.9. However, this time I have encountered the following error.
File "/mnt/hdd/esyoon/anaconda3/envs/iclr24/lib/python3.9/site-packages/omegaconf/_utils.py", line 610, in _raise
    raise ex # set and OC_CAUSE=1 for full backtrace
omegaconf.errors.ConfigKeyError: Key 'submitit_folder' not in 'BasicLauncherConf'
        full_key: hydra.launcher.submitit_folder
        reference_type=Optional[BasicLauncherConf]
        object_type=BasicLauncherConf

I think this has something to do with config/gan/w2vu2.yaml hydra part:
hydra:
  job:
    config:
      override_dirname:
        kv_sep: ':'
        item_sep: '__'
        exclude_keys:
          - run_config
          - distributed_training.distributed_port
          - common.user_dir
          - task.data
          - task.kenlm_path
          - task.text_data
          - model.generator_layers
          - task.labels
          - task.force_model_seed
  sweep:
    dir: /checkpoint/${env:USER}/${env:PREFIX}/${hydra.job.config_name}/${hydra.job.override_dirname}
    subdir: ${hydra.job.num}
  launcher:
    submitit_folder: ${hydra.sweep.dir}
    timeout_min: 3000
    cpus_per_task: 10
    gpus_per_node: 1
    tasks_per_node: 1
    mem_gb: 120
    nodes: 1
    name: ${env:PREFIX}_${hydra.job.config_name}
    partition: devlab,learnlab,learnfair,scavenge
    comment: intern_endding_soon
    constraint: volta32gb
    max_num_timeout: 30

I am assuming I need to edit the sweep directory but I have not confidence.
My question is:
(1) To run the code in python 3.8, how should I fix the problem above?
(2) What should I do with hydra.sweep.dir ?
Code

What have you tried?
Changed python version
searched for relevant issues in fairseq and hydra
What's your environment?

fairseq Version (e.g., 1.0 or main): main
PyTorch Version (e.g., 1.0) 2.0.1+cu117
OS (e.g., Linux): Linux
How you installed fairseq (pip, source): source?
Build command you used (if compiling from source): pip install --editable ./
Python version: 3.8 or 3.9
CUDA/cuDNN version: 11.6
GPU models and configuration: Quadro RTX 8000
Any other relevant information:

Closing comment (2023-08-27T21:00:12Z): Sorry, but I am still having issue on training GAN for wav2vec-U 2.0. The issue is same as #3693 which is
  File "/mnt/hdd/esyoon/workspace/fairseq/examples/wav2vec/unsupervised/models/wav2vec_u.py", line 266, in forward
    x = x.transpose(1, 2)  # BTC -> BCT
AttributeError: 'NoneType' object has no attribute 'transpose'

Although the issue is closed, it is due to a prolonged period of inactivity and I still do not have a clear clue on how to fix this error. I do find that random_label=None in the forward pass is causing such error, but still I do not get the resolution.