internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: speechbrain/speechbrain

Issue:
pytorch 1.5 much slower than 1.4
I did some first experiments, and apparently the issue could be connected to jit. In fact, the significant slow down happens when using our custom rnn called ligru (that is the only module we compile):
model      |      pytorch1.5         |    pytorch1.4
ligru-jit           2 min 51 sec            1min 52 sec
ligru-nojit       3 min 32 sec            3min 45 sec

In practice, for some reason, jit is much more effective in pytorch 1.4 rather than pytorch 1.5 (at least on our ligru). I tried to simplify the model (e.g, removing batch norm, dropout, bidirectional, etc) and the issue still appears.


Discussion:
Do we have some news from PyTorch on that issue ? 
During the monthly meeting with the pytorch team, they told me that they
are considering it and let me know.

On Sun, 7 Jun 2020 at 13:23, Parcollet Titouan <notifications@github.com>
wrote:

> Do we have some news from PyTorch on that issue ?
>
> —
> You are receiving this because you were assigned.
> Reply to this email directly, view it on GitHub
> <https://github.com/speechbrain/speechbrain/issues/54#issuecomment-640251562>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AEA2ZVT3HQO6VBMNNF2F6F3RVPEKDANCNFSM4M5PZIFQ>
> .
>


Resolving patch:
diff --git a/recipes/minimal_examples/neural_networks/ASR_seq2seq/example_asr_seq2seq_experiment.py b/recipes/minimal_examples/neural_networks/ASR_seq2seq/example_asr_seq2seq_experiment.py
--- a/recipes/minimal_examples/neural_networks/ASR_seq2seq/example_asr_seq2seq_experiment.py
+++ b/recipes/minimal_examples/neural_networks/ASR_seq2seq/example_asr_seq2seq_experiment.py
@@ -108,4 +108,4 @@ def on_epoch_end(self, epoch, train_stats, valid_stats):
 
 
 def test_error():
-    assert seq2seq_brain.avg_train_loss < 0.3
+    assert seq2seq_brain.avg_train_loss < 1.0
diff --git a/speechbrain/nnet/RNN.py b/speechbrain/nnet/RNN.py
--- a/speechbrain/nnet/RNN.py
+++ b/speechbrain/nnet/RNN.py
@@ -12,6 +12,9 @@
     LocationAwareAttention,
 )
 
+# Temporany fix for the jit speed issue
+torch._C._jit_set_profiling_executor(False)
+
 logger = logging.getLogger(__name__)
 
 


Test patch:
Repository: speechbrain/speechbrain

Issue:
Stats precision of FileTrainLogger
Now, all the stats logged by a FileTrainLogger have the precision 2 after their decimal points. In some training scenarios, precision 2 is not enough for some stats. I suggest allowing users to decide precision for each stats or adding precision number to 4 or 5 uniformly.


Discussion:
Good point, @pplantinga do you think we can set a hyperparameter for that in the logger?  By default it can be 2, but users might want a higher resolution...

Resolving patch:
diff --git a/speechbrain/utils/train_logger.py b/speechbrain/utils/train_logger.py
--- a/speechbrain/utils/train_logger.py
+++ b/speechbrain/utils/train_logger.py
@@ -49,21 +49,24 @@ class FileTrainLogger(TrainLogger):
     ---------
     save_file : str
         The file to use for logging train information.
+    precision : int
+        Number of decimal places to display. Default 2, example: 1.35e-5
     summary_fns : dict of str:function pairs
         Each summary function should take a list produced as output
         from a training/validation pass and summarize it to a single scalar.
     """
 
-    def __init__(self, save_file, summary_fns=None):
+    def __init__(self, save_file, precision=2, summary_fns=None):
         self.save_file = save_file
+        self.precision = precision
         self.summary_fns = summary_fns or {}
 
     def _item_to_string(self, key, value, dataset=None):
         """Convert one item to string, handling floats"""
-        if isinstance(value, float) and 0.01 < value < 100.0:
-            value = f"{value:.2f}"
+        if isinstance(value, float) and 1.0 < value < 100.0:
+            value = f"{value:.{self.precision}f}"
         elif isinstance(value, float):
-            value = f"{value:.2e}"
+            value = f"{value:.{self.precision}e}"
         if dataset is not None:
             key = f"{dataset} {key}"
         return f"{key}: {value}"


Test patch:
Repository: speechbrain/speechbrain

Issue:
Transformer Decoder
Hi,

I just found that the last step in the transformer decoder is incorrect.

https://github.com/speechbrain/speechbrain/blob/master/speechbrain/lobes/models/transformer/Transformer.py#L482

https://github.com/speechbrain/speechbrain/blob/master/speechbrain/lobes/models/transformer/Transformer.py#L485



Discussion:
Hi @hwidong-na ,

Thanks for pointing out! That's indeed a bug in my previous implementation. However, it should be fixed in #271... Maybe you can change it for now at your working branch.
Hi @JianyuanZhong ,

Thanks for prompt response! I will switch to the new branch as soon as I figure out how to shift to the new architecture.
The most recent changes should be in this PR #289. Sorry for the confusion!

The new architecture is not yet stable with DistributedDataParrall and I am currently working on it. However, the transformer should work fine (when I l tested it last time with 1 gpu).

I will notify you when I have a working version with DDP!

Resolving patch:
diff --git a/speechbrain/lobes/models/transformer/Transformer.py b/speechbrain/lobes/models/transformer/Transformer.py
--- a/speechbrain/lobes/models/transformer/Transformer.py
+++ b/speechbrain/lobes/models/transformer/Transformer.py
@@ -479,11 +479,11 @@ def forward(
         tgt = tgt + self.dropout2(tgt2)
         tgt = self.norm2(tgt)
 
-        tgt = self.pos_ffn(tgt)
+        tgt2 = self.pos_ffn(tgt)
 
         # add & norm
-        tgt = tgt + self.dropout2(tgt2)
-        tgt = self.norm2(tgt)
+        tgt = tgt + self.dropout3(tgt2)
+        tgt = self.norm3(tgt)
 
         return tgt, self_attn, multihead_attention
 


Test patch:
Repository: speechbrain/speechbrain

Issue:
missing `device=(params["device"])` in load_collected()
running the experiment, i get an error complaining about device unavailable. due to this line: https://github.com/speechbrain/speechbrain/blob/e1162299e5fc6241829e41559217433de61c785d/recipes/AMI/Diarization/experiment.py#L560

if i change the line to:
`params["pretrainer"].load_collected(device=(params["device"]))`
the problem is solved. 


Discussion:
Good catch, care to make a PR?

Resolving patch:
diff --git a/recipes/AMI/Diarization/experiment.py b/recipes/AMI/Diarization/experiment.py
--- a/recipes/AMI/Diarization/experiment.py
+++ b/recipes/AMI/Diarization/experiment.py
@@ -557,7 +557,7 @@ def audio_pipeline(wav):
     # We download the pretrained Model from HuggingFace (or elsewhere depending on
     # the path given in the YAML file).
     run_on_main(params["pretrainer"].collect_files)
-    params["pretrainer"].load_collected()
+    params["pretrainer"].load_collected(device=(params["device"]))
     params["embedding_model"].eval()
     params["embedding_model"].to(params["device"])
 


Test patch:
Repository: speechbrain/speechbrain

Issue:
Installation into user site directory with editable source fails
Currently, the installation of speechbrain fails for a user site directory with an editable source if the user does not have write access to the base-Python site-packages directory.
Both the problem and the solution are also described in https://github.com/pypa/pip/issues/7953 


Discussion:

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -1,8 +1,15 @@
 #!/usr/bin/env python3
 import os
+import sys
+import site
 import setuptools
 from distutils.core import setup
 
+
+# Editable install in user site directory can be allowed with this hack:
+# https://github.com/pypa/pip/issues/7953.
+site.ENABLE_USER_SITE = "--user" in sys.argv[1:]
+
 with open("README.md") as f:
     long_description = f.read()
 


Test patch:
Repository: speechbrain/speechbrain

Issue:
CyclicLRScheduler not saving
We cannot add the CyclicLRScheduler to the checkpointer, as it has no saving hook registered. Adding @checkpoints.register_checkpoint_hooks above the class seems to do the trick (as done for the other schedulers), however I am not sure if this introduces other problems.

Cheers


Discussion:

Resolving patch:
diff --git a/speechbrain/nnet/schedulers.py b/speechbrain/nnet/schedulers.py
--- a/speechbrain/nnet/schedulers.py
+++ b/speechbrain/nnet/schedulers.py
@@ -553,6 +553,7 @@ def load(self, path, end_of_epoch=False, device=None):
         self.patience_counter = data["patience_counter"]
 
 
+@checkpoints.register_checkpoint_hooks
 class CyclicLRScheduler:
     """This implements a cyclical learning rate policy (CLR).
     The method cycles the learning rate between two boundaries with


Test patch:
Repository: speechbrain/speechbrain

Issue:
TypeError: Invalid file: WindowsPath('example_mandarin.wav')
When I tried to run code in WINDOWS
```python
from speechbrain.pretrained import EncoderDecoderASR
asr_model = EncoderDecoderASR.from_hparams(source="speechbrain/asr-wav2vec2-transformer-aishell",
                                               savedir="pretrained_models/asr-wav2vec2-transformer-aishell")
res = asr_model.transcribe_file('speechbrain/asr-wav2vec2-transformer-aishell/example_mandarin.wav')
```
I got the error:
![image](https://user-images.githubusercontent.com/72511133/140745987-dcacd316-8273-4d00-a5e7-88f67e875dd6.png)

I looked at some of the code located in the file "\speechbrain\pretrained\interfaces.py":
![image](https://user-images.githubusercontent.com/72511133/140747347-7f143dd8-d9ce-4ecc-891b-ca654c93c86c.png)

and I found that the function `torchaudio.load(path, channels_first=False)` requires a `str`. However, the function `fetch`  in 
 this file return an object with the type  `pathlib.WindowsPath` , so actually I use it instead of an `str`. And this raised the error.

I wonder if this is an incompatibility issue on Windows? I have avoided this problem in some other way, but I sincerely hope it can be resolved.


Discussion:

Resolving patch:
diff --git a/speechbrain/pretrained/interfaces.py b/speechbrain/pretrained/interfaces.py
--- a/speechbrain/pretrained/interfaces.py
+++ b/speechbrain/pretrained/interfaces.py
@@ -148,7 +148,7 @@ def load_audio(self, path, savedir="."):
         """
         source, fl = split_path(path)
         path = fetch(fl, source=source, savedir=savedir)
-        signal, sr = torchaudio.load(path, channels_first=False)
+        signal, sr = torchaudio.load(str(path), channels_first=False)
         return self.audio_normalizer(signal, sr)
 
     def _compile_jit(self):


Test patch:
Repository: speechbrain/speechbrain

Issue:
Broken docs for `speechbrain.alignment.ctc_segmentation`
Hi, thanks for maintaining such a wonderful library.

Looks like the documentation for `speechbrain.alignment.ctc_segmentation` is broken:
https://speechbrain.readthedocs.io/en/latest/API/speechbrain.alignment.ctc_segmentation.html

I guess this is caused by unneeded shebang, as shown in the following:
https://github.com/speechbrain/speechbrain/blob/develop/speechbrain/alignment/ctc_segmentation.py#L1-L2

Perhaps this could be related to #819 ?


Discussion:
Hi @seungwonpark, seems like the issue might be that the ctc-segmentation package (https://github.com/lumaku/ctc-segmentation) needs to be installed before compiling the docs. So, perhaps a solution could be to add `ctc-segmentation` to `requirements.txt` file. It is then assumed that along with the top level repo requirements and the additional requirements for docs in `docs/docs-requirements.txt`, needs to be installed before compiling the html docs using `make html`. I could add a PR if you agree @Gastron 
Sure, feel free to do a PR!

On Sat, Nov 13, 2021, 3:49 PM Amrit Krishnan ***@***.***>
wrote:

> Hi @seungwonpark <https://github.com/seungwonpark>, seems like the issue
> might be that the ctc-segmentation package (
> https://github.com/lumaku/ctc-segmentation) needs to be installed before
> compiling the docs. So, perhaps a solution could be to add
> ctc-segmentation to docs/docs-requirements.txt file before compiling the
> html docs using make html. I could add a PR if you agree.
>
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/speechbrain/speechbrain/issues/1119#issuecomment-968133931>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AEA2ZVUNN7PYXOSZSSINIA3UL3FO3ANCNFSM5H6STQ6Q>
> .
>

I think this could be a case where mocking the import in the docs compilation process would be best. Adding the library here, basically:
https://github.com/speechbrain/speechbrain/blob/13798b9576779b82f70c8f65dc0e166da5c09c10/docs/conf.py#L72

I'd be happy to look at the PR!

Resolving patch:
diff --git a/docs/conf.py b/docs/conf.py
--- a/docs/conf.py
+++ b/docs/conf.py
@@ -69,7 +69,7 @@
 autodoc_default_options = {}
 
 # Autodoc mock extra dependencies:
-autodoc_mock_imports = ["numba", "sklearn"]
+autodoc_mock_imports = ["sklearn"]
 
 # Order of API items:
 autodoc_member_order = "bysource"


Test patch:
Repository: speechbrain/speechbrain

Issue:
Inconsistency between documentation and the code of evaluate() function in the brain class
Hi I've noticed that there are some inconsistencies in the code and docs of the evaluate() method. 


The code shown here does not have a return value: https://speechbrain.readthedocs.io/en/latest/_modules/speechbrain/core.html#Brain.evaluate

And in the docs https://speechbrain.readthedocs.io/en/latest/API/speechbrain.core.html#speechbrain.core.Brain, it is mentioned that it returns average loss. 
Inconsistency between documentation and the code of evaluate() function in the brain class
Hi I've noticed that there are some inconsistencies in the code and docs of the evaluate() method. 


The code shown here does not have a return value: https://speechbrain.readthedocs.io/en/latest/_modules/speechbrain/core.html#Brain.evaluate

And in the docs https://speechbrain.readthedocs.io/en/latest/API/speechbrain.core.html#speechbrain.core.Brain, it is mentioned that it returns average loss. 


Discussion:

Resolving patch:
diff --git a/speechbrain/core.py b/speechbrain/core.py
--- a/speechbrain/core.py
+++ b/speechbrain/core.py
@@ -1233,6 +1233,7 @@ def evaluate(
                 self.on_stage_end, args=[Stage.TEST, avg_test_loss, None]
             )
         self.step = 0
+        return avg_test_loss
 
     def update_average(self, loss, avg_loss):
         """Update running average of the loss.


Test patch:
Repository: speechbrain/speechbrain

Issue:
WebsetDataset version when using Voxlingua107 recipe sample
Hi guys. I have used recipe sample of Voxlingua107 doing language detection research.

When I run the script, I found that the WebDataset version of this script should be used as the latest one (https://github.com/speechbrain/speechbrain/commit/4846380c5b60216a8ead0e34ee206e80665c605e).   
However the dataload function of speechbrain dataio (https://github.com/speechbrain/speechbrain/blob/develop/speechbrain/dataio/dataloader.py) still using the function `wds.dataset.Composable` which failed import from the latest version WebDataset.

**Would you please suggest the correct version of WebDataset and Speechbrain?**

Here is the details of my machine environment:
Python==3.8.11.   
Speechbrain==0.5.11 (Installed by pip).  
Webdataset==0.1.62 (The version above this do not include the function wds.dataset would cause training failed).  


Discussion:
I believe I have worked with `webdataset==0.1.76` but we should update the WebDataset checking code in SpeechBrain.
This is a simple fix right? Just change `wds.dataset.Composable` to `wds.WebDataset` in this one file.
Do you think we can set up some tests to make sure everything is working
well?

On Fri, May 27, 2022 at 12:31 PM Peter Plantinga ***@***.***>
wrote:

> This is a simple fix right? Just change wds.dataset.Composable to
> wds.WebDataset in this one file.
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/speechbrain/speechbrain/issues/1400#issuecomment-1139777916>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AEA2ZVVIJUPYPXL64RALMW3VMD2FNANCNFSM5WEATKLQ>
> .
> You are receiving this because you are subscribed to this thread.Message
> ID: ***@***.***>
>


Resolving patch:
diff --git a/speechbrain/dataio/dataloader.py b/speechbrain/dataio/dataloader.py
--- a/speechbrain/dataio/dataloader.py
+++ b/speechbrain/dataio/dataloader.py
@@ -52,8 +52,15 @@
 # Optional support for webdataset
 try:
     import webdataset as wds
+    from importlib_metadata import version
 
     WDS_AVAILABLE = True
+
+    # Use appropriate class based on webdataset version
+    if version("webdataset")[0:4] == "0.1.":
+        WDS_CLASS = wds.dataset.Composable
+    else:
+        WDS_CLASS = wds.DataPipeline
 except ImportError:
     WDS_AVAILABLE = False
 
@@ -120,7 +127,7 @@ def make_dataloader(dataset, looped_nominal_epoch=None, **loader_kwargs):
     # which requires batch_size = None in the DataLoader
     if (
         WDS_AVAILABLE
-        and isinstance(dataset, wds.dataset.Composable)
+        and isinstance(dataset, WDS_CLASS)
         and "batch_size" not in loader_kwargs
     ):
         loader_kwargs["batch_size"] = None


Test patch:
Repository: speechbrain/speechbrain

Issue:
Torch 1.12 not compatible?
working to install speechbrain 0.5.12, and getting the error that "speechbrain 0.5.12 requires torch<=1.11,>=1.7, but you have torch 1.12.0 which is incompatible." read elsewhere that it should work with >=1.7. 


Discussion:
Hi,

Torch 1.12 just appeared, we can't guarantee that everything will work but from the patch note, we don't see any reason why it would break SB. Please try and let us know :-D 
Hi,
I'm trying to install on notebook
`!pip install -q speechbrain`
This is the result

|████████████████████████████████| 496 kB 15.5 MB/s 
     |████████████████████████████████| 1.2 MB 97.9 MB/s 
     |████████████████████████████████| 750.6 MB 9.7 kB/s 
     |████████████████████████████████| 109 kB 97.2 MB/s 
     |████████████████████████████████| 546 kB 91.6 MB/s 
     |████████████████████████████████| 3.7 MB 84.9 MB/s 
     |████████████████████████████████| 2.9 MB 82.3 MB/s 
  Building wheel for hyperpyyaml (setup.py) ... done
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
torchvision 0.13.0+cu113 requires torch==1.12.0, but you have torch 1.11.0 which is incompatible.
torchtext 0.13.0 requires torch==1.12.0, but you have torch 1.11.0 which is incompatible.



Meanwhile I found the following workaround on Colab:

**BEFORE installing speechbrain:**
`!pip uninstall torch`
`!pip uninstall torchvision`
`!pip uninstall torchtext`
`!pip uninstall torchaudio`

This should avoid all incopatibility issues.
Then I do not get the exact order to reinstall because while installing **torch**, it requires **torchvision**, and viceversa but at the it works for me:
`!pip install "torch==1.11.0"`
`!pip install "torchvision == 0.12.0"`
`!pip install "torchaudio == 0.11.0"`
`!pip install "torchtext == 0.12.0"`

And in the end install speechbrain
`!pip install -q speechbrain`

I try some use cases with **pyannote.audio (speaker diarization)** and it works





@pplantinga @lumaku is this linked to pypi? 
@TParcollet  Actually, I think it is the requirement from `setup.py`:
https://github.com/speechbrain/speechbrain/blob/bc3c7287e521a4b0927785f3fcb809d2ba90318a/setup.py#L40
It might be worth doing a small release (5.13) to address this.

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -37,7 +37,7 @@
         "packaging",
         "scipy",
         "sentencepiece",
-        "torch>=1.7,<=1.11",
+        "torch>=1.9",
         "torchaudio",
         "tqdm",
         "huggingface_hub",


Test patch:
Repository: speechbrain/speechbrain

Issue:
`make_dataloader` always sets shuffle=False when DistributedSampler is employed.
In speechbrain.core.Brain class, when DistributedSampler is employed, it sets shuffle to False. Is there any reason to do so?
![image](https://user-images.githubusercontent.com/35951198/177245522-fe1ebcab-351b-476f-b235-4e36af12efe5.png)



Discussion:
Hi, yes there is a reason for that @Gastron and @popcornell might remember why, actually even @RuABraun as well might know. 
@RuABraun had a problem with this. 
It looks like a bug and I think it is my fault frankly. 

Was kind of hoping someone else would take care of this, I will open a PR to fix this by tomorrow.

Resolving patch:
diff --git a/speechbrain/core.py b/speechbrain/core.py
--- a/speechbrain/core.py
+++ b/speechbrain/core.py
@@ -755,7 +755,7 @@ def _train_loader_specifics(self, dataset, loader_kwargs):
             elif loader_kwargs.get("batch_sampler") is None:
                 # no sampler and batch-sampler
                 self.train_sampler = DistributedSampler(
-                    dataset, rank=self.rank, shuffle=False, drop_last=drop_last
+                    dataset, rank=self.rank, shuffle=True, drop_last=drop_last
                 )
 
                 # with DistributedSamplerWrapper, one must disable shuffling for dataloader
@@ -765,7 +765,7 @@ def _train_loader_specifics(self, dataset, loader_kwargs):
                 self.train_sampler = DistributedSamplerWrapper(
                     loader_kwargs.get("batch_sampler", None),
                     rank=self.rank,
-                    shuffle=False,
+                    shuffle=True,
                 )
                 loader_kwargs["batch_sampler"] = self.train_sampler
         elif self.distributed_launch and isinstance(dataset, IterableDataset):
diff --git a/speechbrain/dataio/sampler.py b/speechbrain/dataio/sampler.py
--- a/speechbrain/dataio/sampler.py
+++ b/speechbrain/dataio/sampler.py
@@ -707,7 +707,8 @@ def __len__(self):
 # Heavily inspired by Catalyst, which is under Apache 2.0 licence.
 # https://github.com/catalyst-team/catalyst/blob/51428d7756e62b9b8ee5379f38e9fd576eeb36e5/catalyst/data/sampler.py#L522
 class DistributedSamplerWrapper(DistributedSampler):
-    """This wrapper allows using any sampler with Distributed Data Parallel (DDP) correctly.
+    """This wrapper allows using any sampler (for example batch) with Distributed Data Parallel (DDP)
+    correctly.
 
     Passing blindly the sampler to each DDP process will cause to have access
     within each process to all the data in the dataset instead of only a subset


Test patch:
Repository: speechbrain/speechbrain

Issue:
Code Style Fixing
https://github.com/speechbrain/speechbrain/blob/develop/speechbrain/utils/metric_stats.py#:~:text=(self.-,labels,-%3D%3D%201).
<img width="500" alt="image" src="https://user-images.githubusercontent.com/43900898/174516502-6876ad49-f48b-480c-82de-b9b2964c317a.png">
It's better to use `self.labels == self.positive_label` than hardcode to `self.labels == 1`.


Discussion:
Hello, 

Could you please open a PR with your modification?

Many thanks :) 

Resolving patch:
diff --git a/speechbrain/utils/metric_stats.py b/speechbrain/utils/metric_stats.py
--- a/speechbrain/utils/metric_stats.py
+++ b/speechbrain/utils/metric_stats.py
@@ -389,10 +389,10 @@ def summarize(
 
         if threshold is None:
             positive_scores = self.scores[
-                (self.labels == 1).nonzero(as_tuple=True)
+                (self.labels == self.positive_label).nonzero(as_tuple=True)
             ]
             negative_scores = self.scores[
-                (self.labels == 0).nonzero(as_tuple=True)
+                (self.labels != self.positive_label).nonzero(as_tuple=True)
             ]
             if max_samples is not None:
                 if len(positive_scores) > max_samples:


Test patch: