internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: NVIDIA/NeMo

Issue:
vad_infer.py script fails with `IndexError` if option `--dont_auto_split` is not set
**Describe the bug**

`NeMo/examples/asr/vad_infer.py` script fails with `IndexError` if option `--dont_auto_split` is not set and at least 1 `.wav` file is long enough so that `nemo.collections.asr.parts.utils.vad_utils.prepare_manifest()` function split the `.wav` file.

**Steps/Code to reproduce bug**

```
mkdir -p ~/debug_data
wget http://i13pc106.ira.uka.de/~jniehues/IWSLT-SLT/data/eval/en-de/IWSLT-SLT.tst2019.en-de.tgz -O ~/debug_data/IWSLT-SLT.tst2019.en-de.tgz
tar xzf ~/debug_data/IWSLT-SLT.tst2019.en-de.tgz -C ~/debug_data/
cd ~/NeMo/examples/asr/
wget https://raw.githubusercontent.com/NVIDIA/NeMo/feat/asr/iwslt_audio_to_nemo_format/scripts/dataset_processing/prepare_iwslt_audio_data.py
python prepare_iwslt_audio_data.py -a ~/debug_data/IWSLT.tst2019/wavs/ -t ~/debug_data/IWSLT.tst2019/IWSLT.TED.tst2019.en-de.en.xml -o ~/debug_data/IWSLT.tst2019/manifest.json
python vad_infer.py --dataset ~/debug_data/IWSLT.tst2019/manifest.json --out_dir ~/debug_data/IWSLT.tst2019/vad --vad_model vad_marblenet
```


**Expected behavior**

No errors

**Environment overview (please complete the following information)**

 - Environment location: Bare-metal
 - Method of NeMo install: pip install nemo_toolkit[all]

**Environment details**

If NVIDIA docker image is used you don't need to specify these.
Otherwise, please provide:
- OS version: Ubuntu 20.04.2 LTS
- PyTorch version: 1.8.1
- Python version: 3.8.10




Discussion:

Resolving patch:
diff --git a/examples/asr/vad_infer.py b/examples/asr/vad_infer.py
--- a/examples/asr/vad_infer.py
+++ b/examples/asr/vad_infer.py
@@ -134,7 +134,7 @@ def main():
     all_len = 0
 
     data = []
-    for line in open(args.dataset, 'r'):
+    for line in open(manifest_vad_input, 'r'):
         file = json.loads(line)['audio_filepath'].split("/")[-1]
         data.append(file.split(".wav")[0])
     logging.info(f"Inference on {len(data)} audio files/json lines!")


Test patch:
Repository: NVIDIA/NeMo

Issue:
Spec augmentation time masking
**Describe the bug**

I was wondering if this line of [code](https://github.com/NVIDIA/NeMo/blob/main/nemo/collections/asr/parts/numba/spec_augment/spec_aug_numba.py#L286) for time masking of spec augmentation is correct.
Unless intended, it looks more reasonable with `time_starts = torch.cat(time_starts, 0)` rather than `time_starts = torch.cat(time_lengths, 0)` as this line of code makes `time_starts` and `time_lengths` equivalent to each other.

**Steps/Code to reproduce bug**

Any scripts useing spec augmentation, e.g., `python speech_to_text_bpe.py --config-path conf/conformer --config-name conformer_ctc_bpe`

**Expected behavior**

It currently generates time masks only at the beginning of the sequence.
![Screenshot from 2021-12-07 22-47-11](https://user-images.githubusercontent.com/50283958/145161693-8069705e-3599-494c-89b4-70eec6140545.png)

**Environment overview (please complete the following information)**

`pip install -e .`

**Environment details**

N/A

**Additional context**

N/A



Discussion:
Oh boy. Good catch, that's a bug. I create the time_starts list above but assign both as time_lengths concat. I'll send a but fix tomorrow. 

Resolving patch:
diff --git a/nemo/collections/asr/modules/audio_preprocessing.py b/nemo/collections/asr/modules/audio_preprocessing.py
--- a/nemo/collections/asr/modules/audio_preprocessing.py
+++ b/nemo/collections/asr/modules/audio_preprocessing.py
@@ -491,6 +491,7 @@ def __init__(
 
         # Check if numba is supported, and use a Numba kernel if it is
         if use_numba_spec_augment and numba_utils.numba_cuda_is_supported(__NUMBA_MINIMUM_VERSION__):
+            logging.info('Numba CUDA SpecAugment kernel is being used')
             self.spec_augment_numba = SpecAugmentNumba(
                 freq_masks=freq_masks,
                 time_masks=time_masks,
diff --git a/nemo/collections/asr/parts/numba/spec_augment/spec_aug_numba.py b/nemo/collections/asr/parts/numba/spec_augment/spec_aug_numba.py
--- a/nemo/collections/asr/parts/numba/spec_augment/spec_aug_numba.py
+++ b/nemo/collections/asr/parts/numba/spec_augment/spec_aug_numba.py
@@ -283,7 +283,7 @@ def forward(self, input_spec, length):
                     torch.randint(0, time_width[idx] + 1, size=[1, self.time_masks], device=input_spec.device)
                 )
 
-            time_starts = torch.cat(time_lengths, 0)
+            time_starts = torch.cat(time_starts, 0)
             time_lengths = torch.cat(time_lengths, 0)
 
         else:


Test patch:
Repository: NVIDIA/NeMo

Issue:
RegexTokenizer: Default token arguments are not mapped to vocabulary index
The ID's of the default special tokens in RegexTokenizer  are set here:

https://github.com/NVIDIA/NeMo/blob/72d78d832c6332ccc7f44811fb777eb7044df1d8/nemo/collections/common/tokenizers/regex_tokenizer.py#L75-L82

However, the constructor, allows custom special tokens:

https://github.com/NVIDIA/NeMo/blob/72d78d832c6332ccc7f44811fb777eb7044df1d8/nemo/collections/common/tokenizers/regex_tokenizer.py#L42-L51


So this could cause issues (specifically extra unknowns) when text is converted to tokens, then to ids, then back.


Discussion:

Resolving patch:
diff --git a/nemo/collections/common/tokenizers/regex_tokenizer.py b/nemo/collections/common/tokenizers/regex_tokenizer.py
--- a/nemo/collections/common/tokenizers/regex_tokenizer.py
+++ b/nemo/collections/common/tokenizers/regex_tokenizer.py
@@ -74,12 +74,12 @@ def __init__(
 
         # initialize with default vocab
         self.vocab = {
-            DEFAULT_PAD_TOKEN: 0,  # pad_token
-            DEFAULT_UNK_TOKEN: 1,  # unk_token
-            DEFAULT_BOS_TOKEN: 2,  # begin_token
-            DEFAULT_EOS_TOKEN: 3,  # end_token
-            DEFAULT_MASK_TOKEN: 4,  # mask_token
-            DEFAULT_SEP_TOKEN: 5,  # sep_token
+            self.pad_token: 0,  # pad_token
+            self.unk_token: 1,  # unk_token
+            self.bos_token: 2,  # begin_token
+            self.eos_token: 3,  # end_token
+            self.mask_token: 4,  # mask_token
+            self.sep_token: 5,  # sep_token
         }
         self._update_cache()
 


Test patch:
Repository: NVIDIA/NeMo

Issue:
Use of undeclared `logger`.
`logger` is used here but not defined

https://github.com/NVIDIA/NeMo/blob/6abfbbfda654f44313068b950edb0f70b01449b1/nemo/collections/nlp/data/language_modeling/text_memmap_dataset.py#L157

I believe this should be switched to `logging`, which is used throughout the rest of the module.


Discussion:

Resolving patch:
diff --git a/nemo/collections/nlp/data/language_modeling/text_memmap_dataset.py b/nemo/collections/nlp/data/language_modeling/text_memmap_dataset.py
--- a/nemo/collections/nlp/data/language_modeling/text_memmap_dataset.py
+++ b/nemo/collections/nlp/data/language_modeling/text_memmap_dataset.py
@@ -154,7 +154,9 @@ def load_file(self, fn):
             if 'newline_int' in idx_dict:
                 newline_int = idx_dict['newline_int']
                 if self._newline_int != newline_int:
-                    logger.warning(f"Mismatch in newline_int, expected = {self._newline_int} but loaded {newline_int}")
+                    logging.warning(
+                        f"Mismatch in newline_int, expected = {self._newline_int} but loaded {newline_int}"
+                    )
 
             # test for version mismatch (useful to force recreation of index files)
             idx_version = idx_dict.get('version', '0.0')


Test patch:
Repository: NVIDIA/NeMo

Issue:
Fix links to speaker identification notebook
# What does this PR do ?

Fixes #5258

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)



Discussion:

Resolving patch:
diff --git a/examples/speaker_tasks/recognition/speaker_reco.py b/examples/speaker_tasks/recognition/speaker_reco.py
--- a/examples/speaker_tasks/recognition/speaker_reco.py
+++ b/examples/speaker_tasks/recognition/speaker_reco.py
@@ -37,7 +37,7 @@
     exp_manager.name=$EXP_NAME +exp_manager.use_datetime_version=False \
     exp_manager.exp_dir='./speaker_exps'
 
-See https://github.com/NVIDIA/NeMo/blob/main/tutorials/speaker_tasks/Speaker_Recognition_Verification.ipynb for notebook tutorial
+See https://github.com/NVIDIA/NeMo/blob/main/tutorials/speaker_tasks/Speaker_Identification_Verification.ipynb for notebook tutorial
 
 Optional: Use tarred dataset to speech up data loading.
    Prepare ONE manifest that contains all training data you would like to include. Validation should use non-tarred dataset.


Test patch:
Repository: NVIDIA/NeMo

Issue:
Enable gpu device selection in ClusteringDiarizer()
Can you add the ability to select the GPU device when using [ClusteringDiarizer](https://github.com/NVIDIA/NeMo/blob/main/nemo/collections/asr/models/clustering_diarizer.py)? Currently, it always gets mapped to GPU 0 but it seems like that should be configurable unless I'm missing something.


Discussion:

Resolving patch:
diff --git a/nemo/collections/asr/models/clustering_diarizer.py b/nemo/collections/asr/models/clustering_diarizer.py
--- a/nemo/collections/asr/models/clustering_diarizer.py
+++ b/nemo/collections/asr/models/clustering_diarizer.py
@@ -106,7 +106,8 @@ def __init__(self, cfg: DictConfig, speaker_model=None):
 
         # Clustering params
         self._cluster_params = self._diarizer_params.clustering.parameters
-        self._device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
+        default_device = "cuda" if torch.cuda.is_available() else "cpu"
+        self._device = torch.device(cfg.device if cfg.device else default_device)
 
     @classmethod
     def list_available_models(cls):


Test patch:
Repository: coreruleset/coreruleset

Issue:
Move data files from util/regexp-assemble directory to the top level
### Description

Data files used to generate regular expressions have been somehow in a difficult-to-find place, dependent on the tool.

Now with the new crs-toolchain, this is not needed anymore.

So let's move the data files to the top level directory.

### Requirements

- move all data files to the top level dir
- review dependencies and check that all references are updated


Discussion:

Resolving patch:
diff --git a/util/regexp-assemble/lib/context.py b/util/regexp-assemble/lib/context.py
--- a/util/regexp-assemble/lib/context.py
+++ b/util/regexp-assemble/lib/context.py
@@ -10,8 +10,8 @@ def __init__(self, root_directory: Path, namespace: argparse.Namespace=None):
         self.rules_directory = self.root_directory / "rules"
         self.util_directory = self.root_directory / "util"
         self.regexp_assemble_directory = self.util_directory / "regexp-assemble"
-        self.data_files_directory = self.regexp_assemble_directory / "data"
-        self.include_files_directory = self.regexp_assemble_directory / "data" / "include"
+        self.data_files_directory = self.root_directory / "data"
+        self.include_files_directory = self.root_directory / "data" / "include"
         self.regexp_assemble_pl_path = self.regexp_assemble_directory / "lib" / "regexp-assemble.pl"
         self.single_rule_id = namespace.rule_id if namespace else None
         self.single_chain_offset = None


Test patch:
Repository: coreruleset/coreruleset

Issue:
.changes-pending.md lacks space before asterisk
Our CHANGES.md has a leading space before the bullet / asterisk. The .changes-pending.md does not.


Discussion:

Resolving patch:
diff --git a/.github/create-changelog-prs.py b/.github/create-changelog-prs.py
--- a/.github/create-changelog-prs.py
+++ b/.github/create-changelog-prs.py
@@ -106,7 +106,7 @@ def generate_content(prs: list, merged_by: str) -> (str, str):
 		pr_number = pr["number"]
 		pr_title = pr["title"]
 		pr_author = get_pr_author_name(pr["author"]["login"])
-		new_line = f"* {pr_title} ({pr_author}) [#{pr_number}]\n"
+		new_line = f" * {pr_title} ({pr_author}) [#{pr_number}]\n"
 		pr_body += new_line
 		pr_links += f"- #{pr_number}\n"
 


Test patch:
Repository: python-poetry/poetry

Issue:
poetry script: unable to find entry point module, if package resides in src
`poetry script` is unable to find the entry point module in `src`

```
[NameError]
name 'Module' is not defined
script <script-name> [<args>]...
```


Discussion:

Resolving patch:
diff --git a/poetry/console/commands/script.py b/poetry/console/commands/script.py
--- a/poetry/console/commands/script.py
+++ b/poetry/console/commands/script.py
@@ -1,5 +1,6 @@
 import sys
 
+from ...masonry.utils.module import Module
 from .venv_command import VenvCommand
 
 
@@ -25,19 +26,32 @@ def handle(self):
 
         module, callable_ = scripts[script].split(':')
 
+        src_in_sys_path = 'sys.path.append(\'src\'); '\
+            if self._module.is_in_src() else ''
+
         cmd = ['python', '-c']
 
         cmd += [
             '"import sys; '
             'from importlib import import_module; '
-            'sys.argv = {!r}; '
+            'sys.argv = {!r}; {}'
             'import_module(\'{}\').{}()"'.format(
-               argv, module, callable_
+               argv, src_in_sys_path, module, callable_
             )
         ]
 
         self.venv.run(*cmd, shell=True, call=True)
 
+    @property
+    def _module(self):
+        poetry = self.poetry
+        package = poetry.package
+        path = poetry.file.parent
+        module = Module(
+            package.name, path.as_posix()
+        )
+        return module
+
     def merge_application_definition(self, merge_args=True):
         if self._application is None \
                 or (self._application_definition_merged


Test patch:
Repository: python-poetry/poetry

Issue:
"poetry remove" case-sensitivity (qol)
```bash
$ poetry add pyyaml
Using version ^3.12 for PyYAML

Updating dependencies
Resolving dependencies...


Package operations: 1 install, 0 updates, 0 removals

Writing lock file

  - Installing pyyaml (3.12)
$ poetry remove pyyaml

[KeyError]

remove [-D|--dev] [--dry-run] [--] <packages> (<packages>)...

$ poetry remove PyYAML
Updating dependencies
Resolving dependencies...


Package operations: 0 installs, 0 updates, 1 removal

Writing lock file

  - Removing pyyaml (3.12)
```

Not urgent but sending a hint such as "Dependencies are case sensitive." would have been really helpful.


Discussion:
Why are removals case sensitive and installs case insensitive? Seems to be a misfeature.

Resolving patch:
diff --git a/poetry/console/commands/remove.py b/poetry/console/commands/remove.py
--- a/poetry/console/commands/remove.py
+++ b/poetry/console/commands/remove.py
@@ -39,7 +39,7 @@ def handle(self):
             for key in poetry_content[section]:
                 if key.lower() == name.lower():
                     found = True
-                    requirements[name] = poetry_content[section][name]
+                    requirements[key] = poetry_content[section][key]
                     break
 
             if not found:


Test patch:
Repository: python-poetry/poetry

Issue:
Trailing newline not included in profile `export PATH` (new style installer)
- **OS version and name**: Debian Buster
- **Poetry version**: 0.12.0a1

## Issue
I installed Poetry with the new style installer :
```
curl -sSL -O https://raw.githubusercontent.com/sdispater/poetry/develop/get-poetry.py
python3 get-poetry.py --preview
```
The installation went well, but the line added to the shell startup files for updating the `PATH` environment variable doesn't end with a newline.


Discussion:

Resolving patch:
diff --git a/get-poetry.py b/get-poetry.py
--- a/get-poetry.py
+++ b/get-poetry.py
@@ -581,7 +581,7 @@ def update_path(self):
         # Updating any profile we can on UNIX systems
         export_string = self.get_export_string()
 
-        addition = "\n{}".format(export_string)
+        addition = "\n{}\n".format(export_string)
 
         updated = []
         profiles = self.get_unix_profiles()


Test patch:
Repository: python-poetry/poetry

Issue:
Poetry run: ModuleOrPackageNotFound with implicit namespace packages (PEP420)
<!-- Checked checkbox should look like this: [x] -->
- [x] I am on the [latest](https://github.com/sdispater/poetry/releases/latest) Poetry version.
- [x] I have searched the [issues](https://github.com/sdispater/poetry/issues) of this repo and believe that this is not a duplicate.
- [x] If an exception occurs when executing a command, I executed it again in debug mode (`-vvv` option).

- **OS version and name**: Arch Linux 4.18.16
- **Poetry version**: 0.12.5
- **Link of a [Gist](https://gist.github.com/) with the contents of your pyproject.toml file**: https://gist.github.com/Kazy/692963f6a41c64d38f38ac9a3f95619a

## Issue
I'm using implicit namespace packages to organize the packages at work, which works well by specifying the `packages` value in the `pyproject.toml` like that:
```toml
packages = [
    { include = "org" }
]
```
The file structure is like that:
```
├── org
│   └── subpackage
│       ├── __init__.py
│       └── command
│           └── cli.py
└── pyproject.toml
```

The issue is when you try to do `poetry run my-command`, you get:
```
[ModuleOrPackageNotFound]
No file/folder found for package org-subpackage-command
```

I already found how to fix it and will follow with a PR, but I wanted to create the issue in case my fix isn't the right one, and to make organization easier on your side as well.



Discussion:

Resolving patch:
diff --git a/poetry/console/commands/run.py b/poetry/console/commands/run.py
--- a/poetry/console/commands/run.py
+++ b/poetry/console/commands/run.py
@@ -47,7 +47,7 @@ def _module(self):
         poetry = self.poetry
         package = poetry.package
         path = poetry.file.parent
-        module = Module(package.name, path.as_posix())
+        module = Module(package.name, path.as_posix(), package.packages)
         return module
 
     def merge_application_definition(self, merge_args=True):


Test patch:
Repository: python-poetry/poetry

Issue:
Support customizable POETRY_HOME
It would be nice to define where poetry gets installed (via get-poetry.py).

By reading the docstring I had assumed it would work in $POETRY_HOME, but that was quickly disproven.

Ideally this could be defined via an environment variable (POETRY_HOME) or via a flag to get-poetry.py.


Discussion:
Aside, this is to make things like Travis and Docker easier to configure without a lot of custom code in each. Example of me continually guessing and messing this up is available here:

https://github.com/getsentry/zeus/pull/166

Hopefully by the time you've read this I have it all functional ;)
Yes, please. I need to run my docker app as non-root, while all packages are installed as root. Poetry is getting installed in `/root/.poetry` and then a non-root user can't use it.

Resolving patch:
diff --git a/get-poetry.py b/get-poetry.py
--- a/get-poetry.py
+++ b/get-poetry.py
@@ -187,7 +187,7 @@ def expanduser(path):
 
 
 HOME = expanduser("~")
-POETRY_HOME = os.path.join(HOME, ".poetry")
+POETRY_HOME = os.environ.get("POETRY_HOME") or os.path.join(HOME, ".poetry")
 POETRY_BIN = os.path.join(POETRY_HOME, "bin")
 POETRY_ENV = os.path.join(POETRY_HOME, "env")
 POETRY_LIB = os.path.join(POETRY_HOME, "lib")


Test patch:
Repository: python-poetry/poetry

Issue:
--no-root behavior is inverted on latest develop
[This](https://github.com/sdispater/poetry/commit/37ec1447b3508ee0bbdb41f8e5773ed5bfae0654#diff-427299ba040b8502b4d29846e595c2d0R59) should probably be `if self.option("no-root")`, to _not_ install the root package when `--no-root` is provided.


Discussion:
CC @seleem1337


Resolving patch:
diff --git a/poetry/console/commands/install.py b/poetry/console/commands/install.py
--- a/poetry/console/commands/install.py
+++ b/poetry/console/commands/install.py
@@ -56,7 +56,7 @@ def handle(self):
         if return_code != 0:
             return return_code
 
-        if not self.option("no-root"):
+        if self.option("no-root"):
             return 0
 
         try:


Test patch:
Repository: python-poetry/poetry

Issue:
Confusing output for install sub-command
- [x] I am on the [latest](https://github.com/sdispater/poetry/releases/latest) Poetry version.
- [x] I have searched the [issues](https://github.com/sdispater/poetry/issues) of this repo and believe that this is not a duplicate.
- [x] If an exception occurs when executing a command, I executed it again in debug mode (`-vvv` option).

- **OS version and name**: MacOS 10.15
- **Poetry version**: 1.0.0b2
- **Link of a [Gist](https://gist.github.com/) with the contents of your pyproject.toml file**: https://gist.github.com/pikeas/5c45fc9fd32353b5aee37163f95ef14d (reusing from unrelated issue https://github.com/sdispater/poetry/issues/1470)

## Issue

```
$ poetry install
Installing dependencies from lock file

Nothing to install or update

  - Installing foo (0.1.0)
$ poetry install -vvv
Using virtualenv: /Users/pikeas/Library/Caches/pypoetry/virtualenvs/foo-0mmka8-W-py3.7
Installing dependencies from lock file

Nothing to install or update

  - Skipping more-itertools (7.2.0) Already installed
  - Skipping zipp (0.6.0) Already installed
  - Skipping importlib-metadata (0.23) Already installed
  - Skipping atomicwrites (1.3.0) Already installed
  - Skipping attrs (19.3.0) Already installed
  - Skipping colorama (0.4.1) Not needed for the current environment
  - Skipping pluggy (0.13.0) Already installed
  - Skipping py (1.8.0) Already installed
  - Skipping six (1.12.0) Already installed
  - Skipping pytest (3.10.1) Already installed
  - Installing foo (0.1.0)
```

It is quite confusing to see a message saying "nothing to install" immediately followed by a message saying something is being installed!


Discussion:
I can think of why this is confusing. But the "Nothing to install" is the answer to "Installing dependencies". `foo` is not a dependency but the project package. So the wording is not wrong in my opinion.

One could make it more clearer by printing "No dependencies to install or update".

Resolving patch:
diff --git a/poetry/installation/installer.py b/poetry/installation/installer.py
--- a/poetry/installation/installer.py
+++ b/poetry/installation/installer.py
@@ -234,7 +234,7 @@ def _do_install(self, local_repo):
         # Execute operations
         actual_ops = [op for op in ops if not op.skipped]
         if not actual_ops and (self._execute_operations or self._dry_run):
-            self._io.write_line("Nothing to install or update")
+            self._io.write_line("No dependencies to install or update")
 
         if actual_ops and (self._execute_operations or self._dry_run):
             installs = []


Test patch: