Repository: lhotse-speech/lhotse

Issue:
Loading data
I'm trying to create a Recording using the same "command" that I used in kaldi. I have a binary that converts speex to wav and in wav.scp I do "wav_id spx2wav /path/to/file - |", here I tried new_wav = Recording.from_file("spx2wav /path/to/file - |" which gave me error (file spx2wav /path/to/file - | doesn't exist). Can someone point me to a document that shows how to this? 
I understand that I can creat the wav.scp file and then use "kaldi.load_kaldi_data_dir" and I was wondering if there is a direct way of doing it here. 


Discussion:
For now, you'd have to create it manually, you may use the following code from Kaldi import function:

https://github.com/lhotse-speech/lhotse/blob/c714b5c633b2372e21247f7ce2b3b83dabb63300/lhotse/kaldi.py#L88-L102

If you think it's useful, maybe you (or somebody else) could wrap this and add as a method on `Recording`, e.g. `Recording.from_cmd()` or `from_pipe()`?
I tried to create the recording with the following code:
`def from_cmd(
    path_or_cmd,
    recording_id,
    sampling_rate,
    duration = 0) -> "Recording":

    source = path_or_cmd[:-1] if path_or_cmd.endswith("|") else path_or_cmd
    if duration==0:
        duration = kaldi.get_duration(path_or_cmd)

    return Recording(
        id=recording_id,
        sources=[
            AudioSource(
                type="command" if path_or_cmd.endswith("|") else "file",
                channels=[0],
                source=source,
            )
        ],
        sampling_rate=sampling_rate,
        num_samples=compute_num_samples(duration, sampling_rate),
        duration=duration,
    )` 
but I get the following error :
ERROR ([5.5.201~1-7d4bd]:Write():wave-reader.cc:379) Error writing wave data to stream.
[ Stack-Trace: ]
spx2wav() [0x408630]
spx2wav() [0x409426]
spx2wav() [0x4c02a6]
spx2wav() [0x4079c8]
spx2wav() [0x40748a]
__libc_start_main
spx2wav() [0x407175]
I'm using the the same wav.scp with kaldi and don't get this error. What am I doing wrong?

Can you paste output of `print(recording)` that you got with this code?
Recording(id='test1', sources=[AudioSource(type='command', channels=[0], source=' spx2wav /mnt/disk1/test1.speex - ')], sampling_rate=16000, num_samples=211840, duration=13.239999771118164, transforms=None)
Would it work if you ran:
```python
from subprocess import run, PIPE

raw_data = run(recording.sources[0].source, shell=True, stdout=PIPE).stdout
```
Just to make sure, I ran the following code and got the same output as before. Is this what you meant?
`recording = from_cmd(test_path, test_id, 16000)
raw_data = run(recording.sources[0].source, shell=True, stdout=PIPE).stdout
print(recording)`
Yes, I meant this:

```python
recording = from_cmd(test_path, test_id, 16000) 
raw_data = run(recording.sources[0].source, shell=True, stdout=PIPE).stdout
```

If you got the same error as before, it means your Python process is not able to execute the following command successfully `' spx2wav /mnt/disk1/test1.speex - '`, but I don't know why. It would be helpful if you could show how the command looks like when the Kaldi import is successful. If we compare the two versions, we'll be able to highlight the differences and find the issue.
I'm running the following
```
call_args = ['/bin/bash', 'local/get_utt2dur.sh', "--nj", "1", "--read_entire_file", "true", "."]
subprocess.run(call_args, check=True)
```
with the following output

local/get_utt2dur.sh: segments file does not exist so getting durations from wave files
local/get_utt2dur.sh: could not get utterance lengths from sphere-file headers, using wav-to-duration
local/get_utt2dur.sh: computed ./utt2dur

and the content of utt2dur is
test1 13.24 

get_utt2dur is the following script:
 https://github.com/kaldi-asr/kaldi/blob/master/egs/wsj/s5/utils/data/get_utt2dur.sh
Just to clarify, I haven't been able to import from kaldi wav.scp when it contains "piped wav files", and to make sure that the wav.scp content has no issues, I used it with kaldi's get_utt2dur.sh and make_mfcc.sh and encountered no errors. Any suggestions?
Wait, I thought it worked if you used `kaldi import` / `kaldi.load_kaldi_data_dir`? If it does, can you post the output of `print` for the recordings obtained with both methods (i.e., using load_kaldi_data_dir, and using your code from_cmd)? 
No, using `kaldi import` gives the same error message. The only way that I checked that my wav.scp file is ok, was to give it as input to the original kaldi scripts to extract file duration (I have to do this step since the duration from header for some of my files are incorrect) and then mfcc features.
The only way that I think could work (at the moment) is to generate the resulting files (add sox -t wav - ... at the end of the pipe) and create the new wav.scp for the new files and bypass the pipes!
This may (or may not) be related to the following issue: https://github.com/csukuangfj/kaldi_native_io/issues/23
I would also check if `spx2wav` is visible in PATH env var. @is2022 try setting `export PATH="$PATH:/path/to/spx2wav"` and re-running your script.
@pzelasko yes, `spx2wav` is visible in PATH. 
The error
> ERROR ([5.5.201~1-7d4bd]:Write():wave-reader.cc:379) Error writing wave data to stream.

is from kaldi https://github.com/kaldi-asr/kaldi/blob/master/src/feat/wave-reader.cc#L379
```cpp
  if (os.fail())
    KALDI_ERR << "Error writing wave data to stream.";
```

wondering if `spx2wav /mnt/disk1/test1.speex - ` works when called from the command line.
@csukuangfj 
Yes, `spx2wav /mnt/disk1/test1.speex -`  works from the command line, also I have a wav.scp with just this command and I'm able to get the duration and exract mfcc features using the "original" kaldi.
Interesting... can you try checking out some fairly old Lhotse commit (e.g. this one is from beginning of February: 0eeca4941d9e17e9c7a0fb4af5d97186d805d15d ) which doesn't use `kaldi_native_io` and re-try?
(you'll need to `pip install kaldiio` (or `kaldi_io`)
@pzelasko The older version works without any errors! Thanks for your help :)
Cool, at least we know that the issue seems to be related to `kaldi_native_io`. @csukuangfj would you look into this more?
> Cool, at least we know that the issue seems to be related to `kaldi_native_io`. @csukuangfj would you look into this more?

Yes, I am looking into it.

Resolving patch:
diff --git a/lhotse/kaldi.py b/lhotse/kaldi.py
--- a/lhotse/kaldi.py
+++ b/lhotse/kaldi.py
@@ -35,10 +35,10 @@ def get_duration(
             )
         import kaldi_native_io
 
-        wave_info = kaldi_native_io.read_wave_info(path)
-        assert wave_info.num_channels == 1, wave_info.num_channels
+        wave = kaldi_native_io.read_wave(path)
+        assert wave.data.shape[0] == 1, f"Expect 1 channel. Given {wave.data.shape[0]}"
 
-        return wave_info.duration
+        return wave.duration
     try:
         # Try to parse the file using pysoundfile first.
         import soundfile


Test patch: