Package: kaldi-native-io
IntroductionPython wrapper for Kaldi's native I/O. The internal implementation
uses C++ code fromKaldi.
A Python wrapper withpybind11is provided to read ark/scp files from Kaldi in Python.Note: This project is self-contained and does not depend onKaldi.Installationpipinstall--verbosekaldi_native_ioorgitclonehttps://github.com/csukuangfj/kaldi_native_iocdkaldi_native_io
python3setup.pyinstallorcondainstall-ckaldi_native_iokaldi_native_ioFeaturesNative support forALLtypes ofrspecifierandwspecifiersince the C++ code is borrowed from Kaldi.Support the following data types (More will be added later on request.)Note: We also support PythonbytesclassC++ Data TypeWriterSequential ReaderRandom Access ReaderPython'sbytesBlobWriterSequentialBlobReaderRandomAccessBlobReaderint32Int32WriterSequentialInt32ReaderRandomAccessInt32Readerstd::vector<int32>Int32VectorWriterSequentialInt32VectorReaderRandomAccessInt32VectorReaderstd::vector<int8>Int8VectorWriterSequentialInt8VectorReaderRandomAccessInt8VectorReaderstd::vector<std::vector<int32>>Int32VectorVectorWriterSequentialInt32VectorVectorReaderRandomAccessInt32VectorVectorReaderstd::vector<std::pair<int32, int32>>Int32PairVectorWriterSequentialInt32PairVectorReaderRandomAccessInt32PairVectorReaderfloatFloatWriterSequentialFloatReaderRandomAccessFloatReaderstd::vector<std::pair<float, float>>FloatPairVectorWriterSequentialFloatPairVectorReaderRandomAccessFloatPairVectorReaderdoubleDoubleWriterSequentialDoubleReaderRandomAccessDoubleReaderboolBoolWriterSequentialBoolReaderRandomAccessBoolReaderstd::stringTokenWriterSequentialTokenReaderRandomAccessTokenReaderstd::vector<std::string>TokenVectorWriterSequentialTokenVectorReaderRandomAccessTokenVectorReaderkaldi::Vector<float>FloatVectorWriterSequentialFloatVectorReaderRandomAccessFloatVectorReaderkaldi::Vector<double>DoubleVectorWriterSequentialDoubleVectorReaderRandomAccessDoubleVectorReaderkaldi::Matrix<float>FloatMatrixWriterSequentialFloatMatrixReaderRandomAccessFloatMatrixReaderkaldi::Matrix<double>DoubleMatrixWriterSequentialDoubleMatrixReaderRandomAccessDoubleMatrixReaderstd::pair<kaldi::Matrix<float>, HtkHeader>HtkMatrixWriterSequentialHtkMatrixReaderRandomAccessHtkMatrixReaderkaldi::CompressedMatrixCompressedMatrixWriterSequentialCompressedMatrixReaderRandomAccessCompressedMatrixReaderkaldi::PosteriorPosteriorWriterSequentialPosteriorReaderRandomAccessPosteriorReaderkaldi::GausPostGaussPostWriterSequentialGaussPostReaderRandomAccessGaussPostReaderkaldi::WaveInfo-SequentialWaveInfoReaderRandomAccessWaveInfoReaderkaldi::WaveData-SequentialWaveReaderRandomAccessWaveReaderMatrixShape-SequentialMatrixShapeReaderRandomAccessMatrixShapeReaderNote:MatrixShapedoes not exist in Kaldi. Its purpose is to get the shape information
of a matrix without reading all the data.UsageTable readers and writersWriteCreate awriterinstance with awspecifierand usewriter[key] = value.For instance, the following code useskaldi_native_io.FloatMatrixWriterto
writekaldi::Matrix<float>to awspecifier.importnumpyasnpimportkaldi_native_iobase="float_matrix"wspecifier=f"ark,scp,t:{base}.ark,{base}.scp"deftest_float_matrix_writer():withkaldi_native_io.FloatMatrixWriter(wspecifier)asko:ko.write("a",np.array([[1,2],[3,4]],dtype=np.float32))ko["b"]=np.array([[10,20,30],[40,50,60]],dtype=np.float32)ReadSequential ReadCreate a sequential reader instance with anrspecifierand usefor key, value in readerto read the file.For instance, the following code useskaldi_native_io.SequentialFloatMatrixReaderto
readkaldi::Matrix<float>from anrspecifier.importnumpyasnpimportkaldi_native_iobase="float_matrix"rspecifier=f"scp:{base}.scp"deftest_sequential_float_matrix_reader():withkaldi_native_io.SequentialFloatMatrixReader(rspecifier)aski:forkey,valueinki:ifkey=="a":assertnp.array_equal(value,np.array([[1,2],[3,4]],dtype=np.float32))elifkey=="b":assertnp.array_equal(value,np.array([[10,20,30],[40,50,60]],dtype=np.float32),)else:raiseValueError(f"Unknown key{key}with value{value}")Random Access ReadCreate a random access reader instance with anrspecifierand usereader[key]to read the file.For instance, the following code useskaldi_native_io.RandomAccessFloatMatrixReaderto
readkaldi::Matrix<float>from anrspecifier.importnumpyasnpimportkaldi_native_iobase="float_matrix"rspecifier=f"scp:{base}.scp"deftest_random_access_float_matrix_reader():withkaldi_native_io.RandomAccessFloatMatrixReader(rspecifier)aski:assert"b"inkiassert"a"inkiassertnp.array_equal(ki["a"],np.array([[1,2],[3,4]],dtype=np.float32))assertnp.array_equal(ki["b"],np.array([[10,20,30],[40,50,60]],dtype=np.float32))There are unit tests for all supported types. Please visithttps://github.com/csukuangfj/kaldi_native_io/tree/master/kaldi_native_io/python/testsfor more examples.Read and write a single matrixSeehttps://github.com/csukuangfj/kaldi_native_io/blob/master/kaldi_native_io/python/tests/test_float_matrix_writer_reader.pyhttps://github.com/csukuangfj/kaldi_native_io/blob/master/kaldi_native_io/python/tests/test_double_matrix_writer_reader.pydeftest_read_write_single_mat():arr=np.array([[0,1,2,22,33],[3,4,5,-1,-3],[6,7,8,-9,0],[9,10,11,5,100],],dtype=np.float32,)mat=kaldi_native_io.FloatMatrix(arr)mat.write(wxfilename="binary.ark",binary=True)mat.write(wxfilename="matrix.txt",binary=False)m1=kaldi_native_io.FloatMatrix.read("binary.ark")m2=kaldi_native_io.FloatMatrix.read("matrix.txt")assertnp.array_equal(mat,m1)assertnp.array_equal(mat,m2)# read range# Note: the upper bound is inclusive!m3=kaldi_native_io.FloatMatrix.read("binary.ark[0:1]")# row 0 and row 1assertnp.array_equal(mat.numpy()[0:2],m3.numpy())m4=kaldi_native_io.FloatMatrix.read("matrix.txt[:,3:4]")# column 3 and column 4assertnp.array_equal(mat.numpy()[:,3:5],m4.numpy())os.remove("binary.ark")os.remove("matrix.txt")a=np.array([[1,2],[3,4]],dtype=np.float32)b=np.array([[10,20,30],[40,50,60]],dtype=np.float32)withkaldi_native_io.FloatMatrixWriter("ark,scp:m.ark,m.scp")asko:ko.write("a",a)ko["b"]=b"""m.scp contains:a m.ark:2b m.ark:35"""m5=kaldi_native_io.FloatMatrix.read("m.ark:2")assertnp.array_equal(m5.numpy(),a)m6=kaldi_native_io.FloatMatrix.read("m.ark:35")assertnp.array_equal(m6.numpy(),b)os.remove("m.scp")os.remove("m.ark")Read and write a single vectorSeehttps://github.com/csukuangfj/kaldi_native_io/blob/master/kaldi_native_io/python/tests/test_float_vector_writer_reader.pyhttps://github.com/csukuangfj/kaldi_native_io/blob/master/kaldi_native_io/python/tests/test_double_vector_writer_reader.pydeftest_read_write_single_vector():a=np.array([1,2],dtype=np.float32)v=kaldi_native_io.FloatVector(a)v.write(wxfilename="binary.ark",binary=True)b=kaldi_native_io.FloatVector.read("binary.ark")assertnp.array_equal(a,b.numpy())a=np.array([1,2],dtype=np.float32)b=np.array([10.5],dtype=np.float32)withkaldi_native_io.FloatVectorWriter("ark,scp:v.ark,v.scp")asko:ko.write("a",a)ko["b"]=b"""v.scp contains:a v.ark:2b v.ark:22"""va=kaldi_native_io.FloatVector.read("v.ark:2")assertnp.array_equal(va.numpy(),a)vb=kaldi_native_io.FloatVector.read("v.ark:22")assertnp.array_equal(vb.numpy(),b)os.remove("v.scp")os.remove("v.ark")deftest_read_write_single_vector():a=np.array([1,2],dtype=np.float64)v=kaldi_native_io.DoubleVector(a)v.write(wxfilename="binary.ark",binary=True)b=kaldi_native_io.DoubleVector.read("binary.ark")assertnp.array_equal(a,b.numpy())os.remove("binary.ark")a=np.array([1,2],dtype=np.float64)b=np.array([10.5],dtype=np.float64)withkaldi_native_io.DoubleVectorWriter("ark,scp:v.ark,v.scp")asko:ko.write("a",a)ko["b"]=b"""v.scp contains:a v.ark:2b v.ark:30"""va=kaldi_native_io.DoubleVector.read("v.ark:2")assertnp.array_equal(va.numpy(),a)vb=kaldi_native_io.DoubleVector.read("v.ark:30")assertnp.array_equal(vb.numpy(),b)os.remove("v.scp")os.remove("v.ark")Read a single int32 vectorSeehttps://github.com/csukuangfj/kaldi_native_io/blob/master/kaldi_native_io/python/tests/test_int32_vector_writer_reader.pydeftest_read_single_item():a=[10,20]b=[100,200,300]# You can also generate a text format by adding ",t" if you like#  with kaldi_native_io.Int32VectorWriter("ark,scp,t:v.ark,v.scp") as ko:withkaldi_native_io.Int32VectorWriter("ark,scp:v.ark,v.scp")asko:ko.write("a",a)ko["b"]=b"""v.scp contains:a v.ark:2b v.ark:21"""va=kaldi_native_io.read_int32_vector("v.ark:2")assertva==avb=kaldi_native_io.read_int32_vector("v.ark:21")assertva==bRead/Write WavesSeehttps://github.com/csukuangfj/kaldi_native_io/blob/master/kaldi_native_io/python/tests/test_wave_reader.pyhttps://github.com/csukuangfj/kaldi_native_io/blob/master/kaldi_native_io/python/tests/test_wave_data.pydeftest_wave_writer():file1="/ceph-fj/fangjun/open-source-2/kaldi_native_io/build/BAC009S0002W0123.wav"ifnotPath(file1).is_file():returnfile2="/ceph-fj/fangjun/open-source-2/kaldi_native_io/build/BAC009S0002W0124.wav"ifnotPath(file2).is_file():returnprint("-----test_wave_writer------")file2=f"cat{file2}|"wave1=kaldi_native_io.read_wave(file1)wave2=kaldi_native_io.read_wave(file2)wspecifier="ark,scp:wave.ark,wave.scp"withkaldi_native_io.WaveWriter(wspecifier)asko:ko.write("a",wave1)ko["b"]=wave2"""wave.scp has the following content:a wave.ark:2b wave.ark:123728"""wave3=kaldi_native_io.read_wave("wave.ark:2")wave4=kaldi_native_io.read_wave("wave.ark:123728")assertwave1.sample_freq==wave3.sample_freqassertwave2.sample_freq==wave4.sample_freqassertnp.array_equal(wave1.data.numpy(),wave3.data.numpy())assertnp.array_equal(wave2.data.numpy(),wave4.data.numpy())Read/Write Python's bytesSeehttps://github.com/csukuangfj/kaldi_native_io/blob/master/kaldi_native_io/python/tests/test_blob_writer_reader.pybase="blob"wspecifier=f"ark,scp:{base}.ark,{base}.scp"rspecifier=f"scp:{base}.scp"deftest_blob_writer():withkaldi_native_io.BlobWriter(wspecifier)asko:ko.write("a",bytes([0x30,0x31]))ko["b"]=b"1234"deftest_sequential_blob_reader():withkaldi_native_io.SequentialBlobReader(rspecifier)aski:forkey,valueinki:ifkey=="a":assertvalue==bytes([0x30,0x31])elifkey=="b":assertvalue==b"1234"else:raiseValueError(f"Unknown key{key}with value{value}")deftest_read_single_item():a=bytes([10,20])b=b"1234"withkaldi_native_io.BlobWriter("ark,scp:b.ark,b.scp")asko:ko.write("a",a)ko["b"]=b"""b.scp contains:a b.ark:2b b.ark:20"""va=kaldi_native_io.read_blob("b.ark:2")assertva==a,(va,a)vb=kaldi_native_io.read_blob("b.ark:20")assertvb==b,(vb,b)# test range read# [start:end], both ends are inclusive# Must satisfy 0 <= start <= end < length of the data# start 0, end 2vc=kaldi_native_io.read_blob("b.ark:20[0:2]")assertvc==b"123",(vc,b"123")# start 1, end 2vd=kaldi_native_io.read_blob("b.ark:20[1:2]")assertvd==b"23",(vd,b"23")# start 2, end 2ve=kaldi_native_io.read_blob("b.ark:20[2:2]")assertve==b"3",(ve,b"3")# start 2, end -1# -1 means the end of the datavf=kaldi_native_io.read_blob("b.ark:20[2:-1]")assertvf==b"34",(vf,b"34")# [:] means all the datavg=kaldi_native_io.read_blob("b.ark:20[:]")assertvg==b"1234",(vg,b"1234")os.remove("b.scp")os.remove("b.ark")