Package: kaldiio
KaldiioA pure python module for reading and writing kaldi ark filesIntroductionWhat is this? What arearkandscp?FeaturesSimilar projectsInstallUsageReadHelperWriteHelperMore low level APIIntroductionWhat arearkandscp?kaldiiois an IO utility  implemented in pure Python language for several file formats used inkaldi, which are named asarkandscp.arkandscpare used in  in order to archive some objects defined in Kaldi, typically it is Matrix object of Kaldi.In this section, we describe the basic concept ofarkandscp. More detail about the File-IO inKaldi-asr:http://kaldi-asr.org/doc/io.htmlBasic of File IO in kaldi: Ark and copy-featsarkis an archive format to save anyKaldi objects. This library mainly supportKaldiMatrix/KaldiVector.
This ia an example of ark file of KaldiMatrix:ark fileIf you haveKaldi, you can convert it to text format as following# copy-feats <read-specifier> <write-specifier>copy-featsark:test.arkark,t:text.arkcopy-featsis designed to have high affinity with unix command line:arkcan be flushed to and from unix pipe.cat test.ark | copy-feats ark:- ark,t:- | less # Show the contents in the ark-indicates standard input stream or output stream.Unix command can be used asread-specifierandwspecifiercopy-feats ark:'gunzip -c some.ark.gz |' ark:some.arkScp filescpis a text file such as,uttid1 /some/where/feats.ark:123
uttid2 /some/where/feats.ark:156
uttid3 /some/where/feats.ark:245The first column,uttid1, indicates the utterance id and the second,/some/where/feats.ark:123, is the file path of matrix/vector of kaldi formats.  The number after colon is a starting addressof the object of the file.scplooks very simple format, but has several powerful features.Mutual conversion betweenarkandscpcopy-feats scp:foo.scp ark:foo.ark  # scp -> ark
 copy-feats ark:foo.ark ark,scp:bar.ark,bar.scp  # ark -> ark,scpUnix command can be used insead of direct file pathFor example, the following scp file can be also used.uttid1 cat /some/where/feats1.mat |
 uttid2 cat /some/where/feats2.mat |
 uttid3 cat /some/where/feats3.mat |wav.scpwav.scpis ascpto describe wave file paths.uttid1 /some/path/a.wav
uttid2 /some/path/b.wav
uttid3 /some/path/c.wavwav.scpis also can be embeded unix command as normal scp file. This is often used for converting file format in kaldi recipes.uttid1 sph2pipe -f wav /some/path/a.wv1 |
uttid2 sph2pipe -f wav /some/path/b.wv1 |
uttid3 sph2pipe -f wav /some/path/c.wv1 |FeaturesKaldiio supports:Read/Write for archive formats: ark, scpBinary/Text - Float/Double Matrix: DM, FMBinary/Text - Float/Double Vector: DV, FVCompressed Matrix for loading: CM, CM2, CM3Compressed Matrix for writing: All compressoin_method are supported: 1,2,3,4,5,6,7Binary/Text for Int-vector, typically used foralifiles.Read/Write via a pipe: e.g. "ark: cat feats.ark |"Read wav.scp / wav.ark(New!) Some extended ark formatnot supportedin Kaldi originally.The ark file for numpy, pickle, wav, flac files.The followings arenot supportedWrite in existing scp fileNNet2/NNet3 egsLattice fileSimilar projectsPython-C++ bindinghttps://github.com/pykaldi/pykaldiLooks great. I recommend pykaldi if you aren't particular about pure python.https://github.com/janchorowski/kaldi-python/Maybe not enough maintained now.https://github.com/t13m/kaldi-readers-for-tensorflowArk reader for tensorflowhttps://github.com/csukuangfj/kaldi_native_ioImplemented in C++Have interface for PythonSupport all types ofrspecifierandwspecifierHave a uniform interface for writing, sequential reading, and random access readingpip install kaldi_native_ioPure Pythonhttps://github.com/vesis84/kaldi-io-for-pythonkaldiiois based on this module, butkaldiiosupports more features than it.https://github.com/funcwj/kaldi-python-ioPython>=3.6.nnet3-egsis also supported.InstallpipinstallkaldiioUsagekaldiiodoesn't distinguish the API for each kaldi-objects, i.e.Kaldi-Matrix,Kaldi-Vector, not depending on whether it is binary or text, or compressed or not,
can be handled by the same API.ReadHelperReadHelpersupports sequential accessing forscporark. If you need to access randomly, then usekaldiio.load_scp.Read matrix-scpfromkaldiioimportReadHelperwithReadHelper('scp:file.scp')asreader:forkey,numpy_arrayinreader:...Read gziped arkfromkaldiioimportReadHelperwithReadHelper('ark: gunzip -c file.ark.gz |')asreader:forkey,numpy_arrayinreader:...# Ali filewithReadHelper('ark: gunzip -c exp/tri3_ali/ali.*.gz |')asreader:forkey,numpy_arrayinreader:...Read wav.scpfromkaldiioimportReadHelperwithReadHelper('scp:wav.scp')asreader:forkey,(rate,numpy_array)inreader:...- v2.11.0: Removedwavoption. You can loadwav.scpwithout any addtional argument.Read wav.scp with segmentsfromkaldiioimportReadHelperwithReadHelper('scp:wav.scp',segments='segments')asreaderforkey,(rate,numpy_array)inreader:...Read from stdinfromkaldiioimportReadHelperwithReadHelper('ark:-')asreader:forkey,numpy_arrayinreader:...WriteHelperWrite matrices and vectors in a ark with scpimportnumpyfromkaldiioimportWriteHelperwithWriteHelper('ark,scp:file.ark,file.scp')aswriter:foriinrange(10):writer(str(i),numpy.random.randn(10,10))# The following is equivalent# writer[str(i)] = numpy.random.randn(10, 10)Write in compressed matriximportnumpyfromkaldiioimportWriteHelperwithWriteHelper('ark:file.ark',compression_method=2)aswriter:foriinrange(10):writer(str(i),numpy.random.randn(10,10))Write matrices in textimportnumpyfromkaldiioimportWriteHelperwithWriteHelper('ark,t:file.ark')aswriter:foriinrange(10):writer(str(i),numpy.random.randn(10,10))Write in gziped arkimportnumpyfromkaldiioimportWriteHelperwithWriteHelper('ark:| gzip -c > file.ark.gz')aswriter:foriinrange(10):writer(str(i),numpy.random.randn(10,10))Write matrice to stdoutimportnumpyfromkaldiioimportWriteHelperwithWriteHelper('ark:-')aswriter:foriinrange(10):writer(str(i),numpy.random.randn(10,10))(New!) Extended ark format using numpy, pickle, soundfileimportnumpyfromkaldiioimportWriteHelper# NPY ARKwithWriteHelper('ark:-',write_function="numpy")aswriter:writer("foo",numpy.random.randn(10,10))# PICKLE ARKwithWriteHelper('ark:-',write_function="pickle")aswriter:writer("foo",numpy.random.randn(10,10))# FLAC ARKwithWriteHelper('ark:-',write_function="soundfile_flac")aswriter:writer("foo",numpy.random.randn(1000))Note thatsoundfileis an optional module and you need to install it to use this feature.pipinstallsoundfileMore low level APIWriteHelperandReadHelperare high level wrapper of the following API to support kaldi style arguments.load_arkimportkaldiiod=kaldiio.load_ark('a.ark')# d is a generator objectforkey,numpy_arrayind:...# === load_ark can accepts file descriptor, toowithopen('a.ark')asfd:forkey,numpy_arrayinkaldiio.load_ark(fd):...# === Use with open_like_kaldifromkaldiioimportopen_like_kaldiwithopen_like_kaldi('gunzip -c file.ark.gz |','r')asf:forkey,numpy_arrayinkaldiio.load_ark(fd):...load_arkcan load both matrices of ark and vectors of ark and also, it can be both text and binary.load_scpload_scpcreates "lazy dict", i.e.
The data are loaded in memory when accessing the element.importkaldiiod=kaldiio.load_scp('a.scp')forkeyind:numpy_array=d[key]withopen('a.scp')asfd:kaldiio.load_scp(fd)d=kaldiio.load_scp('data/train/wav.scp',segments='data/train/segments')forkeyind:rate,numpy_array=d[key]The object created byload_scpis a dict-like object, thus it has methods ofdict.importkaldiiod=kaldiio.load_scp('a.scp')d.keys()d.items()d.values()'uttid'indd.get('uttid')load_scp_sequential (from v2.13.0)load_scp_sequentialcreates "generator" as same asload_ark.
If you don't need random-accessing for each elements
and use it just to iterate for whole data,
then this method possibly performs faster thanload_scp.importkaldiiod=kaldiio.load_scp_sequential('a.scp')forkey,numpy_arrayind:...load_wav_scpd=kaldiio.load_scp('wav.scp')forkeyind:rate,numpy_array=d[key]# Supporting "segments"d=kaldiio.load_scp('data/train/wav.scp',segments='data/train/segments')forkeyind:rate,numpy_array=d[key]v2.11.0:load_wav_scpis deprecated now. Useload_scp.load_matnumpy_array=kaldiio.load_mat('a.mat')numpy_array=kaldiio.load_mat('a.ark:1134')# Seek and load# If the file is wav, gets Tuple[int, numpy.ndarray]rate,numpy_array=kaldiio.load_mat('a.wav')load_matcan load kaldi-matrix, kaldi-vector, and wavesave_ark# === Create ark file from numpykaldiio.save_ark('b.ark',{'key':numpy_array,'key2':numpy_array2})# Create ark with scp _file, tookaldiio.save_ark('b.ark',{'key':numpy_array,'key2':numpy_array2},scp='b.scp')# === Writes arrays to sys.stdoutimportsyskaldiio.save_ark(sys.stdout,{'key':numpy_array})# === Writes arrays for each keys# generate a.arkkaldiio.save_ark('a.ark',{'key':numpy_array,'key2':numpy_array2})# After here, a.ark is opened with 'a' (append) mode.kaldiio.save_ark('a.ark',{'key3':numpy_array3},append=True)# === Use with open_like_kaldifromkaldiioimportopen_like_kaldiwithopen_like_kaldi('| gzip a.ark.gz','w')asf:kaldiio.save_ark(f,{'key':numpy_array})kaldiio.save_ark(f,{'key2':numpy_array2})save_mat# array.ndim must be 1 or 2kaldiio.save_mat('a.mat',numpy_array)save_matcan save both kaldi-matrix and kaldi-vectoropen_like_kaldikaldiio.open_like_kaldiis a useful tool if you are familiar with Kaldi. This function can performs as following,fromkaldiioimportopen_like_kaldiwithopen_like_kaldi('echo -n hello |','r')asf:assertf.read()=='hello'withopen_like_kaldi('| cat > out.txt','w')asf:f.write('hello')withopen('out.txt','r')asf:assertf.read()=='hello'importsyswithopen_like_kaldi('-','r')asf:assertfissys.stdinwithopen_like_kaldi('-','w')asf:assertfissys.stdoutFor example, if there are gziped alignment file, then you can load it as:fromkaldiioimportopen_like_kaldi,load_arkwithopen_like_kaldi('gunzip -c exp/tri3_ali/ali.*.gz |','rb')asf:# Alignment format equals ark of IntVectorg=load_ark(f)fork,numpy_arraying:...parse_specifierfromkaldiioimportparse_specifier,open_like_kaldi,load_arkrspecifier='ark:gunzip -c file.ark.gz |'spec_dict=parse_specifier(rspecifier)# spec_dict = {'ark': 'gunzip -c file.ark.gz |'}withopen_like_kaldi(spec_dict['ark'],'rb')asfark:forkey,numpy_arrayinload_ark(fark):...