Repository: espnet/espnet

Issue:
`TypeError: cannot serialize '_io.BufferedReader' object` in single node multi-gpu distributed training
I met the following errors when `num_workers > 0` + single node multi-gpu distributed training in ESPnet2.


```bash
[cx004:0/4] 2020-07-22 18:24:02,669 (abs_task:1029) INFO: Scheduler: None
[cx004:0/4] 2020-07-22 18:24:02,670 (abs_task:1037) INFO: Saving the configuration in exp/tts_train_raw_phn_tacotron_g2p_en_no_space_num_workers2/config.yaml
[cx004:0/4] 2020-07-22 18:24:04,074 (abs_task:1397) INFO: [train] dataset:
ESPnetDataset(
  text: {"path": "dump/raw/tr_no_dev/text", "type": "text"}
  speech: {"path": "dump/raw/tr_no_dev/wav.scp", "type": "sound"}
  preprocess: <espnet2.train.preprocessor.CommonPreprocessor object at 0x2b385ff03cf8>)
[cx004:0/4] 2020-07-22 18:24:04,074 (abs_task:1398) INFO: [train] Batch sampler: NumElementsBatchSampler(N-batch=433, batch_bins=5120000, sort_in_batch=descending, sort_batch=descending)
[cx004:0/4] 2020-07-22 18:24:04,074 (abs_task:1400) INFO: [train] mini-batch sizes summary: N-batch=433, mean=29.1, min=15, max=102
[cx004:0/4] 2020-07-22 18:24:04,829 (abs_task:1397) INFO: [valid] dataset:
ESPnetDataset(
  text: {"path": "dump/raw/dev/text", "type": "text"}
  speech: {"path": "dump/raw/dev/wav.scp", "type": "sound"}
  preprocess: <espnet2.train.preprocessor.CommonPreprocessor object at 0x2b386f0590b8>)
[cx004:0/4] 2020-07-22 18:24:04,829 (abs_task:1398) INFO: [valid] Batch sampler: NumElementsBatchSampler(N-batch=9, batch_bins=5120000, sort_in_batch=descending, sort_batch=descending)
[cx004:0/4] 2020-07-22 18:24:04,829 (abs_task:1400) INFO: [valid] mini-batch sizes summary: N-batch=9, mean=27.8, min=20, max=42
[cx004:0/4] 2020-07-22 18:24:05,473 (abs_task:1397) INFO: [plot_att] dataset:
ESPnetDataset(
  text: {"path": "dump/raw/dev/text", "type": "text"}
  speech: {"path": "dump/raw/dev/wav.scp", "type": "sound"}
  preprocess: <espnet2.train.preprocessor.CommonPreprocessor object at 0x2b38cce14dd8>)
[cx004:0/4] 2020-07-22 18:24:05,474 (abs_task:1398) INFO: [plot_att] Batch sampler: UnsortedBatchSampler(N-batch=250, batch_size=1, key_file=exp/tts_stats_raw_phn_tacotron_g2p_en_no_space/valid/text_shape.phn,
[cx004:0/4] 2020-07-22 18:24:05,474 (abs_task:1400) INFO: [plot_att] mini-batch sizes summary: N-batch=3, mean=1.0, min=1, max=1
cx004:3181:3181 [0] NCCL INFO Bootstrap : Using [0]eno1:10.102.1.4<0> [1]ib0:172.18.21.4<0> [2]ib1:172.18.22.4<0> [3]ib1:ost1p4IP:172.18.24.4<0> [4]krm-virbr0:192.168.120.1<0>
cx004:3181:3181 [0] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so).
cx004:3181:3181 [0] NCCL INFO NET/IB : Using [0]mlx5_1:1/IB [1]mlx5_0:1/IB ; OOB eno1:10.102.1.4<0>
NCCL version 2.4.8+cuda10.1
cx004:3184:3184 [3] NCCL INFO Bootstrap : Using [0]eno1:10.102.1.4<0> [1]ib0:172.18.21.4<0> [2]ib1:172.18.22.4<0> [3]ib1:ost1p4IP:172.18.24.4<0> [4]krm-virbr0:192.168.120.1<0>
cx004:3183:3183 [2] NCCL INFO Bootstrap : Using [0]eno1:10.102.1.4<0> [1]ib0:172.18.21.4<0> [2]ib1:172.18.22.4<0> [3]ib1:ost1p4IP:172.18.24.4<0> [4]krm-virbr0:192.168.120.1<0>
cx004:3184:3184 [3] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so).
cx004:3183:3183 [2] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so).
cx004:3182:3182 [1] NCCL INFO Bootstrap : Using [0]eno1:10.102.1.4<0> [1]ib0:172.18.21.4<0> [2]ib1:172.18.22.4<0> [3]ib1:ost1p4IP:172.18.24.4<0> [4]krm-virbr0:192.168.120.1<0>
cx004:3182:3182 [1] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so).
cx004:3183:3183 [2] NCCL INFO NET/IB : Using [0]mlx5_1:1/IB [1]mlx5_0:1/IB ; OOB eno1:10.102.1.4<0>
cx004:3184:3184 [3] NCCL INFO NET/IB : Using [0]mlx5_1:1/IB [1]mlx5_0:1/IB ; OOB eno1:10.102.1.4<0>
cx004:3182:3182 [1] NCCL INFO NET/IB : Using [0]mlx5_1:1/IB [1]mlx5_0:1/IB ; OOB eno1:10.102.1.4<0>
cx004:3183:3227 [2] NCCL INFO Setting affinity for GPU 2 to 0e6398
cx004:3184:3228 [3] NCCL INFO Setting affinity for GPU 3 to 0e6398
cx004:3182:3230 [1] NCCL INFO Setting affinity for GPU 1 to 019c67
cx004:3181:3221 [0] NCCL INFO Setting affinity for GPU 0 to 019c67
cx004:3181:3221 [0] NCCL INFO Channel 00 :    0   1   2   3
cx004:3181:3221 [0] NCCL INFO Channel 01 :    0   1   3   2
cx004:3181:3221 [0] NCCL INFO Channel 02 :    0   2   1   3
cx004:3181:3221 [0] NCCL INFO Channel 03 :    0   2   3   1
cx004:3181:3221 [0] NCCL INFO Channel 04 :    0   3   1   2
cx004:3181:3221 [0] NCCL INFO Channel 05 :    0   3   2   1
cx004:3181:3221 [0] NCCL INFO Channel 06 :    0   1   2   3
cx004:3181:3221 [0] NCCL INFO Channel 07 :    0   1   3   2
cx004:3181:3221 [0] NCCL INFO Channel 08 :    0   2   1   3
cx004:3181:3221 [0] NCCL INFO Channel 09 :    0   2   3   1
cx004:3181:3221 [0] NCCL INFO Channel 10 :    0   3   1   2
cx004:3181:3221 [0] NCCL INFO Channel 11 :    0   3   2   1
cx004:3181:3221 [0] NCCL INFO Ring 00 : 0[0] -> 1[1] via P2P/IPC
cx004:3182:3230 [1] NCCL INFO Ring 00 : 1[1] -> 2[2] via P2P/IPC
cx004:3184:3228 [3] NCCL INFO Ring 00 : 3[3] -> 0[0] via P2P/IPC
cx004:3183:3227 [2] NCCL INFO Ring 00 : 2[2] -> 3[3] via P2P/IPC
cx004:3181:3221 [0] NCCL INFO Ring 01 : 0[0] -> 1[1] via P2P/IPC
cx004:3182:3230 [1] NCCL INFO Ring 01 : 1[1] -> 3[3] via P2P/IPC
cx004:3184:3228 [3] NCCL INFO Ring 01 : 3[3] -> 2[2] via P2P/IPC
cx004:3183:3227 [2] NCCL INFO Ring 01 : 2[2] -> 0[0] via P2P/IPC
cx004:3182:3230 [1] NCCL INFO Ring 02 : 1[1] -> 3[3] via P2P/IPC
cx004:3181:3221 [0] NCCL INFO Ring 02 : 0[0] -> 2[2] via P2P/IPC
cx004:3184:3228 [3] NCCL INFO Ring 02 : 3[3] -> 0[0] via P2P/IPC
cx004:3183:3227 [2] NCCL INFO Ring 02 : 2[2] -> 1[1] via P2P/IPC
cx004:3181:3221 [0] NCCL INFO Ring 03 : 0[0] -> 2[2] via P2P/IPC
cx004:3182:3230 [1] NCCL INFO Ring 03 : 1[1] -> 0[0] via P2P/IPC
cx004:3184:3228 [3] NCCL INFO Ring 03 : 3[3] -> 1[1] via P2P/IPC
cx004:3183:3227 [2] NCCL INFO Ring 03 : 2[2] -> 3[3] via P2P/IPC
cx004:3184:3228 [3] NCCL INFO Ring 04 : 3[3] -> 1[1] via P2P/IPC
cx004:3181:3221 [0] NCCL INFO Ring 04 : 0[0] -> 3[3] via P2P/IPC
cx004:3182:3230 [1] NCCL INFO Ring 04 : 1[1] -> 2[2] via P2P/IPC
cx004:3183:3227 [2] NCCL INFO Ring 04 : 2[2] -> 0[0] via P2P/IPC
cx004:3181:3221 [0] NCCL INFO Ring 05 : 0[0] -> 3[3] via P2P/IPC
cx004:3182:3230 [1] NCCL INFO Ring 05 : 1[1] -> 0[0] via P2P/IPC
cx004:3184:3228 [3] NCCL INFO Ring 05 : 3[3] -> 2[2] via P2P/IPC
cx004:3183:3227 [2] NCCL INFO Ring 05 : 2[2] -> 1[1] via P2P/IPC
cx004:3181:3221 [0] NCCL INFO Ring 06 : 0[0] -> 1[1] via P2P/IPC
cx004:3182:3230 [1] NCCL INFO Ring 06 : 1[1] -> 2[2] via P2P/IPC
cx004:3184:3228 [3] NCCL INFO Ring 06 : 3[3] -> 0[0] via P2P/IPC
cx004:3183:3227 [2] NCCL INFO Ring 06 : 2[2] -> 3[3] via P2P/IPC
cx004:3181:3221 [0] NCCL INFO Ring 07 : 0[0] -> 1[1] via P2P/IPC
cx004:3182:3230 [1] NCCL INFO Ring 07 : 1[1] -> 3[3] via P2P/IPC
cx004:3184:3228 [3] NCCL INFO Ring 07 : 3[3] -> 2[2] via P2P/IPC
cx004:3183:3227 [2] NCCL INFO Ring 07 : 2[2] -> 0[0] via P2P/IPC
cx004:3181:3221 [0] NCCL INFO Ring 08 : 0[0] -> 2[2] via P2P/IPC
cx004:3182:3230 [1] NCCL INFO Ring 08 : 1[1] -> 3[3] via P2P/IPC
cx004:3184:3228 [3] NCCL INFO Ring 08 : 3[3] -> 0[0] via P2P/IPC
cx004:3183:3227 [2] NCCL INFO Ring 08 : 2[2] -> 1[1] via P2P/IPC
cx004:3181:3221 [0] NCCL INFO Ring 09 : 0[0] -> 2[2] via P2P/IPC
cx004:3182:3230 [1] NCCL INFO Ring 09 : 1[1] -> 0[0] via P2P/IPC
cx004:3184:3228 [3] NCCL INFO Ring 09 : 3[3] -> 1[1] via P2P/IPC
cx004:3183:3227 [2] NCCL INFO Ring 09 : 2[2] -> 3[3] via P2P/IPC
cx004:3181:3221 [0] NCCL INFO Ring 10 : 0[0] -> 3[3] via P2P/IPC
cx004:3182:3230 [1] NCCL INFO Ring 10 : 1[1] -> 2[2] via P2P/IPC
cx004:3184:3228 [3] NCCL INFO Ring 10 : 3[3] -> 1[1] via P2P/IPC
cx004:3183:3227 [2] NCCL INFO Ring 10 : 2[2] -> 0[0] via P2P/IPC
cx004:3181:3221 [0] NCCL INFO Ring 11 : 0[0] -> 3[3] via P2P/IPC
cx004:3182:3230 [1] NCCL INFO Ring 11 : 1[1] -> 0[0] via P2P/IPC
cx004:3184:3228 [3] NCCL INFO Ring 11 : 3[3] -> 2[2] via P2P/IPC
cx004:3183:3227 [2] NCCL INFO Ring 11 : 2[2] -> 1[1] via P2P/IPC
cx004:3181:3221 [0] NCCL INFO Using 256 threads, Min Comp Cap 7, Trees disabled
cx004:3181:3221 [0] NCCL INFO comm 0x2b38e0001b30 rank 0 nranks 4 cudaDev 0 nvmlDev 0 - Init COMPLETE
cx004:3182:3230 [1] NCCL INFO comm 0x2aaf98001b30 rank 1 nranks 4 cudaDev 1 nvmlDev 1 - Init COMPLETE
cx004:3181:3181 [0] NCCL INFO Launch mode Parallel
cx004:3184:3228 [3] NCCL INFO comm 0x2b0190001b30 rank 3 nranks 4 cudaDev 3 nvmlDev 3 - Init COMPLETE
cx004:3183:3227 [2] NCCL INFO comm 0x2ba478001b30 rank 2 nranks 4 cudaDev 2 nvmlDev 2 - Init COMPLETE
[cx004:0/4] 2020-07-22 18:24:07,432 (trainer:200) INFO: 1/200epoch started
Process SpawnProcess-3:
Traceback (most recent call last):
  File "/home/test/work/espnet/tools/venv/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
  File "/home/test/work/espnet/tools/venv/lib/python3.7/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "/home/test/work/espnet/espnet2/tasks/abs_task.py", line 1222, in main_worker
    distributed_option=distributed_option,
  File "/home/test/work/espnet/espnet2/train/trainer.py", line 212, in run
    options=trainer_options,
  File "/home/test/work/espnet/espnet2/train/trainer.py", line 367, in train_one_epoch
    reporter.measure_iter_time(iterator, "iter_time"), 1
  File "/home/test/work/espnet/espnet2/train/reporter.py", line 204, in measure_iter_time
    iterator = iter(iterable)
  File "/home/test/work/espnet/tools/venv/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 279, in __iter__
    return _MultiProcessingDataLoaderIter(self)
  File "/home/test/work/espnet/tools/venv/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 719, in __init__
    w.start()
  File "/home/test/work/espnet/tools/venv/lib/python3.7/multiprocessing/process.py", line 112, in start
    self._popen = self._Popen(self)
  File "/home/test/work/espnet/tools/venv/lib/python3.7/multiprocessing/context.py", line 223, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "/home/test/work/espnet/tools/venv/lib/python3.7/multiprocessing/context.py", line 284, in _Popen
    return Popen(process_obj)
  File "/home/test/work/espnet/tools/venv/lib/python3.7/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
  File "/home/test/work/espnet/tools/venv/lib/python3.7/multiprocessing/popen_fork.py", line 20, in __init__
    self._launch(process_obj)
  File "/home/test/work/espnet/tools/venv/lib/python3.7/multiprocessing/popen_spawn_posix.py", line 47, in _launch
    reduction.dump(process_obj, fp)
  File "/home/test/work/espnet/tools/venv/lib/python3.7/multiprocessing/reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
TypeError: cannot serialize '_io.BufferedReader' object
Process SpawnProcess-4:
Traceback (most recent call last):
  File "/home/test/work/espnet/tools/venv/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
  File "/home/test/work/espnet/tools/venv/lib/python3.7/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "/home/test/work/espnet/espnet2/tasks/abs_task.py", line 1222, in main_worker
    distributed_option=distributed_option,
  File "/home/test/work/espnet/espnet2/train/trainer.py", line 212, in run
    options=trainer_options,
  File "/home/test/work/espnet/espnet2/train/trainer.py", line 367, in train_one_epoch
    reporter.measure_iter_time(iterator, "iter_time"), 1
  File "/home/test/work/espnet/espnet2/train/reporter.py", line 204, in measure_iter_time
    iterator = iter(iterable)
  File "/home/test/work/espnet/tools/venv/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 279, in __iter__
    return _MultiProcessingDataLoaderIter(self)
  File "/home/test/work/espnet/tools/venv/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 719, in __init__
    w.start()
  File "/home/test/work/espnet/tools/venv/lib/python3.7/multiprocessing/process.py", line 112, in start
    self._popen = self._Popen(self)
  File "/home/test/work/espnet/tools/venv/lib/python3.7/multiprocessing/context.py", line 223, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "/home/test/work/espnet/tools/venv/lib/python3.7/multiprocessing/context.py", line 284, in _Popen
    return Popen(process_obj)
  File "/home/test/work/espnet/tools/venv/lib/python3.7/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
  File "/home/test/work/espnet/tools/venv/lib/python3.7/multiprocessing/popen_fork.py", line 20, in __init__
    self._launch(process_obj)
  File "/home/test/work/espnet/tools/venv/lib/python3.7/multiprocessing/popen_spawn_posix.py", line 47, in _launch
    reduction.dump(process_obj, fp)
  File "/home/test/work/espnet/tools/venv/lib/python3.7/multiprocessing/reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
TypeError: cannot serialize '_io.BufferedReader' object
Process SpawnProcess-1:
Traceback (most recent call last):
  File "/home/test/work/espnet/tools/venv/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
  File "/home/test/work/espnet/tools/venv/lib/python3.7/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "/home/test/work/espnet/espnet2/tasks/abs_task.py", line 1222, in main_worker
    distributed_option=distributed_option,
  File "/home/test/work/espnet/espnet2/train/trainer.py", line 212, in run
    options=trainer_options,
  File "/home/test/work/espnet/espnet2/train/trainer.py", line 367, in train_one_epoch
    reporter.measure_iter_time(iterator, "iter_time"), 1
  File "/home/test/work/espnet/espnet2/train/reporter.py", line 204, in measure_iter_time
    iterator = iter(iterable)
  File "/home/test/work/espnet/tools/venv/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 279, in __iter__
    return _MultiProcessingDataLoaderIter(self)
  File "/home/test/work/espnet/tools/venv/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 719, in __init__
    w.start()
  File "/home/test/work/espnet/tools/venv/lib/python3.7/multiprocessing/process.py", line 112, in start
    self._popen = self._Popen(self)
  File "/home/test/work/espnet/tools/venv/lib/python3.7/multiprocessing/context.py", line 223, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "/home/test/work/espnet/tools/venv/lib/python3.7/multiprocessing/context.py", line 284, in _Popen
    return Popen(process_obj)
  File "/home/test/work/espnet/tools/venv/lib/python3.7/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
  File "/home/test/work/espnet/tools/venv/lib/python3.7/multiprocessing/popen_fork.py", line 20, in __init__
    self._launch(process_obj)
  File "/home/test/work/espnet/tools/venv/lib/python3.7/multiprocessing/popen_spawn_posix.py", line 47, in _launch
    reduction.dump(process_obj, fp)
  File "/home/test/work/espnet/tools/venv/lib/python3.7/multiprocessing/reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
TypeError: cannot serialize '_io.BufferedReader' object
Process SpawnProcess-2:
Traceback (most recent call last):
  File "/home/test/work/espnet/tools/venv/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
  File "/home/test/work/espnet/tools/venv/lib/python3.7/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "/home/test/work/espnet/espnet2/tasks/abs_task.py", line 1222, in main_worker
    distributed_option=distributed_option,
  File "/home/test/work/espnet/espnet2/train/trainer.py", line 212, in run
    options=trainer_options,
  File "/home/test/work/espnet/espnet2/train/trainer.py", line 367, in train_one_epoch
    reporter.measure_iter_time(iterator, "iter_time"), 1
  File "/home/test/work/espnet/espnet2/train/reporter.py", line 204, in measure_iter_time
    iterator = iter(iterable)
  File "/home/test/work/espnet/tools/venv/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 279, in __iter__
    return _MultiProcessingDataLoaderIter(self)
  File "/home/test/work/espnet/tools/venv/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 719, in __init__
    w.start()
  File "/home/test/work/espnet/tools/venv/lib/python3.7/multiprocessing/process.py", line 112, in start
    self._popen = self._Popen(self)
  File "/home/test/work/espnet/tools/venv/lib/python3.7/multiprocessing/context.py", line 223, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "/home/test/work/espnet/tools/venv/lib/python3.7/multiprocessing/context.py", line 284, in _Popen
    return Popen(process_obj)
  File "/home/test/work/espnet/tools/venv/lib/python3.7/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
  File "/home/test/work/espnet/tools/venv/lib/python3.7/multiprocessing/popen_fork.py", line 20, in __init__
    self._launch(process_obj)
  File "/home/test/work/espnet/tools/venv/lib/python3.7/multiprocessing/popen_spawn_posix.py", line 47, in _launch
    reduction.dump(process_obj, fp)
  File "/home/test/work/espnet/tools/venv/lib/python3.7/multiprocessing/reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
TypeError: cannot serialize '_io.BufferedReader' object
Traceback (most recent call last):
  File "/home/test/work/espnet/tools/venv/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/home/test/work/espnet/tools/venv/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/test/work/espnet/espnet2/bin/tts_train.py", line 22, in <module>
    main()
  File "/home/test/work/espnet/espnet2/bin/tts_train.py", line 18, in main
    TTSTask.main(cmd=cmd)
  File "/home/test/work/espnet/espnet2/tasks/abs_task.py", line 931, in main
    while not ProcessContext(processes, error_queues).join():
  File "/home/test/work/espnet/tools/venv/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 113, in join
    (error_index, exitcode)
Exception: process 0 terminated with exit code 1
# Accounting: time=24 threads=1
# Ended (code 1) at Wed Jul 22 18:24:10 JST 2020, elapsed time 24 seconds
```

## Environment:

- CentOS 7.7
- Conda / Python 3.7.3 / Pytorch 1.5.1 / cuda 10.1
- ESPnet2

## Note: 

- Use `cmd_backend="local"`
- Single GPU + `num_workers > 0` is fine.
- Single node multi-gpu distributed training + `num_workers=0` is fine.


Discussion:

Resolving patch:
diff --git a/espnet2/text/phoneme_tokenizer.py b/espnet2/text/phoneme_tokenizer.py
--- a/espnet2/text/phoneme_tokenizer.py
+++ b/espnet2/text/phoneme_tokenizer.py
@@ -1,4 +1,3 @@
-import functools
 from pathlib import Path
 from typing import Iterable
 from typing import List
@@ -26,12 +25,6 @@ def pyopenjtalk_g2p_kana(text) -> List[str]:
     return list(kanas)
 
 
-def g2p_en_no_space(g2p, text) -> List[str]:
-    # remove space which represents word serapater
-    phones = list(filter(lambda s: s != " ", g2p(text)))
-    return phones
-
-
 def pypinyin_g2p(text) -> List[str]:
     from pypinyin import pinyin
     from pypinyin import Style
@@ -58,6 +51,30 @@ def pypinyin_g2p_phone(text) -> List[str]:
     return phones
 
 
+class G2p_en:
+    """On behalf of g2p_en.G2p.
+
+    g2p_en.G2p isn't pickalable and it can't be copied to the other processes
+    via multiprocessing module.
+    As a workaround, g2p_en.G2p is instantiated upon calling this class.
+
+    """
+
+    def __init__(self, no_space: bool = False):
+        self.no_space = no_space
+        self.g2p = None
+
+    def __call__(self, text) -> List[str]:
+        if self.g2p is None:
+            self.g2p = g2p_en.G2p()
+
+        phones = self.g2p(text)
+        if self.no_space:
+            # remove space which represents word serapater
+            phones = list(filter(lambda s: s != " ", phones))
+        return phones
+
+
 class PhonemeTokenizer(AbsTokenizer):
     def __init__(
         self,
@@ -68,11 +85,9 @@ def __init__(
     ):
         assert check_argument_types()
         if g2p_type == "g2p_en":
-            self.g2p = g2p_en.G2p()
+            self.g2p = G2p_en(no_space=False)
         elif g2p_type == "g2p_en_no_space":
-            # TODO(kan-bayashi): Include within a function?
-            g2p = g2p_en.G2p()
-            self.g2p = functools.partial(g2p_en_no_space, g2p)
+            self.g2p = G2p_en(no_space=True)
         elif g2p_type == "pyopenjtalk":
             self.g2p = pyopenjtalk_g2p
         elif g2p_type == "pyopenjtalk_kana":


Test patch: