Repository: PaddlePaddle/PaddleSpeech

Issue:
mac下run_train.sh内存占用持续增加，有泄漏？
我想用自己的语料来训练DeepSpeech， 训练过程系统内存占用持续增高，直至交换文件把磁盘撑爆。
但是python进程本身的内存占用又没有增加，不知道是哪里吃的内存。
```
(paddle)loong@MacBook-Pro:~/l/lab/py/ml/baidu/wav on master$ sh run_train.sh 
-----------  Configuration Arguments -----------
augment_conf_path: arg.config
batch_size: 4
dev_manifest: data/manifest.train
init_model_path: None
is_local: 1
learning_rate: 5e-05
max_duration: 27.0
mean_std_path: data/mean_std.npz
min_duration: 0.0
num_conv_layers: 2
num_iter_print: 100
num_passes: 40
num_proc_data: 16
num_rnn_layers: 3
output_model_dir: ./models
rnn_layer_size: 1024
share_rnn_weights: 0
shuffle_method: batch_shuffle_clipped
specgram_type: linear
test_off: 0
train_manifest: data/manifest.train
trainer_count: 1
use_gpu: 0
use_gru: 0
use_sortagrad: 1
vocab_path: data/vocab.txt
------------------------------------------------
I0202 21:40:14.468683 2907198400 Util.cpp:166] commandline:  --use_gpu=0 --rnn_use_batch=True --log_clipping=True --trainer_count=1 
[INFO 2018-02-02 21:40:14,484 layers.py:2689] output for __conv_0__: c = 32, h = 81, w = 54, size = 139968
[INFO 2018-02-02 21:40:14,485 layers.py:3251] output for __batch_norm_0__: c = 32, h = 81, w = 54, size = 139968
[INFO 2018-02-02 21:40:14,487 layers.py:7409] output for __scale_sub_region_0__: c = 32, h = 81, w = 54, size = 139968
[INFO 2018-02-02 21:40:14,488 layers.py:2689] output for __conv_1__: c = 32, h = 41, w = 54, size = 70848
[INFO 2018-02-02 21:40:14,490 layers.py:3251] output for __batch_norm_1__: c = 32, h = 41, w = 54, size = 70848
[INFO 2018-02-02 21:40:14,493 layers.py:7409] output for __scale_sub_region_1__: c = 32, h = 41, w = 54, size = 70848
I0202 21:40:14.751354 2907198400 GradientMachine.cpp:94] Initing parameters..
I0202 21:40:15.287204 2907198400 GradientMachine.cpp:101] Init parameters done.
```
![1](https://loongw.github.io/assets/train-1.png)
![2](https://loongw.github.io/assets/train-2.png)
![3](https://loongw.github.io/assets/train-3.png)



Discussion:
@loongw 在mac上我们没有做长时间的测试，或者是你环境的原因，内存泄漏的问题是可能存在的。建议用GPU做训练，或者在内存压力过大时中断训练、再从保存的模型恢复训练来回避这个问题。
@kuke 谢谢建议，我暂时没有硬件资源用GPU做训练。
我尝试了每次训练11个pass后重启run_train.sh, 然后接着最后一个模型恢复训练。 但是这样操作后cost表现出周期性变化，感觉和连续的训练并不等效，这是什么原因呢？
下面是截取部分日志：
```
I0203 17:46:44.953709 2950288320 GradientMachine.cpp:94] Initing parameters..
I0203 17:46:45.454922 2950288320 GradientMachine.cpp:101] Init parameters done.
..................................................
------- Time: 235 sec,  Pass: 0, ValidationCost: 0.995058694774
..................................................
------- Time: 256 sec,  Pass: 1, ValidationCost: 0.960387979783
..................................................
------- Time: 245 sec,  Pass: 2, ValidationCost: 3.14510204502
..................................................
------- Time: 242 sec,  Pass: 3, ValidationCost: 1.61891988583
..................................................
------- Time: 237 sec,  Pass: 4, ValidationCost: 3.42624592213
..................................................
------- Time: 247 sec,  Pass: 5, ValidationCost: 0.16420545101
..................................................
------- Time: 260 sec,  Pass: 6, ValidationCost: 0.305833122782
..................................................
------- Time: 236 sec,  Pass: 7, ValidationCost: 0.81359860881
..................................................
------- Time: 264 sec,  Pass: 8, ValidationCost: 4.03333070365
..................................................
------- Time: 231 sec,  Pass: 9, ValidationCost: 0.0967882538334
..................................................
------- Time: 234 sec,  Pass: 10, ValidationCost: 3.35563764986

I0203 18:32:29.012394 2950288320 GradientMachine.cpp:94] Initing parameters..
I0203 18:32:29.517359 2950288320 GradientMachine.cpp:101] Init parameters done.
..................................................
------- Time: 233 sec,  Pass: 0, ValidationCost: 1.05707970927
..................................................
------- Time: 258 sec,  Pass: 1, ValidationCost: 1.00350321605
..................................................
------- Time: 244 sec,  Pass: 2, ValidationCost: 3.23925462924
..................................................
------- Time: 246 sec,  Pass: 3, ValidationCost: 1.70067558736
..................................................
------- Time: 237 sec,  Pass: 4, ValidationCost: 3.57716395282
..................................................
------- Time: 253 sec,  Pass: 5, ValidationCost: 0.172555627104
..................................................
------- Time: 261 sec,  Pass: 6, ValidationCost: 0.319965159111
..................................................
------- Time: 240 sec,  Pass: 7, ValidationCost: 0.860942301395
..................................................
------- Time: 256 sec,  Pass: 8, ValidationCost: 4.20200282017
..................................................
------- Time: 232 sec,  Pass: 9, ValidationCost: 0.0996196654899
..................................................
------- Time: 242 sec,  Pass: 10, ValidationCost: 3.53666487252

I0203 19:18:29.237448 2950288320 GradientMachine.cpp:94] Initing parameters..
I0203 19:18:29.797590 2950288320 GradientMachine.cpp:101] Init parameters done.
..................................................
------- Time: 235 sec,  Pass: 0, ValidationCost: 1.10210716111
..................................................
------- Time: 256 sec,  Pass: 1, ValidationCost: 1.04657455226
..................................................
------- Time: 245 sec,  Pass: 2, ValidationCost: 3.32439532554
..................................................
------- Time: 240 sec,  Pass: 3, ValidationCost: 1.77741941035
..................................................
------- Time: 240 sec,  Pass: 4, ValidationCost: 3.71267391714
..................................................
------- Time: 240 sec,  Pass: 5, ValidationCost: 0.183090231603
..................................................
------- Time: 262 sec,  Pass: 6, ValidationCost: 0.342262154668
..................................................
------- Time: 238 sec,  Pass: 7, ValidationCost: 0.898480577733
..................................................
------- Time: 263 sec,  Pass: 8, ValidationCost: 4.36341414607
..................................................
------- Time: 231 sec,  Pass: 9, ValidationCost: 0.104793594162
..................................................
------- Time: 238 sec,  Pass: 10, ValidationCost: 3.74533850967
```
@loongw 应该是恢复训练的方式不对，你每次恢复训练的时候正确指定```init_model_path```了吗
@kuke  每轮生成0-10,共11个params文件，每次恢复训练init_model_path指向的是params.pass-10.tar.gz，这样应该是正确的吧？
我又倒回去查了一下内存泄露的原因，找到原因了
每个epoch结束后worker进程没收拾干净
我提一个PR吧

Resolving patch:
diff --git a/data_utils/data.py b/data_utils/data.py
--- a/data_utils/data.py
+++ b/data_utils/data.py
@@ -11,7 +11,6 @@
 import numpy as np
 import paddle.v2 as paddle
 from threading import local
-import atexit
 from data_utils.utility import read_manifest
 from data_utils.utility import xmap_readers_mp
 from data_utils.augmentor.augmentation import AugmentationPipeline
@@ -194,15 +193,18 @@ def batch_reader():
                     raise ValueError("Unknown shuffle method %s." %
                                      shuffle_method)
             # prepare batches
-            instance_reader = self._instance_reader_creator(manifest)
+            instance_reader, cleanup = self._instance_reader_creator(manifest)
             batch = []
-            for instance in instance_reader():
-                batch.append(instance)
-                if len(batch) == batch_size:
+            try:
+                for instance in instance_reader():
+                    batch.append(instance)
+                    if len(batch) == batch_size:
+                        yield self._padding_batch(batch, padding_to, flatten)
+                        batch = []
+                if len(batch) >= min_batch_size:
                     yield self._padding_batch(batch, padding_to, flatten)
-                    batch = []
-            if len(batch) >= min_batch_size:
-                yield self._padding_batch(batch, padding_to, flatten)
+            finally:
+                cleanup()
             self._epoch += 1
 
         return batch_reader
@@ -280,10 +282,7 @@ def reader():
             lambda instance: self.process_utterance(instance["audio_filepath"], instance["text"]),
             reader, self._num_threads, 4096)
 
-        # register callback to main process
-        atexit.register(cleanup_callback)
-
-        return reader
+        return reader, cleanup_callback
 
     def _padding_batch(self, batch, padding_to=-1, flatten=False):
         """


Test patch: