Repository: OpenNMT/OpenNMT-py

Issue:
'BARTNoiseTransform' object has no attribute 'bart_noise'
I update the code and use the configure like
`
insert_ratio: 0.1 
random_ratio: 1.0 
mask_ratio: 0.15
rotate_ratio: 0.0 
mask_length: 'span-poisson'
poisson_lambda: 1
replace_length: 1`
`
the problem would happen,

Traceback (most recent call last):
 18   File "/data/app/anaconda3/envs/pytorch-1.2/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
 19     self.run()
 20   File "/data/app/anaconda3/envs/pytorch-1.2/lib/python3.7/multiprocessing/process.py", line 99, in run
 21     self._target(*self._args, **self._kwargs)
 22   File "/data/app/anaconda3/envs/pytorch-1.2/lib/python3.7/site-packages/OpenNMT_py-2.0.0rc1-py3.7.egg/onmt/utils/distributed.py", line 189, in batch_producer
 23     b = next_batch()
 24   File "/data/app/anaconda3/envs/pytorch-1.2/lib/python3.7/site-packages/OpenNMT_py-2.0.0rc1-py3.7.egg/onmt/utils/distributed.py", line 185, in next_batch
 25     new_batch = next(generator_to_serve)
 26   File "/data/app/anaconda3/envs/pytorch-1.2/lib/python3.7/site-packages/OpenNMT_py-2.0.0rc1-py3.7.egg/onmt/inputters/dynamic_iterator.py", line 172, in __iter__
 27     self._init_datasets()
 28   File "/data/app/anaconda3/envs/pytorch-1.2/lib/python3.7/site-packages/OpenNMT_py-2.0.0rc1-py3.7.egg/onmt/inputters/dynamic_iterator.py", line 151, in _init_datasets
 29     stride=self.stride, offset=self.offset)
 30   File "/data/app/anaconda3/envs/pytorch-1.2/lib/python3.7/site-packages/OpenNMT_py-2.0.0rc1-py3.7.egg/onmt/inputters/corpus.py", line 255, in build_corpora_iters
 31     logger.info(f"{c_id}'s transforms: {str(transform_pipe)}")
 32   File "/data/app/anaconda3/envs/pytorch-1.2/lib/python3.7/site-packages/OpenNMT_py-2.0.0rc1-py3.7.egg/onmt/transforms/transform.py", line 85, in __repr__
 33     cls_args = self._repr_args()
 34   File "/data/app/anaconda3/envs/pytorch-1.2/lib/python3.7/site-packages/OpenNMT_py-2.0.0rc1-py3.7.egg/onmt/transforms/transform.py", line 208, in _repr_args
 35     info_args.append(repr(transform))
 36   File "/data/app/anaconda3/envs/pytorch-1.2/lib/python3.7/site-packages/OpenNMT_py-2.0.0rc1-py3.7.egg/onmt/transforms/transform.py", line 85, in __repr__
 37     cls_args = self._repr_args()
 38   File "/data/app/anaconda3/envs/pytorch-1.2/lib/python3.7/site-packages/OpenNMT_py-2.0.0rc1-py3.7.egg/onmt/transforms/bart.py", line 386, in _repr_args
 39     return repr(self.bart_noise)
`


Discussion:
That's strange.
@Zenglinxiao the `bart_noise` attribute should always be set in `BARTNoiseTransform.warm_up`, right? (Or if not it should raise a more explicit exception.)

@lianrzh can you share your full config file?
@francoishernandez ,
I checked the code, the `bart_noise` should always be set in `warm_up` except when vocabs=None which may happens in `build_vocab`. So we may need to fix this by add a line `self.bart_noise = None` for this case in: https://github.com/OpenNMT/OpenNMT-py/blob/dfdd4554cf1c9327c2c249ab11a978efe72a93d9/onmt/transforms/bart.py#L350-L353
But this should run correctely in `train.py` and have been tested when I made relevent changes. The Traceback reported by @lianrzh seems from the train.py script? If this is the case, could you please provide full Traceback with the Exception line? And you may test by add the mentioned line to the corresponding code to see if that also fix your issue? 
@Zenglinxiao I add the mentional line, and the Traceback is "AttributeError: 'BARTNoiseTransform' object has no attribute 'vocabs'". It seems that the vocabs is None. It's strange that is works well when the world_size = 1 and the problem happens when using more  world.  Is there something wrong with multiple gpus?
> @Zenglinxiao I add the mentional line, and the Traceback is "AttributeError: 'BARTNoiseTransform' object has no attribute 'vocabs'". It seems that the vocabs is None. It's strange that is works well when the world_size = 1 and the problem happens when using more world. Is there something wrong with multiple gpus?

I've reproduced this error. No, the multiprocessing works fine. The issue linked to the picking/unpicking method of `Transform` class. Where the vocabs passing to `warm_up` is always `None`. This method will only be used in the case of multi-gpu by producer process for which reason you would not have such problem in single process. https://github.com/OpenNMT/OpenNMT-py/blob/dfdd4554cf1c9327c2c249ab11a978efe72a93d9/onmt/transforms/transform.py#L68-L73

I'll open a PR to fix it.

Resolving patch:
diff --git a/onmt/transforms/bart.py b/onmt/transforms/bart.py
--- a/onmt/transforms/bart.py
+++ b/onmt/transforms/bart.py
@@ -350,6 +350,7 @@ def add_options(cls, parser):
     def warm_up(self, vocabs):
         super().warm_up(None)
         if vocabs is None:
+            self.bart_noise = None
             return
         self.vocabs = vocabs
 
diff --git a/onmt/transforms/transform.py b/onmt/transforms/transform.py
--- a/onmt/transforms/transform.py
+++ b/onmt/transforms/transform.py
@@ -53,7 +53,10 @@ def apply(self, example, is_train=False, stats=None, **kwargs):
 
     def __getstate__(self):
         """Pickling following for rebuild."""
-        return self.opts
+        state = {"opts": self.opts}
+        if hasattr(self, 'vocabs'):
+            state['vocabs'] = self.vocabs
+        return state
 
     def _parse_opts(self):
         """Parse opts to set/reset instance's attributes.
@@ -65,11 +68,11 @@ def _parse_opts(self):
         """
         pass
 
-    def __setstate__(self, opts):
+    def __setstate__(self, state):
         """Reload when unpickling from save file."""
-        self.opts = opts
+        self.opts = state["opts"]
         self._parse_opts()
-        vocabs = self.vocabs if hasattr(self, 'vocabs') else None
+        vocabs = state.get('vocabs', None)
         self.warm_up(vocabs=vocabs)
 
     def stats(self):


Test patch: