RAG end to end with RAY throws pickling error
Author: sb1992
Pull request: no

**Environmnet info:**

Transformers:4.5.1
Paltform:Ubuntu
Python:3.7
Torch 1.6.0 
Gpus = yes
Distributed: Ray (1.3.0)

**Information**

Am using RAG end2end-retriever from examples code
examples/research_projects/rag-end2end-retriever

For the time being just trying on dummy data and just trying the script given there
sh ./test_run/test_finetune.sh 
Used the config there except just changes number of gpus to 4 and changed the gpu order

**Who can help**
@shamanez 

**Error**

Throws pickling error

      Loading passages from test_run/dummy-kb/my_knowledge_dataset
      Traceback (most recent call last):
        File "finetune_rag.py", line 790, in <module>
          main(args)
        File "finetune_rag.py", line 727, in main
          model: GenerativeQAModule = GenerativeQAModule(args)
        File "finetune_rag.py", line 124, in __init__
          hparams.model_name_or_path, hparams.actor_handles, config=config
        File "/home/shraeyb/transformers/examples/research_projects/rag-end2end-retriever/distributed_ray_retriever.py", line 165, in from_pretrained
          index=index,
        File "/home/shraeyb/transformers/examples/research_projects/rag-end2end-retriever/distributed_ray_retriever.py", line 93, in __init__
          for worker in self.retrieval_workers
        File "/home/shraeyb/transformers/examples/research_projects/rag-end2end-retriever/distributed_ray_retriever.py", line 93, in <listcomp>
          for worker in self.retrieval_workers
        File "/home/shraeyb/anaconda3/envs/new_st/lib/python3.7/site-packages/ray/actor.py", line 112, in remote
          return self._remote(args, kwargs)
        File "/home/shraeyb/anaconda3/envs/new_st/lib/python3.7/site-packages/ray/actor.py", line 153, in _remote
          return invocation(args, kwargs)
        File "/home/shraeyb/anaconda3/envs/new_st/lib/python3.7/site-packages/ray/actor.py", line 147, in invocation
          num_returns=num_returns)
        File "/home/shraeyb/anaconda3/envs/new_st/lib/python3.7/site-packages/ray/actor.py", line 865, in _actor_method_call
          list_args, name, num_returns, self._ray_actor_method_cpus)
        File "python/ray/_raylet.pyx", line 1359, in ray._raylet.CoreWorker.submit_actor_task
        File "python/ray/_raylet.pyx", line 1364, in ray._raylet.CoreWorker.submit_actor_task
        File "python/ray/_raylet.pyx", line 304, in ray._raylet.prepare_args
        File "/home/shraeyb/anaconda3/envs/new_st/lib/python3.7/site-packages/ray/serialization.py", line 324, in serialize
          return self._serialize_to_msgpack(value)
        File "/home/shraeyb/anaconda3/envs/new_st/lib/python3.7/site-packages/ray/serialization.py", line 304, in _serialize_to_msgpack
          self._serialize_to_pickle5(metadata, python_objects)
        File "/home/shraeyb/anaconda3/envs/new_st/lib/python3.7/site-packages/ray/serialization.py", line 264, in _serialize_to_pickle5
          raise e
        File "/home/shraeyb/anaconda3/envs/new_st/lib/python3.7/site-packages/ray/serialization.py", line 261, in _serialize_to_pickle5
          value, protocol=5, buffer_callback=writer.buffer_callback)
        File "/home/shraeyb/anaconda3/envs/new_st/lib/python3.7/site-packages/ray/cloudpickle/cloudpickle_fast.py", line 73, in dumps
          cp.dump(obj)
        File "/home/shraeyb/anaconda3/envs/new_st/lib/python3.7/site-packages/ray/cloudpickle/cloudpickle_fast.py", line 580, in dump
          return Pickler.dump(self, obj)
        File "pyarrow/io.pxi", line 1021, in pyarrow.lib.Buffer.__reduce_ex__
      AttributeError: module 'pickle' has no attribute 'PickleBuffer'
      INFO:wandb.internal.internal:Internal process exited
      /home/shraeyb/anaconda3/envs/new_st/lib/python3.7/site-packages/ray/autoscaler/_private/cli_logger.py:61: FutureWarning: Not all Ray CLI dependencies were found. In Ray 1.4+, the Ray CLI, autoscaler, and dashboard will only be usable via `pip install 'ray[default]'`. Please update your install command.
        "update your install command.", FutureWarning)
      Stopped all 13 Ray processes.

Comments:
Check this[ StackOverflow question](https://stackoverflow.com/questions/67798070/raytune-is-throwing-error-module-pickle-has-no-attribute-picklebuffer-whe).  btw I use python 3.8 and it worked perfectly for me.

Yup i did but that is also just the question without any answer unfortunately.

Hi there, I executed the code again by installing RAY just to double-check. It is working perfectly for me.   I also use an anaconda and RAY with 1.3.0. 

The only difference is python 3.8 instead of 3.7. 

I would suggest you to update the anaconda and trying to run the script.

Alright thank you, i used python 3.8 now ( though my cuda is 10.1), That error looks resolved but something new pops up, Still trying on dummy data

        File "finetune_rag.py", line 790, in <module>
          main(args)
        File "finetune_rag.py", line 727, in main
          model: GenerativeQAModule = GenerativeQAModule(args)
        File "finetune_rag.py", line 131, in __init__
          retriever.set_ctx_encoder_tokenizer(ctx_encoder_tokenizer)
      AttributeError: 'RagRayDistributedRetriever' object has no attribute 'set_ctx_encoder_tokenizer'
      Exception ignored in: <function ActorHandle.__del__ at 0x7fc4c143aee0>
      Traceback (most recent call last):
        File "/home/shraeyb/anaconda3/envs/py38/lib/python3.8/site-packages/ray/actor.py", line 809, in __del__
      AttributeError: 'NoneType' object has no attribute 'global_worker'
      Exception ignored in: <function ActorHandle.__del__ at 0x7fc4c143aee0>
      Traceback (most recent call last):
        File "/home/shraeyb/anaconda3/envs/py38/lib/python3.8/site-packages/ray/actor.py", line 809, in __del__
      AttributeError: 'NoneType' object has no attribute 'global_worker'
      Exception ignored in: <function ActorHandle.__del__ at 0x7fc4c143aee0>
      Traceback (most recent call last):
        File "/home/shraeyb/anaconda3/envs/py38/lib/python3.8/site-packages/ray/actor.py", line 809, in __del__
      AttributeError: 'NoneType' object has no attribute 'global_worker'
      Exception ignored in: <function ActorHandle.__del__ at 0x7fc4c143aee0>
      Traceback (most recent call last):
        File "/home/shraeyb/anaconda3/envs/py38/lib/python3.8/site-packages/ray/actor.py", line 809, in __del__
      AttributeError: 'NoneType' object has no attribute 'global_worker'
      Stopped all 13 Ray processes.

Ah, I think you have to install the library from sources. Still, this is not in the pip version.

You mean ray 1.3 from source?

Tranformers library.

On Fri, Jun 4, 2021, 02:32 Shraey Bhatia ***@***.***> wrote:

> You mean ray 1.3 from source?
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/transformers/issues/12012#issuecomment-853916373>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AEA4FGRFOV4TWZSIZNP3IODTQ6HARANCNFSM46AZYZBQ>
> .
>


Thank you. This solves it. Indded needs to be python 3.8
Though i have a few 8gb gpus so cannot really fit on my box, goes out of memory. What gpu config did you use?

Oh.  I am working with 32Gb gpus. This has nearly 800 M parameters. Bart
large model and two Bert base models.

On Fri, Jun 4, 2021, 02:58 Shraey Bhatia ***@***.***> wrote:

> Thank you. This solves it. Indded needs to be python 3.8
> Though i have a few 8gb gpus so cannot really fit on my box, goes out of
> memory. What gpu config did you use?
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/transformers/issues/12012#issuecomment-853935772>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AEA4FGS4TI7XJMAZAK3NOXDTQ6KCNANCNFSM46AZYZBQ>
> .
>


No worries. thats fine, I will have to submit jobs to a cluster ( is just tedious to do that). Thank you