[end2end RAG]  AttributeError: module 'pickle' has no attribute 'PickleBuffer'
Author: shunyuzh
Pull request: no

Hi folks,

@shamanez , thanks for your awesome project of end2end RAG. But when I reproduce the results of https://github.com/huggingface/transformers/tree/master/examples/research_projects/rag-end2end-retriever, I face some problems.


```
Traceback (most recent call last):
  File "finetune_rag.py", line 789, in <module>
    main(args)
  File "finetune_rag.py", line 726, in main
    model: GenerativeQAModule = GenerativeQAModule(args)
  File "finetune_rag.py", line 123, in __init__
    hparams.model_name_or_path, hparams.actor_handles, config=config
  File "/home/shunyu/container/Project/transformers/examples/research_projects/rag-end2end-retriever/distributed_ray_retriever.py", line 165, in from_pretrained
    index=index,
  File "/home/shunyu/container/Project/transformers/examples/research_projects/rag-end2end-retriever/distributed_ray_retriever.py", line 93, in __init__
    for worker in self.retrieval_workers
  File "/home/shunyu/container/Project/transformers/examples/research_projects/rag-end2end-retriever/distributed_ray_retriever.py", line 93, in <listcomp>
    for worker in self.retrieval_workers
  File "/anaconda/envs/rag2/lib/python3.6/site-packages/ray/actor.py", line 112, in remote
    return self._remote(args, kwargs)
  File "/anaconda/envs/rag2/lib/python3.6/site-packages/ray/actor.py", line 153, in _remote
    return invocation(args, kwargs)
  File "/anaconda/envs/rag2/lib/python3.6/site-packages/ray/actor.py", line 147, in invocation
    num_returns=num_returns)
  File "/anaconda/envs/rag2/lib/python3.6/site-packages/ray/actor.py", line 865, in _actor_method_call
    list_args, name, num_returns, self._ray_actor_method_cpus)
  File "python/ray/_raylet.pyx", line 1359, in ray._raylet.CoreWorker.submit_actor_task
  File "python/ray/_raylet.pyx", line 1364, in ray._raylet.CoreWorker.submit_actor_task
  File "python/ray/_raylet.pyx", line 304, in ray._raylet.prepare_args
  File "/anaconda/envs/rag2/lib/python3.6/site-packages/ray/serialization.py", line 324, in serialize
    return self._serialize_to_msgpack(value)
  File "/anaconda/envs/rag2/lib/python3.6/site-packages/ray/serialization.py", line 304, in _serialize_to_msgpack
    self._serialize_to_pickle5(metadata, python_objects)
  File "/anaconda/envs/rag2/lib/python3.6/site-packages/ray/serialization.py", line 264, in _serialize_to_pickle5
    raise e
  File "/anaconda/envs/rag2/lib/python3.6/site-packages/ray/serialization.py", line 261, in _serialize_to_pickle5
    value, protocol=5, buffer_callback=writer.buffer_callback)
  File "/anaconda/envs/rag2/lib/python3.6/site-packages/ray/cloudpickle/cloudpickle_fast.py", line 73, in dumps
    cp.dump(obj)
  File "/anaconda/envs/rag2/lib/python3.6/site-packages/ray/cloudpickle/cloudpickle_fast.py", line 580, in dump
    return Pickler.dump(self, obj)
  File "pyarrow/io.pxi", line 1021, in pyarrow.lib.Buffer.__reduce_ex__
AttributeError: module 'pickle' has no attribute 'PickleBuffer'
```

We think it's mainly related to the version and dependency of ray, pyarrow and datasets.
My main pip list:
```
faiss-cpu == 1.7.0
datasets == 1.6.2
psutil == 5.7.0
torch == 1.6.0
pytorch-lightning ==  1.3.1
nvidia-ml-py3 == 7.352.0
ray ==  1.3.0
pyarrow == 3.0.0
```

I think anyone can investigate it from this reproducible example:
```
git clone https://github.com/huggingface/transformers
cd transformers
pip install .

cd ./examples/research_projects/rag-end2end-retriever
pip install -r requirements.txt

bash ./test_run/test_finetune.sh
```

And someone was likely to face the same questions  https://discuss.ray.io/t/cant-pickle-pyarrow-dataset-expression/1685/8

So @shamanez, could you please show the entire pip list of the env to run END2END RAG? Or point out how to fix it?
Let me know if more information is needed and thanks for your help.

Comments:
Hi,

Thanks a lot.  I think this error is due to Python version. Check with 3.8 or above. It will work.

> Hi,
> 
> Thanks a lot. I think this error is due to Python version. Check with 3.8 or above. It will work.

Thank you, it really worked when I run test_finetune.sh. 

Emm, it's silly that I have tried to change Python version from 3.6 to 3.7, but forgot the 3.8. 

Perfect:)

On Mon, Jun 7, 2021, 21:41 Dopaminezsy ***@***.***> wrote:

> Hi,
>
> Thanks a lot. I think this error is due to Python version. Check with 3.8
> or above. It will work.
>
> Thank you, it really worked when I run test_finetune.sh.
>
> Emm, it's silly that I have tried to change Python version from 3.6 to
> 3.7, but forgot the 3.8.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/transformers/issues/12050#issuecomment-855777762>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AEA4FGRPFOPULVKPX2AAR7LTRSH5PANCNFSM46HEVE4A>
> .
>


Hi, friend @shamanez :
Sorry to disturb you again. I face the following bug when run finetune_rag_ray_end2end.sh. 
Could you give some sugguestions?

```
2021-06-08 09:40:19,202	INFO worker.py:726 -- Connecting to existing Ray cluster at address: 10.34.96.6:6379
INFO:__main__:Getting named actors for NODE_RANK 0, LOCAL_RANK 1
Traceback (most recent call last):
  File "/workspaceblobstore/azureml/rag1_1623144585_ed3652ce/transformers/examples/research_projects/rag-end2end-retriever/finetune_rag.py", line 794, in <module>
    main(args)
  File "/workspaceblobstore/azureml/rag1_1623144585_ed3652ce/transformers/examples/research_projects/rag-end2end-retriever/finetune_rag.py", line 726, in main
    named_actors = [ray.get_actor("retrieval_worker_{}".format(i)) for i in range(args.num_retrieval_workers)]
  File "/workspaceblobstore/azureml/rag1_1623144585_ed3652ce/transformers/examples/research_projects/rag-end2end-retriever/finetune_rag.py", line 726, in <listcomp>
    named_actors = [ray.get_actor("retrieval_worker_{}".format(i)) for i in range(args.num_retrieval_workers)]
  File "/home/t-shzhang/.local/lib/python3.8/site-packages/ray/_private/client_mode_hook.py", line 62, in wrapper
    return func(*args, **kwargs)
  File "/home/t-shzhang/.local/lib/python3.8/site-packages/ray/worker.py", line 1659, in get_actor
    handle = worker.core_worker.get_named_actor_handle(name)
  File "python/ray/_raylet.pyx", line 1521, in ray._raylet.CoreWorker.get_named_actor_handle
  File "python/ray/_raylet.pyx", line 159, in ray._raylet.check_status
ValueError: Failed to look up actor with name 'retrieval_worker_0'. You are either trying to look up a named actor you didn't create, the named actor died, or the actor hasn't been created because named actor creation is asynchronous.
```

Seems like a problem in your cluster.  What is your system. Seems like it
is a multi node system.

On Tue, Jun 8, 2021, 21:52 Dopaminezsy ***@***.***> wrote:

> Hi, friend @shamanez <https://github.com/shamanez> :
> Sorry to disturb you again. I face the following bug when run
> finetune_rag_ray_end2end.sh.
> Could you give some sugguestions?
>
> 2021-06-08 09:40:19,202	INFO worker.py:726 -- Connecting to existing Ray cluster at address: 10.34.96.6:6379
> INFO:__main__:Getting named actors for NODE_RANK 0, LOCAL_RANK 1
> Traceback (most recent call last):
>   File "/workspaceblobstore/azureml/rag1_1623144585_ed3652ce/transformers/examples/research_projects/rag-end2end-retriever/finetune_rag.py", line 794, in <module>
>     main(args)
>   File "/workspaceblobstore/azureml/rag1_1623144585_ed3652ce/transformers/examples/research_projects/rag-end2end-retriever/finetune_rag.py", line 726, in main
>     named_actors = [ray.get_actor("retrieval_worker_{}".format(i)) for i in range(args.num_retrieval_workers)]
>   File "/workspaceblobstore/azureml/rag1_1623144585_ed3652ce/transformers/examples/research_projects/rag-end2end-retriever/finetune_rag.py", line 726, in <listcomp>
>     named_actors = [ray.get_actor("retrieval_worker_{}".format(i)) for i in range(args.num_retrieval_workers)]
>   File "/home/t-shzhang/.local/lib/python3.8/site-packages/ray/_private/client_mode_hook.py", line 62, in wrapper
>     return func(*args, **kwargs)
>   File "/home/t-shzhang/.local/lib/python3.8/site-packages/ray/worker.py", line 1659, in get_actor
>     handle = worker.core_worker.get_named_actor_handle(name)
>   File "python/ray/_raylet.pyx", line 1521, in ray._raylet.CoreWorker.get_named_actor_handle
>   File "python/ray/_raylet.pyx", line 159, in ray._raylet.check_status
> ValueError: Failed to look up actor with name 'retrieval_worker_0'. You are either trying to look up a named actor you didn't create, the named actor died, or the actor hasn't been created because named actor creation is asynchronous.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/transformers/issues/12050#issuecomment-856630780>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AEA4FGTIOEOYQODXAUE25S3TRXR6FANCNFSM46HEVE4A>
> .
>


Actually I have discussed this issue previously. 


This happens when you try to run the code in distributed mode. @calderma also mentioned the same thing.

https://github.com/huggingface/transformers/pull/11655#issuecomment-845295355


I think this is not an issue with Ray or anything. It is something with how you run a distributed code in Pytorch Lightining. Sadly I do not have a distributed system to test :(. 

But in the above thread I pointed out some workarounds.  Also I have mentioned the reason to get this issue.

Just to summarize.. we initialize RAY actors only in master process (when initializing the master ddp process). Other DDP processes simply access the RAY worker by its name. 


But when having a distributed system,  I think initialization should happen in each node.  In order to activate distributed training, you need to add **node** variable to lightning trainer. Then you should initialize the training as given in their tutoria. Please let me know how it goes.


Please follow the following commands to run the code in a Cluster.

https://pytorch-lightning.readthedocs.io/en/latest/clouds/cluster.html#general-purpose-cluster


> Actually I have discussed this issue previously.
> 
> This happens when you try to run the code in distributed mode. @calderma also mentioned the same thing.
> 
> [#11655 (comment)](https://github.com/huggingface/transformers/pull/11655#issuecomment-845295355)
> 
> I think this is not an issue with Ray or anything. It is something with how you run a distributed code in Pytorch Lightining. Sadly I do not have a distributed system to test :(.
> 
> But in the above thread I pointed out some workarounds. Also I have mentioned the reason to get this issue.
> 
> Just to summarize.. we initialize RAY actors only in master process (when initializing the master ddp process). Other DDP processes simply access the RAY worker by its name.
> 
> But when having a distributed system, I think initialization should happen in each node. In order to activate distributed training, you need to add **node** variable to lightning trainer. Then you should initialize the training as given in their tutoria. Please let me know how it goes.
> 
> Please follow the following commands to run the code in a Cluster.
> 
> https://pytorch-lightning.readthedocs.io/en/latest/clouds/cluster.html#general-purpose-cluster

Thank you a lot. I think it will help. 

I'm going to try.

Hi again, 
As for the DDP problem, I followed your instructions to add **node** variable before define trainer in lightning_base.py. But it didn't help and the BUG also as before. Do you know there are other instructions on PL?

```
    train_params["accelerator"] = "ddp" or "ddp2" or "dp"
    train_params["num_nodes"] = 1
    trainer = pl.Trainer.from_argparse_args(
        args,
        weights_summary=None,
        callbacks=[logging_callback] + extra_callbacks + [InitCallback()] + [checkpoint_callback],
        logger=logger,
        plugins=[DDPPlugin(find_unused_parameters=True)],  # this is needed in new pytorch-lightning new version
        val_check_interval=1,
        num_sanity_val_steps=2,
        **train_params,
    )
```
By the way, I think it may be related to the RAY, for that RAY works before pl.Trainer in finetune_ray.py? Feel free to point out my naive error.

Yeah Ray workers get initialized before starting the training loop.   Checkout if conditions mentioned in RAY worker initialization part. It basically says initialize only in the master process.


But when you try distributed training, it says it can't find already initialized worker. That means processes hasn't shared between nodes.

This is not a problem when you run the code in normal mode. I did not change anything to the RAY initialization.  You will get the same problem in Original RAG too.

Check https://pytorch-lightning.readthedocs.io/en/latest/clouds/cluster.html#general-purpose-cluster.


This has several ways to start your cluster. I think you need to play around with those environment variables.

@Dopaminezsy 
There is one more thing you can do. I updated the original [RAG](https://github.com/huggingface/transformers/blob/master/examples/research_projects/rag/finetune_rag.py) with the latest PL.  But I use custom plugging there (I did not use it for this project since it is still in an experimental plugging). Can you try to run the original RAG in distributed mode and let me know? 

Also, play around with these [lines](https://github.com/huggingface/transformers/blob/master/examples/research_projects/rag/finetune_rag.py#L535). 



One more thing, try to print something after this [line](https://github.com/huggingface/transformers/blob/master/examples/research_projects/rag-end2end-retriever/finetune_rag.py#L709). I want to know whether your code fails before the DDP process.  If it doesn't go inside the if condition, when starting, it is a problem with RAY, otherwise it is something with PL. Please let me know these things asap.

I forgot to tell you something related: I have changed [https://github.com/huggingface/transformers/blob/master/examples/research_projects/rag/finetune_rag.py#L535](url) to the below (changed **and** to **or**) . 

`if ("LOCAL_RANK" not in os.environ or os.environ["LOCAL_RANK"] == 0) or (            "NODE_RANK" not in os.environ or os.environ["NODE_RANK"] == 0        ):<!--EndFragment-->`

If use the original code with **and**, I got the bug as follows: 
```
INFO worker.py:726 -- Connecting to existing Ray cluster at address: 10.34.64.3:6379
Traceback (most recent call last):
  File "finetune_rag.py", line 790, in <module>
    main(args)
  File "finetune_rag.py", line 718, in main
    os.environ["NODE_RANK"], os.environ["LOCAL_RANK"]
  File "/opt/miniconda/envs/rag/lib/python3.8/os.py", line 675, in __getitem__
    raise KeyError(key) from None
KeyError: 'LOCAL_RANK'
```
You may benefit from the above information. Now I am going to think your new suggestions.



Actually, I think I can solve your problem. Please let me know once you have done the test (find whether the code goes inside the if condition in the initial process).

> I forgot to tell you something related: I have changed [https://github.com/huggingface/transformers/blob/master/examples/research_projects/rag/finetune_rag.py#L535](url) to the below (changed **and** to **or**) .
> 
> `if ("LOCAL_RANK" not in os.environ or os.environ["LOCAL_RANK"] == 0) or ( "NODE_RANK" not in os.environ or os.environ["NODE_RANK"] == 0 ):<!--EndFragment-->`
> 
> If use the original code with **and**, I got the bug as follows:
> 
> ```
> INFO worker.py:726 -- Connecting to existing Ray cluster at address: 10.34.64.3:6379
> Traceback (most recent call last):
>   File "finetune_rag.py", line 790, in <module>
>     main(args)
>   File "finetune_rag.py", line 718, in main
>     os.environ["NODE_RANK"], os.environ["LOCAL_RANK"]
>   File "/opt/miniconda/envs/rag/lib/python3.8/os.py", line 675, in __getitem__
>     raise KeyError(key) from None
> KeyError: 'LOCAL_RANK'
> ```
> 
> You may benefit from the above information. Now I am going to think your new suggestions.

Yes, this should give you an error. You have to use **and** operator. Because in the beginning there is no "Local_Rank" variable and it only checks the **""LOCAL_RANK" not in os. environ "** condition, prior to going into the next term with Nodes.

But if you remove the **and** operator between two conditions, it will try to check this  "os.environ["LOCAL_RANK"] == 0". I know this is bit tricky :) 

Newly: 
These are about when I run latest version of original RAG on the clusters.

When using **and** operator, it faced _KeyError: 'LOCAL_RANK'_ as befor.  
When changing  **and** operator to **or**, it faced _ValueError: Failed to look up actor with name 'retrieval_worker_0'._  as before.
When changing  **and** to **or** and add **train_params["num_nodes"] = 1**, it also faced _ValueError: Failed to look up actor with name 'retrieval_worker_0'._ 

Overall, it's the same problem as them when running END2END-RAG.

Can you find out during the initialization, the code enters the if the condition or not? 

Try  to print something after this [line](https://github.com/huggingface/transformers/blob/master/examples/research_projects/rag-end2end-retriever/finetune_rag.py#L709). I want to know whether your code fails before the DDP process.  If it doesn't go inside the if condition, when starting, it is a problem with RAY, otherwise it is something with PL. Please let me know these things asap.

When I run as follows :
```
        print("debug: LOCAL RANK in if = {}".format("LOCAL_RANK" not in os.environ or os.environ["LOCAL_RANK"] == 0))
        print("debug: NODE RANK in if = {}".format("NODE_RANK" not in os.environ or os.environ["NODE_RANK"] == 0))
        if ("LOCAL_RANK" not in os.environ or os.environ["LOCAL_RANK"] == 0) and (
            "NODE_RANK" not in os.environ or os.environ["NODE_RANK"] == 0
        ):
            print('Debug: 1 Go into successfully')
            remote_cls = ray.remote(RayRetriever)
            named_actors = [
                remote_cls.options(name="retrieval_worker_{}".format(i)).remote()
                for i in range(args.num_retrieval_workers)
            ]
            print('Debug: 2 Initially successfully')
        else:
            print("Debug: 3 in else")
            logger.info(
                "Getting named actors for NODE_RANK {}, LOCAL_RANK {}".format(
                    os.environ["NODE_RANK"], os.environ["LOCAL_RANK"]
                )
            )
            print('Debug: 4444444')
            named_actors = [ray.get_actor("retrieval_worker_{}".format(i)) for i in range(args.num_retrieval_workers)]
            print('Debug: 5555555')
```
Print like: ( I copy all as belows after accomplishing conda env)
```
2021-06-09 08:34:41,460	INFO worker.py:726 -- Connecting to existing Ray cluster at address: 10.36.112.5:6379
shzhang Debug NODE_RANK 0
shzhang Debug  LOCAL_RANK Not exist.
debug: LOCAL RANK in if = True
debug: NODE RANK in if = False
Debug: 3 in else
Traceback (most recent call last):
  File "finetune_rag.py", line 634, in <module>
    main(args)
  File "finetune_rag.py", line 561, in main
    os.environ["NODE_RANK"], os.environ["LOCAL_RANK"]
  File "/opt/miniconda/envs/rag/lib/python3.8/os.py", line 675, in __getitem__
    raise KeyError(key) from None
KeyError: 'LOCAL_RANK'
Starting the daemon thread to refresh tokens in background for process with pid = 956

```


Ok I think problem is with the if conditions.   When you run the code it
never goes inside if condition. Which means your workers doesn't get
initialized. The main issue is you already have a Node variable in
os.envron. do you get me?  If not I can explain. 
On Wed, Jun 9, 2021, 20:44 Dopaminezsy ***@***.***> wrote:

> When I run as follows :
>
>         print("debug: LOCAL RANK in if = {}".format("LOCAL_RANK" not in os.environ or os.environ["LOCAL_RANK"] == 0))
>
>         print("debug: NODE RANK in if = {}".format("NODE_RANK" not in os.environ or os.environ["NODE_RANK"] == 0))
>
>         if ("LOCAL_RANK" not in os.environ or os.environ["LOCAL_RANK"] == 0) and (
>
>             "NODE_RANK" not in os.environ or os.environ["NODE_RANK"] == 0
>
>         ):
>
>             print('Debug: 1 Go into successfully')
>
>             remote_cls = ray.remote(RayRetriever)
>
>             named_actors = [
>
>                 remote_cls.options(name="retrieval_worker_{}".format(i)).remote()
>
>                 for i in range(args.num_retrieval_workers)
>
>             ]
>
>             print('Debug: 2 Initially successfully')
>
>         else:
>
>             print("Debug: 3 in else")
>
>             logger.info(
>
>                 "Getting named actors for NODE_RANK {}, LOCAL_RANK {}".format(
>
>                     os.environ["NODE_RANK"], os.environ["LOCAL_RANK"]
>
>                 )
>
>             )
>
>             print('Debug: 4444444')
>
>             named_actors = [ray.get_actor("retrieval_worker_{}".format(i)) for i in range(args.num_retrieval_workers)]
>
>             print('Debug: 5555555')
>
>
> Print like: ( I copy all as belows after accomplishing conda env)
>
> 2021-06-09 08:34:35,151	INFO scripts.py:560 -- Local node IP: 10.36.112.5
>
> 2021-06-09 08:34:36,497	INFO services.py:1272 -- View the Ray dashboard at �[1m�[32mhttp://127.0.0.1:8265�[39m�[22m
>
> 2021-06-09 08:34:37,529	SUCC scripts.py:592 -- --------------------
>
> 2021-06-09 08:34:37,529	SUCC scripts.py:593 -- Ray runtime started.
>
> 2021-06-09 08:34:37,529	SUCC scripts.py:594 -- --------------------
>
> 2021-06-09 08:34:37,529	INFO scripts.py:596 -- Next steps
>
> 2021-06-09 08:34:37,530	INFO scripts.py:597 -- To connect to this Ray runtime from another node, run
>
> 2021-06-09 08:34:37,530	INFO scripts.py:601 --   ray start --address='10.36.112.5:6379' --redis-password='5241590000000000'
>
> 2021-06-09 08:34:37,530	INFO scripts.py:606 -- Alternatively, use the following Python code:
>
> 2021-06-09 08:34:37,530	INFO scripts.py:609 -- import ray
>
> 2021-06-09 08:34:37,530	INFO scripts.py:610 -- ray.init(address='auto', _redis_password='5241590000000000')
>
> 2021-06-09 08:34:37,530	INFO scripts.py:618 -- If connection fails, check your firewall settings and network configuration.
>
> 2021-06-09 08:34:37,530	INFO scripts.py:623 -- To terminate the Ray runtime, run
>
> 2021-06-09 08:34:37,530	INFO scripts.py:624 --   ray stop
>
> /home/t-shzhang/.local/lib/python3.8/site-packages/ray/autoscaler/_private/cli_logger.py:57: FutureWarning: Not all Ray CLI dependencies were found. In Ray 1.4+, the Ray CLI, autoscaler, and dashboard will only be usable via `pip install 'ray[default]'`. Please update your install command.
>
>   warnings.warn(
>
> 2021-06-09 08:34:41,460	INFO worker.py:726 -- Connecting to existing Ray cluster at address: 10.36.112.5:6379
>
> shzhang Debug NODE_RANK 0
>
> shzhang Debug  LOCAL_RANK Not exist.
>
> debug: LOCAL RANK in if = True
>
> debug: NODE RANK in if = False
>
> Debug: 3 in else
>
> Traceback (most recent call last):
>
>   File "finetune_rag.py", line 634, in <module>
>
>     main(args)
>
>   File "finetune_rag.py", line 561, in main
>
>     os.environ["NODE_RANK"], os.environ["LOCAL_RANK"]
>
>   File "/opt/miniconda/envs/rag/lib/python3.8/os.py", line 675, in __getitem__
>
>     raise KeyError(key) from None
>
> KeyError: 'LOCAL_RANK'
>
> Starting the daemon thread to refresh tokens in background for process with pid = 956
>
>
>
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/transformers/issues/12050#issuecomment-857508306>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AEA4FGSIFGTZSNYBG6QCW23TR4SWZANCNFSM46HEVE4A>
> .
>


I can't see the email for it's ***@***.***, so I will email you at your gmail on Linkedin.

sure.

On Wed, Jun 9, 2021 at 9:35 PM Dopaminezsy ***@***.***> wrote:

> I can't see the email for it's *@*.***, so I will email you at your gmail
> on Linkedin.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/transformers/issues/12050#issuecomment-857544425>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AEA4FGWK5AWQTW24OA62XD3TR4YUVANCNFSM46HEVE4A>
> .
>


-- 
[image: Augmented Human Lab] <http://www.ahlab.org/> [image: uni]
<https://www.auckland.ac.nz/en/abi.html>

Gayal Shamane
Ph.D. Candidate
Augmented Human Lab
Auckland Bioengineering Institute | The University of Auckland


Hey did you manage to solve ?

There seem to be something surpring!

My solvement: 
1. remove the condition after **and** operator in the below, remaining only the front condition.
 `if ("LOCAL_RANK" not in os.environ or os.environ["LOCAL_RANK"] == 0) and ( "NODE_RANK" not in os.environ or os.environ["NODE_RANK"] == 0 )`
2. restrict the version of Ray to ray==1.3.0 not ray>=1.3.0 (it will install ray==1.4.0 on the cluster).

Then it works well as belows:
```
Validating: 100%|█████████▉| 2962/2964 [22:52<00:00,  2.22it/s][A
Epoch 0:   0%|          | 2964/64358290 [23:10<8388:32:39,  2.13it/s, loss=nan, v_num=6]

Validating: 100%|█████████▉| 2963/2964 [22:52<00:00,  2.22it/s][A
Epoch 0:   0%|          | 2965/64358290 [23:11<8388:26:04,  2.13it/s, loss=nan, v_num=6]
```

 I'm going to check it for times, avoiding mistake.

Nice.  Actually  the problem was your cluster  has a default node.
Now you are simply checking whether the DDP process has started or not.

BTW what happens when you are using latest RAY ? 

On Wed, Jun 9, 2021, 23:45 Dopaminezsy ***@***.***> wrote:

> There seem to be something surpring!
>
> My solvement:
>
>    1. remove the condition after *and* operator in the below, remaining
>    only the front condition.
>    if ("LOCAL_RANK" not in os.environ or os.environ["LOCAL_RANK"] == 0)
>    and ( "NODE_RANK" not in os.environ or os.environ["NODE_RANK"] == 0 )
>    2. restrict the version of Ray to ray==1.3.0 not ray>=1.3.0 (it will
>    install ray==1.4.0 on the cluster).
>
> Then it works well as belows:
>
> Validating: 100%|█████████▉| 2962/2964 [22:52<00:00,  2.22it/s]�[A
>
> Epoch 0:   0%|          | 2964/64358290 [23:10<8388:32:39,  2.13it/s, loss=nan, v_num=6]
>
>
>
> Validating: 100%|█████████▉| 2963/2964 [22:52<00:00,  2.22it/s]�[A
>
> Epoch 0:   0%|          | 2965/64358290 [23:11<8388:26:04,  2.13it/s, loss=nan, v_num=6]
>
>
> I'm going to check it for times, avoiding mistake.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/transformers/issues/12050#issuecomment-857625392>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AEA4FGRLAAQ5BSVF4DUFQY3TR5H4HANCNFSM46HEVE4A>
> .
>


This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

[updated]

@shamanez 

Sorry to share my reproduced results later.

I got my result EM=40.31 in end2end way, just following the same setting of [rag-end2end-retriever](https://github.com/huggingface/transformers/tree/master/examples/research_projects/rag-end2end-retriever).


That's nice to hear. Thanks for letting me know.

On Thu, Jul 22, 2021, 22:28 Dopaminezsy ***@***.***> wrote:

> [updated]
>
> @shamanez <https://github.com/shamanez>
>
> Sorry to share my reproduced results later.
>
> I got my result EM=40.31 in end2end way, just following the same setting
> of rag-end2end-retriever
> <https://github.com/huggingface/transformers/tree/master/examples/research_projects/rag-end2end-retriever>
> .
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/transformers/issues/12050#issuecomment-884809427>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AEA4FGSXHGF4MBT3VBWVICDTY7XFPANCNFSM46HEVE4A>
> .
>