Repository: ray-project/kuberay

Issue #805: [Bug] Job Sample YAML `ray_v1alpha1_rayjob.yaml` fails with empty `node-ip-address` `$MY_POD_IP`
Created: 2022-12-07T01:37:31Z
Search before asking

 I searched the issues and found no similar issues.

KubeRay Component
ray-operator
What happened + What you expected to happen
Following the documentation at https://ray-project.github.io/kuberay/guidance/rayjob/ using Kuberay master using kind, the worker pod starts to crashloop:
Every 1.0s: kubectl get pod                     Archits-MBP.local.meter: Tue Dec  6 15:47:13 2022

NAME                                                      READY   STATUS             RESTARTS
  AGE
rayjob-sample-raycluster-plpll-head-wk7pr                 1/1     Running            0
  16s
rayjob-sample-raycluster-plpll-worker-small-group-djcc6   0/1     CrashLoopBackOff   3 (41s ago)
  16s

Inspecting the logs, we see the traceback:
❯ kubectl logs rayjob-sample-raycluster-plpll-worker-small-group-djcc6
Traceback (most recent call last):
 File "/home/ray/anaconda3/bin/ray", line 8, in <module>
   sys.exit(main())
 File "/home/ray/anaconda3/lib/python3.7/site-packages/ray/scripts/scripts.py", line 2596, in main
   return cli()
 File "/home/ray/anaconda3/lib/python3.7/site-packages/click/core.py", line 1128, in __call__
   return self.main(*args, **kwargs)
 File "/home/ray/anaconda3/lib/python3.7/site-packages/click/core.py", line 1053, in main
   rv = self.invoke(ctx)
 File "/home/ray/anaconda3/lib/python3.7/site-packages/click/core.py", line 1659, in invoke
   return _process_result(sub_ctx.command.invoke(sub_ctx))
 File "/home/ray/anaconda3/lib/python3.7/site-packages/click/core.py", line 1395, in invoke
   return ctx.invoke(self.callback, **ctx.params)
 File "/home/ray/anaconda3/lib/python3.7/site-packages/click/core.py", line 754, in invoke
   return __callback(*args, **kwargs)
 File "/home/ray/anaconda3/lib/python3.7/site-packages/ray/autoscaler/_private/cli_logger.py", line 852, in wrapper
   return f(*args, **kwargs)
 File "/home/ray/anaconda3/lib/python3.7/site-packages/ray/scripts/scripts.py", line 588, in start
   node_ip_address = services.resolve_ip_for_localhost(node_ip_address)
 File "/home/ray/anaconda3/lib/python3.7/site-packages/ray/_private/services.py", line 531, in resolve_ip_for_localhost
   raise ValueError(f"Malformed address: {address}")
ValueError: Malformed address:

Reproduction script
kind create cluster and then follow the docs page at  https://ray-project.github.io/kuberay/guidance/rayjob/
Anything else
From tracing through the Ray code, the error is happening because an empty node-ip-address was passed to ray start.  There is a field node-ip-address: $MY_POD_IP in the sample Job yaml, so this environment variable must not have been set.  I assume the Ray operator is supposed to set this environment variable, but I'm not sure where it gets set.
Are you willing to submit a PR?

 Yes I am willing to submit a PR!

Duplicate issue #923: [Bug] rayjob worker could not start and the worker report `ValueError: Malformed address: `
Created: 2023-02-22T07:48:00Z
Search before asking

 I searched the issues and found no similar issues.

KubeRay Component
ray-operator
What happened + What you expected to happen
Hi team, i follow the guide https://ray-project.github.io/kuberay/ to create a rayjob use the example yaml as the doc,
but the worker could not start, and the rayjob keep in created status:
tatus:
  Dashboard URL:          rayjob-sample-raycluster-fwj5h-head-svc.default.svc.cluster.local:8265
  Job Deployment Status:  Initializing
  Job Id:                 rayjob-sample-knr5g
  Ray Cluster Name:       rayjob-sample-raycluster-fwj5h
  Ray Cluster Status:
    Head:
Events:
  Type    Reason   Age   From               Message
  ----    ------   ----  ----               -------
  Normal  Created  11m   rayjob-controller  Created cluster rayjob-sample-raycluster-fwj5h

the below is the worker log
Traceback (most recent call last):
  File "/home/ray/anaconda3/bin/ray", line 8, in <module>
    sys.exit(main())
  File "/home/ray/anaconda3/lib/python3.7/site-packages/ray/scripts/scripts.py", line 2596, in main
    return cli()
  File "/home/ray/anaconda3/lib/python3.7/site-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/home/ray/anaconda3/lib/python3.7/site-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/home/ray/anaconda3/lib/python3.7/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/ray/anaconda3/lib/python3.7/site-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/ray/anaconda3/lib/python3.7/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/home/ray/anaconda3/lib/python3.7/site-packages/ray/autoscaler/_private/cli_logger.py", line 852, in wrapper
    return f(*args, **kwargs)
  File "/home/ray/anaconda3/lib/python3.7/site-packages/ray/scripts/scripts.py", line 588, in start
    node_ip_address = services.resolve_ip_for_localhost(node_ip_address)
  File "/home/ray/anaconda3/lib/python3.7/site-packages/ray/_private/services.py", line 531, in resolve_ip_for_localhost
    raise ValueError(f"Malformed address: {address}")
ValueError: Malformed address: 

Reproduction script
k apply -f ray_v1alpha1_rayjob.yaml
Anything else
this is very strange the code around services.resolve_ip_for_localhost checked it is not none but seem the serveice.py go none
Are you willing to submit a PR?

 Yes I am willing to submit a PR!

Closing comment (2023-04-11T21:53:19Z): This issue seems to be a duplicate of #805. Close this issue. Feel free to reopen this issue if you still observe this issue.