Repository: nerfstudio-project/nerfstudio

Issue:
Scene is not rendering
Hi this is a really cool work.

I was testing this repo and was using the default scene in the quickstart page. Everything loaded correctly and I can see it training, however, when i open the viewer, I can only see the camera poses with the images. I am running this code on a AWS instance (ubuntu 20.04) terminal, with port 7007 opened. I can see all the stats updating, but the scene is not rendering. Thanks!
![Screen Shot 2022-10-12 at 5 07 58 PM](https://user-images.githubusercontent.com/22141463/195448807-36b1d08c-3615-4105-997c-0fb3dac12e61.png)



Discussion:
Same problem here; tested with blender dataset as well.
@eldercrow can you describe your machine setup as well? It’s something with webrtc not connecting.
Same problem
Same problem for me. clean env but I am using cuda 11.7
Does refreshing after the images first show up, then waiting ~10 seconds, work for any of you? Curious if it’s due to the initialization sequence or because webrtc is failing with your machine setup.
Everyone who is running into this issue, can you report the following details (we are having trouble replicating this bug on the dev machines).

1. OS version
2. nerfstudio version
3. Browser type
4. Local or remote compute
5. What is your internet setup, (ie are you on a corp network, or personal plan)

Other tips:
+ Make sure to open link reported in browser, don't use an old one
+ Only have one viewer window open
+ Try refreshing
1. Ubuntu 20.04.4 LTS (GNU/Linux 5.15.0-46-generic x86_64)
2. 0.1.5
3. Chrome
4. Remote
5. Remote on corporate site

Tried tips withouth success
@Blueslikehail Can you try the latest version of the code to see if that fixes the issue. If it doesn't a few followup questions,

+ Are you are able to see the dataset images, but not the NeRF rendering?
+ Is this a new issue, or have previous versions of nerfstudio also not work?

I'm concerned that it may be caused by webrtc being blocked by some corporate network restrictions.
Hi Matthew,Just tried the last version and I have the same problem. Tomorrow I’ll try from within the corporate network to see if it works. Which previous version do you suggest trying should the problem persist?Thanks,WalterOn Oct 17, 2022, at 19:05, Matthew Tancik ***@***.***> wrote:﻿
@Blueslikehail Can you try the latest version of the code to see if that fixes the issue. If it doesn't a few followup questions,

Are you are able to see the dataset images, but not the NeRF rendering?
Is this a new issue, or have previous versions of nerfstudio also not work?

I'm concerned that it may be caused by webrtc being blocked by some corporate network restrictions.

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: ***@***.***>
Ubuntu 20.04 (x86_64)
0.1.5.1
Edge
Remote
Remote on corporate site

I've tried the localhost too.
It gave me this address:
(https://viewer.nerf.studio/versions/22-10-13-0/?websocket_url=ws://localhost:7007)
which did not work too.
Just checking, everyone in this thread is able to see the dataset images, just not the render (same as in the image in the original post)? 
Ubuntu 18.04.4 LTS (GNU/Linux 5.4.0-126-generic)
0.1.5
Chrome
Remote
Remote on corporate site

Yes, I can see the  the dataset images, but no render scene.
Ubuntu 18.04.4 LTS 
Latest version (as of today)
NVIDIA V100
Edge
Remote
Remote on corporate site

Yes, I can see the the dataset images, but no render scene.
Hi All,I am running the same setup on corporate site and I can see the render now.WalterOn Oct 18, 2022, at 08:15, Hyunjoon Lee ***@***.***> wrote:﻿
Ubuntu 18.04.4 LTS
Latest version (as of today)
NVIDIA V100
Edge
Remote
Remote on corporate site
Yes, I can see the the dataset images, but no render scene.

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: ***@***.***>
@Blueslikehail Could you please let me know your setting? Has anything changed?
Hi,No changes, it seems I just have to be within the corporate network. WalterOn Oct 18, 2022, at 09:41, Hyunjoon Lee ***@***.***> wrote:﻿
@Blueslikehail Could you please let me know your setting? Has anything changed?

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: ***@***.***>
Same problem. Sometimes it works sometimes it doesn't.

- OS version: Windows 11 22H1
- nerfstudio version: 0.1.5.1
- Browser type: Firefox 105.0.3 (64-Bit)
- Local
- Internet: shouldn't matter, it's local

I noticed, though, that the change for the viewer to work is higher if I open the webpage before training starts.
same problem with my custom datasets especially:

- ubuntu 20.04
- nerfstudio version: 0.1.5.1
- browser: firefox 94.0
- local
Sorry this is proving to be such an issue. It seems that webrtc initial connection is sometimes flaky. We we continue to try to debug this, unfortunately it isn't something we are experiencing on our machines.

(side note, we have been using chrome, it might be worth seeing if it works better on chrome)
I like to offer help, yet I am not able to reproduce it when I want it.

Btw, I updated my Firefox and not experiencing the issue anymore. In case it helps anyone out there. Also haven't experienced this issue on Chrome(windows 10) as well.
Hi @tancik,
Thanks for your work!

Have the same issue when running `ns-train` on the docker image `nvidia/cuda:11.8.0-cudnn8-devel-ubuntu22.04` and ssh port forwarding. (In the container I've installed a conda environment according to the tutorial using torch with cuda 11.3 and nerfstudio 0.1.6)

![Screenshot from 2022-10-20 14-08-16](https://user-images.githubusercontent.com/1412303/196945055-36241798-d1d5-4881-a066-dab06f5f01ad.png)
Refreshing after a few training steps and then waiting or moving the camera does not help.
The RTC connection seems to be fine and Server + Render are shown as connected.
I've tested on Google Chrome and Firefox and in none of my tests the render is shown only the images appear.


In the terminal everything seems to be normal and the `Vis Rays / Sec` column (which I assume is responsible for the viewer) also shows entries.
![Screenshot from 2022-10-20 14-07-34](https://user-images.githubusercontent.com/1412303/196945584-11e964fa-550b-4f4b-b2d2-7df43937857a.png)

> Ubuntu 18.04.4 LTS (GNU/Linux 5.4.0-126-generic) 0.1.5 Chrome Remote Remote on corporate site
> 
> Yes, I can see the the dataset images, but no render scene.

Update:
I move the camera and wait for a few seconds, the render scene appears.
I have the same issue when using port forwarding on the remote machine. When connecting locally, the scene is rendered correctly. Moving the camera and waiting do not helps. 
I had to reboot the remote Ubuntu machine I was working on.

OS version: OSX 12.5.1
nerfstudio version: 
Browser type: Chrome
Local or remote compute: remote
Internet setup: Personal plan
Configurations that don't work for me:

OS version: OSX 12.6 (local OS, remote Ubuntu 20.04)
nerfstudio version: latest (as of today, also didn't work with 0.1.5)
Browser type: Chrome (and Firefox)
Local or remote compute: remote
Internet setup: Corporate internet

OS version: Ubuntu 20.04 (local OS, remote Ubuntu 20.04)
nerfstudio version: latest (as of today, also didn't work with 0.1.5)
Browser type: Firefox
Local or remote compute: remote
Internet setup: Corporate internet

Does work:
Remotely connected to the same machine with NoMachine using Firefox
Same issue here, but, weirdly, another dataset I trained loaded the view correcly.

Win 10
nerfstudio v0.1.8
Chrome 107.0.5304.88 (64 bits)
Run Local,
Viewing from localhost, so, no ISP issue.
Same issue
I was working on Windows locally and it was fine.

But on the remote server and forward to windows, there is no render.

I just got this issue again as well. 

Is it a cooncidence we have it at the same time? 👀 
Same issue 😭
It used to work (few weeks? ago), but now it doesn't.

Win 11 + WSL (Ubuntu 20.04.5)
nerfstudio v0.1.8
Chrome, Firefox, Edge
Run Local,
Viewing from localhost
> I just got this issue again as well.
> 
> Is it a cooncidence we have it at the same time? 👀

Now it makes sense. I checked locally still no render. I guess its something wrong with the render itself. Not our browser or setting issue. It used to be render connected, but now the render is disconnected all the time.
It's not a coincidence, mine is also down. It is because the TURN webRTC server we are using (https://www.metered.ca/tools/openrelay/) is currently down / overloaded. Hopefully it gets restored soon.
Wait: You need some kind of webservice even if you are running locally?
When you are viewing the localhost link and the connection is unstable, are you going through some remote server?!
> Sorry this is proving to be such an issue. It seems that webrtc initial connection is sometimes flaky. We we continue to try to debug this, unfortunately it isn't something we are experiencing on our machines.

Just ran into this as well, "Server connected" (seeing the image cameras in the viewer), but "Render Disconnected" (no rendered view). I have `ns-train` running on a remote GPU node and use an SSH tunnel to make remote port 7007 available locally, and 
then have the viewer connect to localhost:7007. Looking at some of the replies above it seems metered.ca is used to get around any firewall issues for the separate webRTC connection, in the browser console I see a `WebRTC: ICE failed`:

![image](https://user-images.githubusercontent.com/1408287/201545973-15434c8b-0818-4cea-bcbf-c737b588a13d.png)

Not sure what's going on there, but I suspect that one of the issues of why rendering doesn't work in my (and other) cases. Attaching logs ([aboutWebrtc.zip](https://github.com/nerfstudio-project/nerfstudio/files/9998293/aboutWebrtc.zip)) from Firefox's `about:webrtc`, maybe they provide some insight.

Client OS: Arch Linux, Firefox 106.0 (same issue in Chromium 106.0)
Server OS: Red Hat, nerfstudio 0.1.9
Connecting from home to remote HPC system through cable modem and SSHl tunnel



Thanks for the log! If anyone else has logs like this, please share. Also interested in similar logs in chrome.
I ran into this problem, but only if running nerfstudio inside a docker container. Works fine over ssh on a remote machine if not running in docker, but if I run in docker, I get the same problem as above.

The command I am running is
`ns-train nerfacto --vis viewer --viewer.websocket-port 8844`

And yes I am forwarding the port out of Docker ( `docker run --gpus all --ipc host -it --rm -v /home/me:/home/me -p 8844:8844 nerfstudio`)

Here is what it looks like when I run outside of Docker:
![good](https://user-images.githubusercontent.com/12813713/202664804-2221ad00-1ad4-424c-87f7-53b412462959.png)

And here is what I get if I run within Docker (same command)
![bad](https://user-images.githubusercontent.com/12813713/202665110-d5a5b6c5-d797-4f1f-8735-87ee40291399.png)

Here is the Dockerfile I used to construct the nerfstudio docker container - perhaps the error can be reproduced by others + fixed? Note the build command I used at the end of the Dockerfile.

```
FROM nvidia/cuda:11.6.0-devel-ubuntu20.04

# Prevent stop building ubuntu at time zone selection.  
ENV DEBIAN_FRONTEND=noninteractive

# Prepare and empty machine for building
RUN apt-get update && apt-get install -y \
    git \
    cmake \
    build-essential \
    libboost-program-options-dev \
    libboost-filesystem-dev \
    libboost-graph-dev \
    libboost-system-dev \
    libboost-test-dev \
    libeigen3-dev \
    libsuitesparse-dev \
    libfreeimage-dev \
    libmetis-dev \
    libgoogle-glog-dev \
    libgflags-dev \
    libglew-dev \
    qtbase5-dev \
    libqt5opengl5-dev \
    libcgal-dev \
    libmetis-dev

# Build and install ceres solver
RUN apt-get -y install \
    libatlas-base-dev \
    libsuitesparse-dev
ARG CERES_SOLVER_VERSION=2.1.0
RUN git clone https://github.com/ceres-solver/ceres-solver.git --branch ${CERES_SOLVER_VERSION}
RUN cd ceres-solver && \
	mkdir build && \
	cd build && \
	cmake .. -DBUILD_TESTING=OFF -DBUILD_EXAMPLES=OFF && \
	make -j4 && \
	make install

# Build and install COLMAP

# Note: This Dockerfile has been tested using COLMAP pre-release 3.7.
# Later versions of COLMAP (which will be automatically cloned as default) may
# have problems using the environment described thus far. If you encounter
# problems and want to install the tested release, then uncomment the branch
# specification in the line below
RUN git clone https://github.com/colmap/colmap.git #--branch 3.7

RUN cd colmap && \
	git checkout dev && \
	mkdir build && \
	cd build && \
	cmake .. && \
	make -j4 && \
	make install

RUN apt-get install -y wget

RUN INSTALL_PATH=~/anaconda \
    && wget http://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh \
    && bash Miniconda3-latest* -fbp $INSTALL_PATH

ENV PATH=/root/anaconda/bin:$PATH

RUN conda create --name nerfstudio -y python=3.8

SHELL ["conda", "run", "-n", "nerfstudio", "/bin/bash", "-c"]

RUN pip install --upgrade pip
RUN pip install --no-cache torch==1.12.1+cu116 torchvision==0.13.1+cu116 -f https://download.pytorch.org/whl/torch_stable.html

ENV PATH=/usr/local/cuda/bin:$PATH
ENV CUDA_HOME="/usr/local/cuda-11.6/"
ARG TCNN_CUDA_ARCHITECTURES="86"

RUN pip install git+https://github.com/NVlabs/tiny-cuda-nn/#subdirectory=bindings/torch
RUN pip install nerfstudio

RUN . /root/.bashrc && conda init bash

RUN echo "conda activate nerfstudio" >> ~/.bashrc

RUN apt-get install -y ffmpeg

# docker build -t nerfstudio --build-arg TCNN_CUDA_ARCHITECTURES=$(python -c "import torch; ma, m = torch.cuda.get_device_capability(); print(ma * 10 + m, end='')") .
```
One weird thing (might be a separate issue) is that even though I get correct rendering when running both training and web viewer on the same system, I still get a red `Render Disconnected` in the web UI.
I had the same issue when running the code on an EC2 instance: viewer either didn't show the nerf render, or showed it with a very low framerate.

**I got a 100% working solution** by using XRDP on the instance, launching a browser on the instance with the viewer URL, and connecting via RDP (using Microsoft Remote Desktop because I'm on Mac, but I guess there are linux solutions that work too) through a ssh tunnel.

Setup the ssh tunnel to the remote instance (`[::1]` worked for me, `localhost` didn't, because the XRDP server listened only on the IPv6 address, but `localhost` may work for you) 
```
ssh -N -L '12345:[::1]:3389' username@remote_host_name
```
then connect to localhost:12345 with MS Remote Desktop


Resolving patch:
diff --git a/nerfstudio/viewer/server/viewer_utils.py b/nerfstudio/viewer/server/viewer_utils.py
--- a/nerfstudio/viewer/server/viewer_utils.py
+++ b/nerfstudio/viewer/server/viewer_utils.py
@@ -28,7 +28,12 @@
 
 import numpy as np
 import torch
-from aiortc import RTCPeerConnection, RTCSessionDescription
+from aiortc import (
+    RTCConfiguration,
+    RTCIceServer,
+    RTCPeerConnection,
+    RTCSessionDescription,
+)
 from cryptography.utils import CryptographyDeprecationWarning
 from rich.console import Console
 
@@ -503,7 +508,25 @@ async def send_webrtc_answer(self, data):
         # returns the description to for WebRTC to the specific websocket connection
         offer = RTCSessionDescription(data["sdp"], data["type"])
 
-        pc = RTCPeerConnection()
+        pc = RTCPeerConnection(
+            configuration=RTCConfiguration(
+                iceServers=[
+                    RTCIceServer(urls="stun:stun.l.google.com:19302"),
+                    RTCIceServer(urls="stun:openrelay.metered.ca:80"),
+                    RTCIceServer(
+                        urls="turn:openrelay.metered.ca:80", username="openrelayproject", credential="openrelayproject"
+                    ),
+                    RTCIceServer(
+                        urls="turn:openrelay.metered.ca:443", username="openrelayproject", credential="openrelayproject"
+                    ),
+                    RTCIceServer(
+                        urls="turn:openrelay.metered.ca:443?transport=tcp",
+                        username="openrelayproject",
+                        credential="openrelayproject",
+                    ),
+                ]
+            )
+        )
         self.pcs.add(pc)
 
         video = SingleFrameStreamTrack()


Test patch: