# An Experimental Implementation of a Resilient Graphic Rendering Cluster
## 3 Experiments--Some Ways to Make Our Software Resilient
### Measured Performance Bottleneck  
The test suite box.pov from Persistence of Vision Ray demo scene database in minimal 80 \(\times\) 60 pixel resolution is rendered at a well-pleasing 1390 fps (frames per second), the speed at which the GRC-client consumes 100% of processor time, or an entire CPU core, which as a proof of concept that works by almost two orders of magnitude better than requirements for real-time rendering. As of writing, real-time animation is not a goal in the GRC project; this speed is only used to benchmark and measure bottlenecks in performance. At a normal resolution of 1024 \(\times\) 768, box.pov is rendered at typically 35 to 40 fps. In this example, the GRC-client is no longer a bottleneck, but it occupies about 15% to 20% of CPU core load. By experience we can tell that in this case the bottleneck is Persistence of Vision Ray itself with its rendering speed limit. We were unable to speed up Persistence of Vision Ray, having only provided handshaking by two orders of magnitude faster than it is required, adding little or insignificant overhead to bare Persistence of Vision Ray rendering CPU consumption. This lower overhead has increased scalability as a consequence [15], as the client could serve more rendering servers without choking its CPU core on the job (the multi-threaded implementation of the client is on the to-do list as well). We were happy to learn that the GRC-client and GRC-server presented no performance bottleneck, or present one only in extreme conditions, such as 1390 fps rendering of a very small and simple benchmark scene. Real-life animations have much larger complexity than box.pov, meaning that the CPU load the GRC-client will impose on the web-front host will be on average, in our experience, from 0.7% to 2.0%. Later improvements have raised benchmark performance to up to more than 3500 fps for a box.pov benchmark simple scene example, which is sufficient to demonstrate client-server handshaking of the web-front server and the rendering farm plus the overheads of the protocol and of the Persistence of Vision Ray process startup. As in the cluster, as a rule some Persistence of Vision Ray process will always run and have preloaded binary executable, and the overhead is mostly reduced to that of fork() and exec() POSIX system calls with the few copy-on-write page faults. Using faster mmap() system call instead of open() and read() maps the model directly into memory, and resulting frames are also mapped before sending onto the network socket. Where available, Linux-specific system call sendfile() is used [16]. Because the copying is done within kernel, this is more efficient than read() and write(), which would require copying of image data to and from the user space. This call is also safer than mmap(), which can sometimes fail with bus error (signal SIGBUS) after which there are very few options in signal recovery but to close the server process (but the whole rendering process will not be interrupted as the animation frame will simply be returned to the unrendered frame pool). Additional zero-copy Linux system calls are also being considered, like splice() and vmsplice(), to avoid further user space copying overhead in the GRC-client and GRC-server. (The use of these Linux-specific high performance system calls requires conditional compilation of GRC-server and isolation of Linux-specific code in #ifdef... #endif clauses.) This increased efficiency of 3500 fps has no effect in real life rendering examples but removes the bottleneck from the GRC-client's and GRC-server's frame handling routines completely, as the system limit on open file handles will be reached first unless there has been a change done by the administrator on the resource NOFILE [2] (the system limit on the number of open files). As current implementation uses one connection per CPU core [17], the number of total cores in the cluster is limited with this resource limit parameter.