brockn/hdfs-nfs-proxy — Issue #6: Readahead cache to improve performance
Author: nickb937
Created: 2012-02-27 15:14:27+00:00
State: open

On a 1000Mbit network, I timed a copy of a 1.5GB file to local disk at 45 minutes. The main reason I can find for this is that NFS read/write requests are limited to 64Kbytes, which translates to much slower Hadoop requests of 64Kbytes.

It may be a large performance win if the nfs-proxy answered all read-requests from a local disk based cache. Have the cache populated by nfs-proxy upon receiving a read-request whereby it grabs 64MBytes or 128MBytes from HDFS in one operation, then answers NFS requests from the data stored on local disk. I think this could work well enough even if initially restricted to files opened read-only.