Package: pysendfile
Quick linksHome pageMailing listBlogWhat’s newAboutsendfile(2)is a system call which
provides a “zero-copy” way of copying data from one file descriptor to another
(a socket). The phrase “zero-copy” refers to the fact that all of the copying
of data between the two descriptors is done entirely by the kernel, with no
copying of data into userspace buffers. This is particularly useful when
sending a file over a socket (e.g. FTP).
The normal way of sending a file over a socket involves reading data from the
file into a userspace buffer, then write that buffer to the socket viasend()orsendall():# how a file is tipically sentimportsocketfile=open("somefile","rb")sock=socket.socket()sock.connect(("127.0.0.1",8021))whileTrue:chunk=file.read(65536)ifnotchunk:break# EOFsock.sendall(chunk)This copying of the data twice (once into the userland buffer, and once out
from that userland buffer) imposes some performance and resource penalties.sendfile(2)syscall avoids these
penalties by avoiding any use of userland buffers; it also results in a single
system call (and thus only one context switch), rather than the series ofread(2)/write(2)system calls (each system call
requiring a context switch) used internally for the data copying.importsocketfromsendfileimportsendfilefile=open("somefile","rb")blocksize=os.path.getsize("somefile")sock=socket.socket()sock.connect(("127.0.0.1",8021))offset=0whileTrue:sent=sendfile(sock.fileno(),file.fileno(),offset,blocksize)ifsent==0:break# EOFoffset+=sentA simple benchmarkThisbenchmark scriptimplements the two examples above and compares plain socket.send() and
sendfile() performances in terms of CPU time spent and bytes transmitted per
second resulting in sendfile() being about2.5x faster. These are the
results I get on my Linux 2.6.38 box, AMD dual-core 1.6 GHz:send()CPU time28.84 usec/passtransfer rate359.38 MB/secsendfile()CPU time11.28 usec/passtransfer rate860.88 MB/secWhen do you want to use it?Basically any application sending files over the network can take advantage of
sendfile(2). HTTP and FTP servers are a typical example.proftpdandvsftpdare known to use it, so ispyftpdlib.API documentationsendfile module provides a single function: sendfile().sendfile.sendfile(out, in, offset, nbytes,header="",trailer="",flags=0)Copynbytesbytes from file descriptorin(a regular file) to file
descriptorout(a socket) starting atoffset. Return the number of
bytes just being sent. When the end of file is reached return 0.
On Linux, ifoffsetis given asNone, the bytes are read from the current
position ofinand the position ofinis updated.headersandtrailersare strings that are written before and after the
data frominis written. In cross platform applications their usage is
discouraged
(send()orsendall()can be used instead). On Solaris, _out_ may be the file descriptor of a
regular file or the file descriptor of a socket. On all other platforms,outmust be the file descriptor of an open socket.flagsargument is only supported on FreeBSD.sendfile.SF_NODISKIOsendfile.SF_MNOWAITsendfile.SF_SYNCParameters for the _flags_ argument, if the implementation supports it. They
are available on FreeBSD platforms. SeeFreeBSD’s man sendfile(2).Differences with send()sendfile(2) works with regular (mmap-like) files only (e.g. you can’t use it
with aStringIOobject).Also, it must be clear that the file can only be sent “as is” (e.g. you
can’t modify the content while transmitting).
There might be problems with non regular filesystems such as NFS,
SMBFS/Samba and CIFS. For this please refer toproftpd documentation.OSErroris raised instead ofsocket.error.
The accompainingerror codeshave the same meaning though: EAGAIN, EWOULDBLOCK, EBUSY meaning you are
supposed to retry, ECONNRESET, ENOTCONN, ESHUTDOWN, ECONNABORTED in case of
disconnection. Some examples:benchmark script,test suite,pyftpdlib wrapper.Supported platformsThis module works with Python versions from2.5to3.4. The supported platforms are:LinuxMac OSXFreeBSDDragon Fly BSDSun OSAIX(not properly tested)SupportFeel free to mail me atg.rodola [AT] gmail [DOT] comor post on the the
mailing list:http://groups.google.com/group/py-sendfile.StatusAs of now the code includes a solidtest suiteand its ready for production use.
It’s been included inpyftpdlibproject and used in production environments for almost a year now without any
problem being reported so far.Authorspysendfile was originally written byBen Woolleyincluding Linux, FreeBSD and
DragonFly BSD support. Later onNiklas Edmundssontook over maintenance and
added AIX support. After a couple of years of project stagnationGiampaolo Rodola’took over
maintenance and rewrote it from scratch adding support for:Python 3non-blocking socketslarge filesupportMac OSXSun OSFreeBSD flag argumentmultiple threads (release GIL)a simple benchmark suiteunit testsdocumentation