Package: linux-aio
[english]|[한국어 (korean)]linux_aio: Python wrapper forLinux Kernel AIOPython wrapper module that uses Linux Kernel AIO directlyWhat is Linux Kernel AIO?Linux IO Models tableIn summary, it allows non-blocking and asynchronous use of blocking IO operations such asread(2)andwrite(2).Related documentsLinux Asynchronous I/OLinux Kernel AIO Design NotesHow to use the Linux AIO feature(in C)It is different fromPOSIX AIOThe POSIX AIO APIs have theaio_prefix, but the Linux Kernel AIO has theio_prefix.There is already a POSIX AIO API for asynchronous I/O, but Linux implements it in glibc, a user-space library, which is supposed to use multi-threading internally.
So, as you can see from the experiment below, it's much worse than using the blocking IO API.Implementation & StructurePackagelinux_aioImplemented based onlinux_aio_bindpackage which is low-level binding of Linux kernel AIO.Unlikelinux_aio_bind, it can be used without knowledge ofctypesExamples can be found in the code in thetest directory.ExampleExamples can be found in the code in thetest directory.Notes & LimitsObviously available only on LinuxBecause it is a wrapper, it brings the constraints of Linux.It can not be used for files used as a kernel interface. (e.g.cgroup)Sometimes it works as Blocking.There are some things that have been solved through development after posting.Some features are being added because they are still under development.There are also some features that are not supported when the Linux version is lowYou need to checkLinux man pages (4.16)and its related API documentationEvaluationExperiment script(requires python 3.7)SetupDistribution: Ubuntu Server 16.04.5 LTSLinux: 4.19.0CPU: 2-way Intel(R) Xeon(R) CPU E5-2683 v4 @ 2.10GHzMEM: total 64GBStorage: SK hynix SC300B SATA 512GBPython: 3.7.2 (Ubuntu ppa)Attempts to read a total of1000times in1msintervals.The file size varies from1KBto100KB, but it is small.Experiment with increasing the number of files read at the same timeBecause we have experimented with high-performance server, there may be larger performance differences when testing on a typical desktop.Comparison targetaiofiles- Uses Thread poolaiofile- Uses POSIX AIOlibaio- Useslibaiopython built-in open()It is not a perfectly fair comparison.aiofilesandaiofileare libraries that supportasyncio. Sinceopen()is blocking, there is a disadvantage that you can not do any other work while IO is going on.libaioandlinux_aioare non-blocking, but must be polled.ResultsIt may differ from environment to environment.RuntimeUnit: second# of files161224aiofiles1.6813.3185.3549.768aiofile1.5431.9582.4933.737libaio1.3111.3441.3621.423open()1.2521.3221.3751.481linux_aio1.3051.3271.3531.431Threads# of files161224aiofiles321321321321aiofile381526libaio1111open()1111linux_aio1111MemoryPhysical memory (Virtual memory)# of files161224aiofiles21MB (22.6GB)21MB (22.6GB)21MB (22.6GB)21MB (22.6GB)aiofile17MB (258MB)17MB (654MB)17MB (1080MB)18MB (1949MB)libaio17MB (76MB)17MB (76MB)17MB (76MB)17MB (76MB)open()17MB (76MB)17MB (76MB)17MB (76MB)17MB (76MB)linux_aio17MB (76MB)17MB (76MB)17MB (76MB)17MB (76MB)CPU Utilization# of files161224aiofiles42.8%85.0%102.2%113.2%aiofile31.4%52.4%67.0%84.0%libaio14.0%16.0%17.2%20.6%open()13.4%17.6%21.0%26.2%linux_aio13.0%15.0%16.0%21.0%