Package: aiofile
Real asynchronous file operations with asyncio support.StatusDevelopment - StableFeaturesSince version 2.0.0 usingcaio, which contains linuxlibaioand two
thread-based implementations (c-based and pure-python).AIOFile has no internal pointer. You should passoffsetandchunk_sizefor each operation or use helpers (Reader or Writer).
The simples way is to useasync_openfor creating object with
file-like interface.For Linux using implementation based onlibaio.For POSIX (MacOS X and optional Linux) using implementation
based onthreadpool.Otherwise using pure-python thread-based implementation.Implementation chooses automatically depending on system compatibility.LimitationsLinux native AIO implementation is not able to open special files.
Asynchronous operations against special fs like/proc//sys/are not
supported by the kernel. It’s not aaiofile`s or `caioissue.
In this cases, you might switch to thread-based implementations
(seetroubleshootingsection).
However, when used on supported file systems, the linux implementation has a
smaller overhead and is preferred but it’s not a silver bullet.Code examplesAll code examples requires python 3.6+.High-level APIasync_openhelperHelper mimics python file-like objects, it returns file-like
objects with similar but async methods.Supported methods:async def read(length =-1)- reading chunk from file, when length is-1, will be reading file to the end.async def write(data)- writing chunk to filedef seek(offset)- setting file pointer positiondef tell()- returns current file pointer positionasync defreadline(size=-1,newline="\n")- read chunks until
newline or EOF. Since version 3.7.0__aiter__returnsLineReader.This method is suboptimal for small lines because it doesn’t reuse read buffer.
When you want to read file by lines please avoid usingasync_openuseLineReaderinstead.def __aiter__()->LineReader- iterator over lines.def iter_chunked(chunk_size: int = 32768)->Reader- iterator over
chunks..fileproperty contains AIOFile objectBasic example:importasynciofrompathlibimportPathfromtempfileimportgettempdirfromaiofileimportasync_opentmp_filename=Path(gettempdir())/"hello.txt"asyncdefmain():asyncwithasync_open(tmp_filename,'w+')asafp:awaitafp.write("Hello ")awaitafp.write("world")afp.seek(0)print(awaitafp.read())awaitafp.write("Hello from\nasync world")print(awaitafp.readline())print(awaitafp.readline())loop=asyncio.get_event_loop()loop.run_until_complete(main())Example without context manager:importasyncioimportatexitimportosfromtempfileimportmktempfromaiofileimportasync_openTMP_NAME=mktemp()atexit.register(os.unlink,TMP_NAME)asyncdefmain():afp=awaitasync_open(TMP_NAME,"w")awaitafp.write("Hello")awaitafp.close()asyncio.run(main())assertopen(TMP_NAME,"r").read()=="Hello"Concatenate example program (cat):importasyncioimportsysfromargparseimportArgumentParserfrompathlibimportPathfromaiofileimportasync_openparser=ArgumentParser(description="Read files line by line using asynchronous io API")parser.add_argument("file_name",nargs="+",type=Path)asyncdefmain(arguments):forsrcinarguments.file_name:asyncwithasync_open(src,"r")asafp:asyncforlineinafp:sys.stdout.write(line)asyncio.run(main(parser.parse_args()))Copy file example program (cp):importasynciofromargparseimportArgumentParserfrompathlibimportPathfromaiofileimportasync_openparser=ArgumentParser(description="Copying files using asynchronous io API")parser.add_argument("source",type=Path)parser.add_argument("dest",type=Path)parser.add_argument("--chunk-size",type=int,default=65535)asyncdefmain(arguments):asyncwithasync_open(arguments.source,"rb")assrc,\async_open(arguments.dest,"wb")asdest:asyncforchunkinsrc.iter_chunked(arguments.chunk_size):awaitdest.write(chunk)asyncio.run(main(parser.parse_args()))Example with opening already open file pointer:importasynciofromtypingimportIO,Anyfromaiofileimportasync_openasyncdefmain(fp:IO[Any]):asyncwithasync_open(fp)asafp:awaitafp.write("Hello from\nasync world")print(awaitafp.readline())withopen("test.txt","w+")asfp:asyncio.run(main(fp))Linux native aio doesn’t support reading and writing special files
(e.g. procfs/sysfs/unix pipes/etc.), so you can perform operations with
these files using compatible context objects.importasynciofromaiofileimportasync_openfromcaioimportthread_aio_asynciofromcontextlibimportAsyncExitStackasyncdefmain():asyncwithAsyncExitStack()asstack:# Custom context should be reusedctx=awaitstack.enter_async_context(thread_aio_asyncio.AsyncioContext())# Open special file with custom contextsrc=awaitstack.enter_async_context(async_open("/proc/cpuinfo","r",context=ctx))# Open regular file with default contextdest=awaitstack.enter_async_context(async_open("/tmp/cpuinfo","w"))# Copying file content line by lineasyncforlineinsrc:awaitdest.write(line)asyncio.run(main())Low-level APITheAIOFileclass is a low-level interface for asynchronous file operations, and the read and write methods accept
anoffset=0in bytes at which the operation will be performed.This allows you to do many independent IO operations on an once open file without moving the virtual carriage.For example, you may make 10 concurrent HTTP requests by specifying theRangeheader, and asynchronously write
one opened file, while the offsets must either be calculated manually, or use 10 instances ofWriterwith
specified initial offsets.In order to provide sequential reading and writing, there isWriter,ReaderandLineReader. Keep in mindasync_openis not the same as AIOFile, it provides a similar interface for file operations, it simulates methods
like read or write as it is implemented in the built-in open.importasynciofromaiofileimportAIOFileasyncdefmain():asyncwithAIOFile("hello.txt",'w+')asafp:payload="Hello world\n"awaitasyncio.gather(*[afp.write(payload,offset=i*len(payload))foriinrange(10)])awaitafp.fsync()assertawaitafp.read(len(payload)*10)==payload*10asyncio.run(main())The Low-level API in fact is just little bit sugaredcaioAPI.importasynciofromaiofileimportAIOFileasyncdefmain():asyncwithAIOFile("/tmp/hello.txt",'w+')asafp:awaitafp.write("Hello ")awaitafp.write("world",offset=7)awaitafp.fsync()print(awaitafp.read())loop=asyncio.get_event_loop()loop.run_until_complete(main())ReaderandWriterWhen you want to read or write file linearly following example
might be helpful.importasynciofromaiofileimportAIOFile,Reader,Writerasyncdefmain():asyncwithAIOFile("/tmp/hello.txt",'w+')asafp:writer=Writer(afp)reader=Reader(afp,chunk_size=8)awaitwriter("Hello")awaitwriter(" ")awaitwriter("World")awaitafp.fsync()asyncforchunkinreader:print(chunk)loop=asyncio.get_event_loop()loop.run_until_complete(main())LineReader- read file line by lineLineReader is a helper that is very effective when you want to read a file
linearly and line by line.It contains a buffer and will read the fragments of the file chunk by
chunk into the buffer, where it will try to find lines.The default chunk size is 4KB.importasynciofromaiofileimportAIOFile,LineReader,Writerasyncdefmain():asyncwithAIOFile("/tmp/hello.txt",'w+')asafp:writer=Writer(afp)awaitwriter("Hello")awaitwriter(" ")awaitwriter("World")awaitwriter("\n")awaitwriter("\n")awaitwriter("From async world")awaitafp.fsync()asyncforlineinLineReader(afp):print(line)loop=asyncio.get_event_loop()loop.run_until_complete(main())When you want to read file by lines please avoid to useasync_openuseLineReaderinstead.More examplesUseful examples withaiofileAsync CSV Dict ReaderimportasyncioimportiofromcsvimportDictReaderfromaiofileimportAIOFile,LineReaderclassAsyncDictReader:def__init__(self,afp,**kwargs):self.buffer=io.BytesIO()self.file_reader=LineReader(afp,line_sep=kwargs.pop('line_sep','\n'),chunk_size=kwargs.pop('chunk_size',4096),offset=kwargs.pop('offset',0),)self.reader=DictReader(io.TextIOWrapper(self.buffer,encoding=kwargs.pop('encoding','utf-8'),errors=kwargs.pop('errors','replace'),),**kwargs,)self.line_num=0def__aiter__(self):returnselfasyncdef__anext__(self):ifself.line_num==0:header=awaitself.file_reader.readline()self.buffer.write(header)line=awaitself.file_reader.readline()ifnotline:raiseStopAsyncIterationself.buffer.write(line)self.buffer.seek(0)try:result=next(self.reader)exceptStopIterationase:raiseStopAsyncIterationfromeself.buffer.seek(0)self.buffer.truncate(0)self.line_num=self.reader.line_numreturnresultasyncdefmain():asyncwithAIOFile('sample.csv','rb')asafp:asyncforiteminAsyncDictReader(afp):print(item)asyncio.run(main())TroubleshootingThe caiolinuximplementation works normal for modern linux kernel versions
and file systems. So you may have problems specific for your environment.
It’s not a bug and might be resolved some ways:Upgrade the kernelUse compatible file systemsUse threads based or pure python implementation.The caio since version 0.7.0 contains some ways to do this.1. In runtime use the environment variableCAIO_IMPLwith
possible values:linux- use native linux kernels aio mechanismthread- use thread based implementation written in Cpython- use pure python implementation2. Filedefault_implementationlocated near__init__.pyin caio
installation path. It’s useful for distros package maintainers. This file
might contains comments (lines starts with#symbol) and the first line
should be one oflinuxthreadorpython.You might manually manage contexts:importasynciofromaiofileimportasync_openfromcaioimportlinux_aio_asyncio,thread_aio_asyncioasyncdefmain():linux_ctx=linux_aio_asyncio.AsyncioContext()threads_ctx=thread_aio_asyncio.AsyncioContext()asyncwithasync_open("/tmp/test.txt","w",context=linux_ctx)asafp:awaitafp.write("Hello")asyncwithasync_open("/tmp/test.txt","r",context=threads_ctx)asafp:print(awaitafp.read())asyncio.run(main())