Package: fernet-files
Fernet Files - encryption/decryption of files with cryptography.fernet, but with file-like methodsFernet encryption requires all data to be encrypted or decrypted at once. This is memory intensive, and it is slow if you only want to read part of a file. Fernet Files provides a simple interface that breaks data up into chunks when encrypting and decrypting them to lower memory usage. Data is only encrypted when it's necessary to do so.You may treat the class similar to a file: it hasread,write,seekandclosemethods. It can also be context managed, so you can close it using awithstatement.ContentsExample usageRequirementsInstallationBenchmarkingDocumentation for module usersDocumentation for module developersExample usagefromfernet_filesimportFernetFilekey=FernetFile.generate_key()# Keep thiswithFernetFile(key,"filename.bin")asf:# Use like a normal filef.write(b'123456789')# Returns 9f.seek(4)# Returns 4f.read(3)# Returns b'567'...# If you check the file after closing (leaving the with statement)# the contents will be encrypted and unreadable without using the module# If you use the same key, you can then read the data againwithFernetFile(key,"filename.bin")asf:f.read()# Returns b'123456789'Note: The default chunksize is 64KiB. This means the minimum output file size is 64KiB. If you are encrypting a small amount of data, I recommend you lower the chunksize. However, only do this if necessary as this will damage performance.Requirementscryptography <= 42.0.2, >= 36.0.2Python 3.10 or greater (3.10, 3.11 and 3.12 tested)custom_fernet.py is based oncryptography 41.0.4and is tested up to 42.0.2. Future versions might break this module. If this has happened, create an issue in this repository.Installationpip install fernet_filesBenchmarkingSignificant results:
When encrypting a 4GiB file, vanilla Fernet:Took 106 seconds to encryptUsed 28.6GiB of memory to encryptTook 54 seconds to decrypt the data enough that the first byte of unencrypted data could be readWhen encrypting the same 4GiB file, Fernet Files with the default chunksize:Took 50 seconds to encryptUsed 331KiB of memory to encryptTook less than 100ms to decrypt the data enough that the first byte of unencrypted data could be readFor more information, seeBENCHMARKING.md.Documentation for module usersContentsfernet_files.FernetFilefernet_files.FernetFile.readfernet_files.FernetFile.writefernet_files.FernetFile.seekfernet_files.FernetFile.closefernet_files.FernetFile.generate_keyfernet_files.FernetFile.closedfernet_files.FernetFile.writeablefernet_files.META_SIZEfernet_files.DEFAULT_CHUNKSIZEfernet_files.custom_fernet.FernetNoBase64classfernet_files.FernetFile(self, key, file, chunksize=65536)Parameters:key- A key (recommended) or afernet_files.custom_fernet.FernetNoBase64objectA key must be 32 random bytes. Get usingfernet_files.FernetFile.generate_key()and store somewhere secureAlternatively, pass in afernet_files.custom_fernet.FernetNoBase64objectfile- Accepts a filename as a string, or a file-like object. If passing in a file-like object, it would be opened in binary mode.chunksize- The size of chunks in bytes.Bigger chunks use more memory and take longer to read or write, but smaller chunks can be very slow when trying to read/write in large quantities.Bigger chunks apply padding so a very large chunksize will create a large file. Every chunk has its own metadata so a very small chunk size will create a large file.Defaults to 64KiB (65536 bytes).methodfernet_files.FernetFile.read(self, size=-1)Reads the number of bytes specified and returns them.Parameters:size- Positive integer. If -1 or not specified then read to the end of the file.methodfernet_files.FernetFile.write(self, b)Writes the given bytes. Returns the number of bytes written.Parameters:b- The bytes to be written.methodfernet_files.FernetFile.seek(self, offset, whence=os.SEEK_SET)Can be called as:seek(self, offset, whence)seek(self, offset, whence=whence)Moves through the file by the specified number of bytes. "whence" determines what this is relative to. Returns your new absolute position as an integer.Parameters:offset- Integer. Move this number of bytes relative to whence.whence- Ignored if using a BytesIO object. Accepted values are:os.SEEK_SETor0- relative to the start of the streamos.SEEK_CURor1- relative to the current stream positionos.SEEK_ENDor2- relative to the end of the stream (use negative offset)methodfernet_files.FernetFile.close(self)Writes all outstanding data closes the file. ReturnsNoneunless the file is aBytesIOobject, in which case it returns the object without closing it.static methodfernet_files.FernetFile.generate_key()Static method used to generate a key. Acts as a pointer tocustom_fernet.FernetNoBase64.generate_key().boolfernet_files.FernetFile.closedBoolean attribute representing whether the file is closed or not. True means the file is closed, False means the file is open. It is highly recommended that you do not modify this, and use theclosemethod instead.boolfernet_files.FernetFile.writeableBoolean attribute representing whether the file can be written to or not. True if you can write to the file, False if you can't. Will only be False if you passed in a read-only file. It is highly recommended that you do not modify this.Miscintfernet_files.META_SIZEIt is highly recommended you don't modify this. Defaults to 8. META_SIZE represented as $M$ in formulae.The size of a file's metadata in bytes is $2M$. The first number is a little-endian unsigned $(8M)$-bit integer, representing how many chunks are in the file. The second number is a little-endian unsigned $(8M)$-bit integer, representing the size of the last chunk's padding.This simultaneously limits both chunksize and the number of chunks a file can have:A chunk can have a max size of $2^{8M}-1$ bytes (default 18,446,744,073,709,551,615)A file can have a max $2^{8M}-1$ chunks (default 18,446,744,073,709,551,615)You can change this value in order to bypass these limitations for future-proofing, however, the value you use must be consistent when reading and writing to the same file. Therefore, I recommend you don't change it unless you absolutely have to, for compatibility reasons.intfernet_files.DEFAULT_CHUNKSIZEThe chunksize that is used by default, currently 4096 bytes.classfernet_files.custom_fernet.FernetNoBase64(self, key)cryptography.fernet.Fernetwithout any base64 encoding or decoding. Seecustom_fernet.pyfor more info.Documentation for module developersContentsfernet_files.FernetFilefernet_files.FernetFile.__chunkfernet_files.FernetFile.__filefernet_files.FernetFile.__last_chunkfernet_files.FernetFile.__last_chunk_paddingfernet_files.FernetFile.__data_chunksizefernet_files.FernetFile.__chunksizefernet_files.FernetFile.__chunk_modifiedfernet_files.FernetFile._pos_pointerfernet_files.FernetFile.__pos_pointerfernet_files.FernetFile._chunk_pointerfernet_files.FernetFile.__chunk_pointerfernet_files.FernetFile.__goto_current_chunkfernet_files.FernetFile.__get_file_sizefernet_files.FernetFile.__read_chunkfernet_files.FernetFile.__write_chunkfernet_files.FernetFile.__enter__fernet_files.FernetFile.__exit__fernet_files.FernetFile.__del__fernet_files.FernetFile.__fernetclassfernet_files.FernetFileBytesIOfernet_files.FernetFile.__chunkA BytesIO object that stores the contents of the current chunk in memory. When data is written to a chunk, it is this data in memory that is manipulated. The data is then only written to a file when__write_chunkis called.(RawIOBase or BufferedIOBase or BytesIO)fernet_files.FernetFile.__fileThe file object used for reading and writing. If a filename is provided then this is opened in "wb+" mode.intfernet_files.FernetFile.__last_chunkThe chunk number of the last chunk in the file. Chunks are numbered sequentially, starting from 0.intfernet_files.FernetFile.__last_chunk_paddingThe last chunk is padded with null bytes to fill the size of the chunk. This integer stores the size of the padding in bytes.intfernet_files.FernetFile.__data_chunksizeThe amount of data in a chunk in bytes.intfernet_files.FernetFile.__chunksizeThe size in bytes that chunks take up once they're written to disk. This is calculated with the following formula, where c is chunksize:True chunksize = $c + 73 - (c \mod{16})$This formula calculates the size of a Fernet token, based on theFernet specification.boolfernet_files.FernetFile.__chunk_modifiedBoolean attribute representing whether the data stored inself.__chunkhas been modified relative to the data stored within theself.__file. True if the chunk has been modified, False if it hasn't.property intfernet_files.FernetFile._pos_pointerStores the Fernet file's current position in the chunk in bytes. The getter returnsself.__pos_pointer. The setter ensures that $0\leq$ _pos_pointer $<$ chunksize. If it isn't, then it wraps the value round by adding or subtracting the chunksize, modifying the chunk pointer to compensate.intfernet_files.FernetFile.__pos_pointerStores the value forself._pos_pointer.property intfernet_files.FernetFile._chunk_pointerStores the Fernet file's current chunk number. The getter returnsself.__chunk_pointer. The setter modifies this value. Before it switching chunks it checks if the current chunk has been modified and writes it if it has. After switching chunks, we read the new chunk into memory.intfernet_files.FernetFile.__chunk_pointerStores the value forself._chunk_pointer.methodfernet_files.FernetFile.__goto_current_chunk(self)Moves our position inself.__fileto the location represented by the chunk pointer, taking into account the metadata at the start of the file. Calculated as follows: take the number of the chunk you're currently on, multiply by the size of chunks when they're written to disk. Take the META_SIZE, multiply that by 2 and add it to the number you had before.methodfernet_files.FernetFile.__get__file_size(self)Calculate the size of the data contained within the file in bytes using the file's metadata. This is the size of the data, not the size of what is written to disk. Calculated as follows: take the number of the last chunk and add 1 to get the total number of chunks (because counting starts at 0). Multiply this by the chunksize. Finally, subtract the size of the padding used on the last chunk.methodfernet_files.FernetFile.__read_chunk(self)Reads and decrypts the current chunk, turns it into a BytesIO object, stores that object inself.__chunkand returns it. If the chunk has been modified, it is already loaded into memory so no file operations are done. Also responsible for removing padding if the chunk being read is the last chunk.methodfernet_files.FernetFile.__write_chunk(self)Encrypts and writes the chunk, and setsself.__chunk_modifiedto False. Also responsible for applying padding and modifying the metadata at the start of the file if this is the last chunk.methodfernet_files.FernetFile.__enter__(self)Returns self to allow context management.methodfernet_files.FernetFile.__exit__(self, exc_type, exc_value, exc_traceback)Callsself.closeand returnsNone.methodfernet_files.FernetFile.__del__(self)Callsself.closeand returnsNone.custom_fernet.FernetNoBase64fernet_files.FernetFile.__fernetFernetNoBase64 object created from the key provided. Used for encryption and decryption.