Package: repono
reponoA Python library to make storing files intuitive.:warning: Although there are extensive tests within this project for Python 3.6, 3.7, 3.8 and 3.9, it is a young project and there may be bugs. Be sure and test thoroughly prior to use in a production environment.It is primarily intended to deal with file uploads to a static files directory or an object service likeAWS S3orLinode.
Files can by stored synchronously (forWSGIservers
likeDjango,FlaskorPyramid) or asynchronously
(forASGIservers likeStarlette,FastAPIorDjango Channels).Supports multiple storage services simultaneously or even the same service with multiple configurations.Upload filters are designed to be intuitive to create and a feware includedby default.Table of ContentsreponoTable of ContentsIntroductionInstallationThe StoreConfigure the StoreFoldersAdding FiltersClassesStorageContainerStorageHandlerFilterFileItemExceptionsHandlersLocalFileHandlerAsyncLocalFileHandlerS3HandlerDummyHandlerAsyncDummyHandlerFiltersRandomizeFilenameValidateExtensionTestingIntroductionInstallationThe library is available through thePython Package Indexand can be installed with pip.pipinstallreponoDifferent handlers have additional library requirements that can be optionally installed. For instance, theasync local file handlerrequirements can be installed using:pip install "repono[aio_file]"The extras are:aio_file- requirements for async local file handling.s3- requirements for storing to an AWS S3 bucket.The StoreInteraction with the library is primarily accomplished through a globalstoreobject. Any Python file can access this global object by importing it.fromreponoimportstorestore.finalized# == FalseIf you are uncomfortable by the existence of a global store, don't be! You can make a new instance of the store using theStorageContainerclass.my_store = StorageContainer(), the use that in your program.The store can hold multiple configurations that are accessed through its indices:store['portraits'].finalized# == Falsestore['more']['and more'].finalized# == FalsePORTRAIT_STORE=store['portraits']# global variable for later use!The store and any of its other sub-configurations can be saved and referencedpriorto setting up any configuration.
This allows the store or any other sub-configuration to be imported anywhere and stored within global variables as
needed.Trying to use the store to save files prior to providing it a handler will result in an error.store.save_data(filename='file.txt',data=b'spamity spam')# ReponoConfigError: No handler provided for storeSo it's time to give it ahandler.Configure the StoreAlthough any file can import thestore, it will not actually be usable until it is provided some kind of configuration.
You do this by giving the store a handler. This library includes a few kinds ofhandlerswith different configuration setups. For testing it's easiest to use theDummyHandler:fromfilestore.handlersimportDummmyHandlerstore.handler=DummyHandler()store['portraits']=DummyHandler()The DummyHandler doesn't need any other configuration and stores any saved files in memory.In your app, you would would normally perform this configuration after all files have been imported but before the app starts - for instance in an initialization step where the app configuration is read. Different frameworks (Pyramid/Django/Starlette) will have different mechanisms for this.Now you can try to save some data into the store:store.save_data(filename='file.txt',data=b'spamity spam')# 'file.txt' - the name of the file that was savedstore.exists(filename='file.txt')# TrueHandlers and filters might adjust the filename in the process of storing the file. The return value gives you feedback of the actual name of the file that was saved to the store. This might be useful for storing in a database for later reference.You can set and reset the handler several times. When you want to lock in the configuration and prevent further changes, you need to finalize the store.
When you do so, all handlers and filters will validate their configuration and if any handlers are missing a configuration error will be thrown.  Any attempts to set a handler after this step will throw an error. This ensures that the store is properly set up once and only once.store.finalize_config()store.handler=DummyHandler()# ReponoConfigError: Setting store.handler: store already finalized!If using an ASGI server, you may need to instead use anasync startup taskthat contains:awaitstore.async_finalize_config()The finalization step also will validate any handler configuration.
For instance, thelocal file handlerensures its configured directory
exists, theasync file handlerensures the proper libraries are installed and the S3 handler verifies the credentials by saving and deleting a dummy file in the bucket.If the store or one of the sub-stores are not intended to be used, they must explicitly have their handler set toNone. For instance, suppose
you want to use two different configurations, but don't want to use the store's base config. You can do:store['portraits'].handler=DummyHandler()store['backgrounds'].handler=DummyHandler()# Indicate that we don't want anybody using the base configurations (no `store.save_data()`)store.handler=None# Finalize the configuration to prevent further changesstore.finalize_config()FoldersThe save/exists/delete methods can also be used on any sub-folder of thestore.folder=store/'one_folder'/'another_folder'folder.save_data(b'contents','file.txt')Folders are late-binding and can thus be saved into a variable prior to the handler configuration.If desired, a Folder can be used as a handler too:store.handler=DummyHandler()store['portraits'].handler=store/'portraits'Adding FiltersFilters allow mutating a file to be stored to a Handler. They are called in the order defined and a few filters are provided by this library.For instance, it's often best not to store filenames provided by random Internet uploads. Although this library does scrub the filename, it's not as fool-proof as ignoring the provided filename and using a random string with a consistent length. TheRandomizeFilenamefilter does that.fromrepono.filtersimportRandomizeFilenamestore.handler=DummyHandler(filters=[RandomizeFilename()])store.save_data(filename='ignored name.txt',data=b'contents')'5d29f7e1-50c0-4dc6-a466-b2ae042e8ac0.txt'For more on filters, see theFilterclass definition.ClassesStorageContainerThis is the class for the globalstoreobject.Methods:handler- Gets the configured handler or raises an exception of no handler has been provided. Set this property to set the handler.sync_handler- Gets the configured handler as a sync-only handler. Raises an exception if nohandlerhas been set.async_handler- Gets the configured handler as an async-only handler. Raises an exception if nohandlerhas been set or if the configured handler can't be used asynchronously.finalize_config()- Walk through all configured objects and check to ensure they have a valid configuration. Lock theStorageContainerto prevent any further configuration changes. Will raise areponoConfigErrorif there's a configuration problem.async_finalize_config()- awaitable version of the above call. Necessary for ASGI servers.finalized-Trueif the config has been finalized,Falseotherwise.do_not_use-Trueif thehandlerhas been set toNone,Falseotherwise.name- String name of how this configuration is accessed.store['a'].name == "['a']".[*]- Get a sub-configuration object. Raises areponoConfigErrorif the configuration is finalized and this configuration'shandlerhasn't been set./ 'name'- Obtain aFolderobject with the same save/exist/delete methods as this object which write to the named sub-folder.Once the handler is set, the store object can be used as aStorageHandlerobject.StorageHandler(StorageHandlerBaseorAsyncStorageHandlerBase)All handlers inherit fromStorageHandlerBase.The async version of the Handler can be used for either synchronous or asynchronous operations. TheStorageHandlerBaseby itself can only be used for synchronous operations and anyasync_*method calls will throw an error. To make a new custom handler, start with thehandler template.:warning:Ensure your forms include the attribute enctype=”multipart/form-data”or your uploaded files will be empty.Short exampleandmore detail.Parameters:base_url- Optional string - The URL prefix for any saved file. For example:'https://eppx.com/static/'. If not set, theget_urlmethod will only return the path string, not the full URL.path- Optional list or string - The path within the store (and URL) for any saved file. For example:['folder', 'subfolder']filters- Optional list ofFiltersto apply, in order, when saving any file through this handler. For example:[RandomizeFilename()]:warning: All time related methods return the specified time in the timezone of the handler.Methods:get_url(filename: str)- Return the full URL of the given filename. If thebase_urlparameter isn't set, will only return the path string instead of the full URL.sanitize_filename(filename: str)- Return the string stripped of dangerous characters.Synchronous methods:exists(filename: str)-Trueif the given file exists in the store, false otherwise.size(filename: str)-intThe size in bytes of the given file.get_accessed_time(filename: str)-datetimeThe time of last access for the given file.get_created_time(filename: str)-datetimeThe time of creation for the given file. This returns different times based on operating system. Clickherefor more detail.get_modified_time(filename: str)-datetimeThe time of last modification for the given file.delete(filename: str)- Deletes the given file from the store.save_file(filename: str, data: BinaryIO)- Save the binary IO object to the given file.save_data(filename: str, data: bytes)- Save the binary data to the given file.save_field(field: cgi.FieldStorage)- Save the given field storage object.Asynchronous methods: (all will throw aFilestorageConfigErrorif the handler doesn't support async operations.)async_exists(filename: str)- Awaitable versionasync_size(filename: str)- Awaitable versionasync_get_accessed_time(filename: str)- Awaitable versionasync_get_created_time(filename: str)- Awaitable versionasync_get_modified_time(filename: str)- Awaitable versionasync_delete(filename: str)- Awaitable versionasync_save_file(filename: str, data: BinaryIO)- Awaitable versionasync_save_data(filename: str, data: binary)- Awaitable versionasync_save_field(field: cgi.FieldStorage)- Awaitable versionAbstract Methods to be overridden when sub-classing:_validate()- Check to ensure the provided configuration is correct. Can be an async method or return aFutureobject.Synchronous methods: (All get passed aFileItemobject)_exists(item: FileItem)- ReturnsTrue/Falseto indicate if the item exists in the storage container._size(filename: str)- Returns the size in bytes of the given file._get_accessed_time(filename: str)Returns the time of last access for the given file._get_created_time(filename: str)Returns the time of creation for the given file._get_modified_time(filename: str)Returns the time of last modification for the given file._delete(item: FileItem)- Remove the item from the storage container._save(item: FileItem)- Save the item to the storage container and return the name of the file saved.Asynchronous methods:async _async_exists(item: FileItem)- async version, returnsTrueorFalse.async_size(filename: str)- async version.async_get_accessed_time(filename: str)async version, returns the time of last access for the given file.async_get_created_time(filename: str)async version, returns the time of creation for the given file.async_get_modified_time(filename: str)async version, returns the time of last modification for the given file.async _async_delete(item: FileItem)- async version.async _async_save(item: FileItem)- async version, returns the stored filename.FilterTheFilterBaseis used as a base class for any Filters. These are not intended to be used directly, but to be passed as an optional list to a Handler through thefiltersparameter. To make a new custom filter, start with thefilter template.Properties:async_ok-True/Falseto indicate if this Filter is OK to be used asynchronously.Methods:call(item: FileItem)- Returns the filteredFileItem.async_call(item: FileItem)- Awaitable version ofcall. If the filter can't be used asynchronously, will raise areponoConfigError.validate()- Checks to ensure the Filter is configured correctly. Might return an Awaitable.Abstract Methods to be overridden when sub-classing:_apply(item: FileItem)- Returns the filteredFileItem. It can optionally be an async method that will be awaited on. Can raise an exception if the item fails the filter._validate()- Check to ensure the provided configuration is correct. Can be an async method or return aFutureobject.FileItemThis is the basic item that's passed through Filters to the Handlers. It is based on a NamedTuple and is therefore immutable.Parameters, which are also Properties:filename- Stringpath- Optional tuple of the path under which thefilenamecan be accessed or stored. Example:('folder', 'subfolder').data- OptionalBinaryIOof the contents to store. Example:BytesIO(b'contents').media_type- Optional string describing themedia (or MIME) typecontained within the data. If not provided, the type will beguessedas needed based on the filename's extension.Properties:url_path- Relative path and filename for use in a URL. Example:'folder/subfolder/file.txt'fs_path- Relative path and filename for use in the local filesystem. Example when running on Windows:'folder\subfolder\file.txt'.has_data-Trueif data is populated.content_type- String indicating themedia_typeto be used in HTTP headers as needed.Methods:copy(**kwargs)- Create a copy of this object, overriding any specific parameter. Example:item.copy(filename='new_name.txt'). This is very useful in a Filter that changes the filename or other properties.Context Manager:The FileItem can be used as a context manager, where it will modify the read/seek methods of the underlying object to fit the requested stream as necessary - either sync or async.with FileItem as f- Returns an object (f) that behaves as an open file.f.read()andf.seek()methods are supported.async with FileItem as f- Returns an object (f) that behaves as an async open file.await f.read()andawait f.seek()methods are supported.ExceptionsAll are importable via theexceptionssub-package. For example:fromrepono.exceptionsimportReponoErrorReponoError - Base class for any exceptions raised by this library.FileNotAllowed - The provided file is not allowed, either through aFilteror from aHandler.FileExtensionNotAllowed - The provided file with the given extension is not allowed, either through aFilteror from aHandler.ReponoConfigError - There was some problem with the configuration.HandlersAll handlers are subclasses of theStorageHandlerclass. These can be imported via thehandlerssub-package. For example:fromrepono.handlersimportLocalFileHandlerstore.handler=LocalFileHandler(base_path='/home/www/uploads`)LocalFileHandlerStore files on the local file system using synchronous methods.Async not OK.Parameters:base_path- Where to store the files on the local filesystemauto_make_dir- Automatically create the directory as needed.AsyncLocalFileHandlerStore files on the local file system using asynchronous methods.Async OK.:warning:Requires theaiofileslibrary, which will be installed withpip install repono['aio_file']Parameters:base_path- Where to store the files on the local filesystemauto_make_dir(defualt:False)- Automatically create the directory as needed.allow_sync_methods(default:True) - WhenFalse, all synchronous calls throw aRuntimeError. Might be helpful in preventing accidentally using the syncsave/exists/deletemethods, which would block the async loop too.S3HandlerStore files to an S3 bucket. This handler works for synchronous and asynchronous calls.Async OK.:warning:Requires theaioboto3library, which will be installed withpip install repono['s3']:warning:Requires appropriate AWS permissions to the S3 bucket.Parameters:bucket_name- required - AWS bucket to store the files in.acl(default:'public-read') - Access-Control to apply to newly saved files. Other interesting options are'private','authenticate-read', and'bucket-owner-read'. Seethe S3 documentationfor more information.connect_timeout(default:5) - Seconds to wait for a connection event before timeout.num_retries(default:5- How many times the library will attempt to connect before failing.read_timeout(default:10) Seconds to wait for a read event before timeout.keepalive_timeout(default:12) - Send a packet every few seconds to keep active connections open.host_url- When usingnon-AWS S3 service(likeLinode), use this url to connect. (Example:'https://us-east-1.linodeobjects.com')region_name- Overrides any region_name defined in theAWS configuration fileor  theAWS_DEFAULT_REGIONenvironment variable. Required if using AWS S3 and the value is not already set elsewhere.addressing_style- Overrides any S3.addressing_style set in theAWS configuration file.allow_sync_methods(default:True) - WhenFalse, all synchronous calls throw aRuntimeError. Might be helpful in preventing accidentally using the syncsave/exists/deletemethods, which would block the async loop too.Permissions can be configured in three different ways. They can be stored in environment variables, then can be stored in a particular AWS file, or they can be passed in directly.See theboto3 credentials documentationfor details on how to configure the required keys.:warning:Do not hard-code secret information in your source code!If you wish to directly provide the connection information to this handler, use the following optional parameters:aws_access_key_id- or use theAWS_ACCESS_KEY_IDenvironment variable.aws_secret_access_key- or use theAWS_SECRET_ACCESS_KEYenvironment variable.aws_session_token- Unnecessary when using the two previous options, but here for completeness. Note thatsession tokenshave a maximum age of 36 hours. Could instead use theAWS_SESSION_TOKENenvironment variable.profile_name- Which profile to use within ashared credentials file.DummyHandlerHandler used to test the file store - keeps any stored files in memory.Async not OK - will fail for any async call. For an async version use theAsyncDummyHandler.Accepts no parameters.Properties:files- a dictionary of all saved files. The key is the path/filename string and the value is aFileItemobject.validated-Trueif the handler was validated (which happens while finalizing the config).last_save- The lastFileItemsaved.last_delete- The lastFileItemdeleted.Methods:assert_exists(filename: str, path: Tuple[str, ...]- Asserts that the provided filename and path have been saved.assert_size(self, filename: str, path: Tuple[str, ...], size: int)- Assert that given file size is equal to the anticipated size.assert_get_accessed_time(self, filename: str, path: Tuple[str, ...], date: datetime)- Assert that given file access time is equal to the anticipated time.assert_get_created_time(self, filename: str, path: Tuple[str, ...], date: datetime)- Assert that given file creation time is equal to the anticipated time.assert_get_modified_time(self, filename: str, path: Tuple[str, ...], date: datetime)- Assert that given file modification time is equal to the anticipated time.assert_file_contains(filename: str, path: Tuple[str, ...], data: bytes)- Asserts that the saved file contains the given data.AsyncDummyHandlerIdentical to theDummyHandler, but can be used asynchronously.FiltersAll are importable via thefilterssub-package. For example:fromrepono.filterimportRandomizeFilenamestore.handler=DummyHandler(filters=[RandomizeFilename()])RandomizeFilenameFilter to randomize the filename. It keeps any extension within the filename, but replaces the name with a random string.Async OK.Parameters:name_generatorOptional method that takes the filename (without extension) and returns a new random name. When left off, this Filter uses theuuid4method.ValidateExtensionDon't accept any filename that does not have one of the indicated extensions. Raises aFileExtensionNotAllowedfor a disallowed extension.Async OK.Parameters:extensions- List of exceptions to allow through this filter. Example:['jpg', 'png']TestingTheDummyHandlerorAsyncDummyHandlerare great tools for testing your application. To keep your tests isolated, you can create a newstoreobject and configure it for each test as needed.fromreponoimportStorageContainerfromrepono.handlersimportAsyncDummyHandlerdeftest_store():store=StorageContainer()dummy_handler=AsyncDummyHandler()store.handler=dummy_handler# Do whatever should trigger your app to store a filedummy_handler.assert_exists('name.txt',('folder','subfolder'))If you need to write several tests and want to check the result, it's probably best to create a couple of fixtures and allowpytestto inject them as needed.Withintests/conftest.py:importpytestfromreponoimportStorageContainerfromrepono.handlersimportAsyncDummyHandler@pytest.fixturedefdummy_handler():returnAsyncDummyHandler()@pytest.fixturedefstore(dummy_handler):store=StorageContainer()store.handler=dummy_handlerstore.finalize_config()returnstoreWithintests/test_myapp.pydeftest_store(store,dummy_handler):# The configured dummy handler and store were auto-created and passed in here# Do whatever should trigger your app to store a filedummy_handler.assert_exists('name.txt',('folder','subfolder'))