Package: datajoint-cleaner
DataJoint-CleanerDataJoint-Cleaner is a tool used to clean externalDataJointstores.How It WorksThe user provides information pointing to a particular external table and external store combination. Using this information DataJoint-Cleaner will delete all objects in the external store that have no corresponding entry in the external table, thus freeing storage space.Important ConsiderationsThe creation of a backup is highly recommended before using DataJoint-Cleaner to avoid a potential loss of data due to an user error or a bug.DataJoint-Cleaner should not be used to clean external stores that are used by multiple database servers or multiple storage protocols (e.g. file & s3). If done so it could potentially delete objects that are still referenced in the database.InstallationRecommended installation methodTo avoid messing up the system Python environment, the most recommended way to install DataJoint-Cleaner is viapipx:pipxinstalldatajoint-cleanerOther installation methodsInstall DataJoint-Cleaner into user site withpip:pipinstall--userdatajoint-cleanerConfigurationDataJoint-Cleaner will look for a TOML file calleddatajoint-cleaner.tomlin the current working directory (by default) to configure itself. The configuration file must have two top-level tables calleddatabase_serversandstorage_serversand an array of tables calledcleaning_runs.Specifying Database ServersDatabase servers are specified in the top-leveldatabase_serverstable. Each key in the table corresponds to a distinct database server. The value of each key must be a table that contains the following keys:host,userandpassword.The values of thehost,userandpasswordkeys correspond to the host name of the database server, the name of a user present on said server and the password of said user, respectively.Example:[database_servers.my_db_server]host="192.156.3.65"user="me"password="mypassword"Specifying Storage ServersStorage servers are specified in a sub-table of thestorage_serverstable based on their kind. Currently only MinIO servers are supported which are specified in theminiosub table. The keys necessary to specify a MinIO server areendpoint,access_key,secret_keyandsecure.The values of these keys correspond to the endpoint of the MinIO server, your access and private key and whether a secure connection should be established or not, respectively.Example:[storage_servers.minio.my_minio_server]endpoint="192.543.5.61"access_key="my_access_key"secret_key="my_secret_key"secure=trueSpecifying Cleaning RunsIndividual cleaning runs are specified in the top-level array of tables calledcleaning_runs. Each table in the array corresponds to a distinct cleaning run and must have the following keys:database_server: Name of a database server specified in thedatabase_serverstableschema: Name of a schema on said database serverstore: Name of a DataJoint store for which an external table exists in said schemastorage_server: A storage server specified in thestorage_serverstable in the<kind>.<name>formatbucket: Name of a bucket on said MinIO serverlocation: Location of externally stored objects in said bucketExample:[[cleaning_runs]]database_server="my_db_server"schema="my_schema"store="my_store"storage_server="minio.my_minio_server"bucket="my_bucket"location="my_location"UsageThe cleaning process can be started like so:dj-cleanerThe command above will execute all cleaning runs defined in the configuration file. The--config-fileoption can be used to pass a custom path to a configuration file to DataJoint-Cleaner:dj-cleaner--config-file/path/to/config/file