Package: oeleo
oeleoPython package / app that can be used for transferring files from an instrument-PC to a data server.Features (or limitations)Transferring using an ssh connection should preferably be used with key-pairs. This might involve some
setting up on your server (ACL) to prevent security issues (theoeleouser should only have access to
the data folder on your server).Accessing ssh can be done using password if you are not able to figure out how to set proper ownerships
on your server.oeleois one-eyed. Meaning that tracking of the "state of the duplicates" is only performed on the local side (whereoeleois running).However,oeleocontains acheckmethod that can help you figure out if starting copying is agood idea or not. And populate the database if you want.The db that stores information about the "state of the duplicates" is stored relative to the folderoeleois running from. If you delete it (by accident?),oeleowill make a new empty one from scratch next time you run.Configuration is done using environmental variables.UsageInstall$pipinstalloeleoRunCreate anoeleoworker instance.Connect the worker'sbookkeeperto asqlite3database.Filter local files.Run to copy files.Repeat from step 3.Examples and descriptionsSimple script for copying between local foldersimportosfrompathlibimportPathimporttimeimportdotenvfromoeleo.checkersimportChecksumCheckerfromoeleo.modelsimportSimpleDbHandlerfromoeleo.connectorsimportLocalConnectorfromoeleo.workersimportWorkerfromoeleo.utilsimportstart_loggerdefmain():log=start_logger()# assuming you have made a .env file:dotenv.load_dotenv()db_name=os.environ["OELEO_DB_NAME"]base_directory_from=Path(os.environ["OELEO_BASE_DIR_FROM"])base_directory_to=Path(os.environ["OELEO_BASE_DIR_TO"])filter_extension=os.environ["OELEO_FILTER_EXTENSION"]# Making a worker using the Worker class.# You can also use the `factory` functions in `oeleo.worker`# (e.g. `ssh_worker` and `simple_worker`)bookkeeper=SimpleDbHandler(db_name)checker=ChecksumChecker()local_connector=LocalConnector(directory=base_directory_from)external_connector=LocalConnector(directory=base_directory_to)worker=Worker(checker=checker,local_connector=local_connector,external_connector=external_connector,bookkeeper=bookkeeper,extension=filter_extension)# Running the worker with 5 minutes intervals.# You can also use an oeleo scheduler for this.worker.connect_to_db()whileTrue:worker.filter_local()worker.run()time.sleep(300)if__name__=="__main__":main()Environment.envfileOELEO_BASE_DIR_FROM=C:\data\local
OELEO_BASE_DIR_TO=C:\data\pub
OELEO_FILTER_EXTENSION=.csv
OELEO_DB_NAME=local2pub.db
OELEO_LOG_DIR=C:\oeleo\logs

## only needed for advanced connectors:
# OELEO_DB_HOST=<db host>
# OELEO_DB_PORT=<db port>
# OELEO_DB_USER=<db user>
# OELEO_DB_PASSWORD=<db user>
# OELEO_EXTERNAL_HOST<ssh hostname>
# OELEO_USERNAME=<ssh username>
# OELEO_PASSWORD=<ssh password>
# OELEO_KEY_FILENAME=<ssh key-pair filename>

## only needed for SharePointConnector:
# OELEO_SHAREPOINT_USERNAME=<sharepoint username (fallbacks to ssh username if missing)>
# OELEO_SHAREPOINT_URL=<url to sharepoint>
# OELEO_SHAREPOINT_SITENAME=<name of sharepoint site>
# OELEO_SHAREPOINT_DOC_LIBRARY=<name of sharepoint library>DatabaseThe database contains one table calledfilelist:idprocessed_datelocal_nameexternal_namechecksumcode12022-07-05 15:55:02.521154file_number_1.xyzC:\oeleo\check\to\file_number_1.xyzc976e564825667d7c11ba200457af263122022-07-05 15:55:02.536152file_number_10.xyzC:\oeleo\check\to\file_number_10.xyzd502512c0d32d7503feb3fd3dd287376132022-07-05 15:55:02.553157file_number_2.xyzC:\oeleo\check\to\file_number_2.xyzcb89d576f5bd57566c78247892baffa31Theprocessed_dateis when the file was last updated (meaning last timeoeleofound a new checksum for it).The table below shows what the different values ofcodemean:codemeaning0should-be-copied1should-be-copied-if-changed2should-not-be-copiedHint! You canlock(chose to never copy) a file by editing thecodemanually to 2.Using anoeleoschedulerInstead of for example using a while loop to keepoeleorunning continuously or at selected intervals,
you can use a scheduler (e.g.rocketry,watchdog,schedule, or more advanced options such asAirFlow).oeleoalso includes its own schedulers. This is an example of how to use theoeleo.SimpleScheduler:importdotenvfromoeleo.schedulersimportSimpleSchedulerfromoeleo.workersimportsimple_worker# assuming you have created an appropriate .env filedotenv.load_dotenv()worker=simple_worker()s=SimpleScheduler(worker,run_interval_time=4,# secondsmax_run_intervals=4,)s.start()Copy files from a Windows PC to a Linux server through sshimportloggingimportosfrompathlibimportPathimportdotenvfromoeleo.connectorsimportregister_passwordfromoeleo.utilsimportstart_loggerfromoeleo.workersimportssh_workerlog=start_logger()print(" ssh ".center(80,"-"))log.setLevel(logging.DEBUG)log.info(f"Starting oeleo!")dotenv.load_dotenv()external_dir="/srv/data"filter_extension=".res"register_password(os.environ["OELEO_PASSWORD"])worker=ssh_worker(db_name="ssh_to_server.db",base_directory_from=Path(r"data\raw"),base_directory_to=external_dir,extension=filter_extension,)worker.connect_to_db()try:worker.check(update_db=True)worker.filter_local()worker.run()finally:worker.close()Future planned improvementsJust plans, no promises given.make even nicer printing and logging.create CLI.create an executable.create a web-app.create a GUI (not likely).StatusWorks on my PC → PCWorks on my PC → my serverWorks on my server → my serverWorks on my instrument PC → my instrument PCWorks on my instrument PC → my serverWorks OKDeployableOn testpypiOn pypiCode understandable for othersLooking goodFairly easy to useEasy to useEasy to debug runs (e.g. editing sql)LicenceMITDevelopmentDeveloped usingpoetryonpython 3.11.Must also run onpython 3.8for Windows 7 support.Some useful commandsUpdate version# update version e.g. from 0.3.1 to 0.3.2:poetryversionpatchThen edit__init__.py:__version__="0.3.2"BuildpoetrybuildPublishIf you are using 2-factor authentication, you need to create a token on pypi.org and run:poetryconfigpypi-token.pypi<token>Then run:poetrypublishNextImprove loggingDevelopment leadJan Petter Maehlen, IFE