Package: litedfs
LiteDFSA distributed file system, based on Python3, tornado, inspired by HDFS.It's for data processing cache, not for permanent storage!All code based on Python3, do not use Python2!It still under development, so, maybe have some bugs or not stable enough!See more details athttps://github.com/fiefdx/LiteDFSFeaturesper file replica settings, support dynamic replica change, no data resharding functions, currentlyscalable with add / remove nodelightweight, pure python implementationsupport command line interfaceConceptionsname node(ldfsname): the central node of the cluster, manage all files & directories index.data node(ldfsdata): the data node of the cluster, store real file's blocks data.command line client(ldfs): the command line tool for communicate with the cluster.graphic client(ldfsviewer): the graphic tool for communicate with the cluster.DeploymentInstall LiteDFS# this will install 4 commands: ldfsname, ldfsdata, ldfs, ldfsviewer$pip3installlitedfsRun Name NodeConfigurationlog_level:NOSET# NOSET, DEBUG, INFO, WARNING, ERROR, CRITICALlog_path:/home/pi/litedfs_name/logs# log file directory, can auto generate by ldfsnamehttp_host:0.0.0.0# name node's http hosthttp_port:9000# name node's http porttcp_host:0.0.0.0# name node's tcp hosttcp_port:6061# name node's tcp portblock_size:67108864# 67108864 = 64M, file block sizedata_path:/home/pi/litedfs_name/data# name node data store directory, can auto generate by ldfsnameRun# generate configuration file & scriptsmkdir./litedfs_namecd./litedfs_name# this will generate configuration.yml and other scriptsldfsname-g./# run manuallyldfsname-c./configuration.ymlornohupldfsname-c./configuration.yml>/dev/null2>&1&# install systemd service, user and group set to use which user and group to run ldfsnamesudo./install_systemd_service.shusergroup# startsystemctlstartlitedfs-name# stopsystemctlstoplitedfs-name# uninstall systemd servicesudo./uninstall_systemd_service.sh# test$curllocalhost:9000{"message":"LiteDFS name service"}Run NodeConfigurationlog_level:NOSET# NOSET, DEBUG, INFO, WARNING, ERROR, CRITICALlog_path:/home/pi/litedfs_data/logs# log file directory, can auto generate by ldfsdatahttp_host:0.0.0.0# data node's http hosthttp_port:8002# data node's http portname_http_host:127.0.0.1# name node's http hostname_http_port:9000# name node's http portname_tcp_host:127.0.0.1# name node's tcp hostname_tcp_port:6061# name node's tcp portheartbeat_interval:1# heartbeat interval, 1 secondsheartbeat_timeout:30# heartbeat timeout, 30 secondsretry_interval:5# retry to connect name node interval, when lost connection, 5 secondsdata_path:/home/pi/litedfs_data/data# data node data store directory, can auto generate by ldfsdataRun# generate configuration file & scriptsmkdir./litedfs_datacd./litedfs_data# this will generate configuration.yml and other scriptsldfsdata-g./# run manuallyldfsdata-c./configuration.ymlornohupldfsdata-c./configuration.yml>/dev/null2>&1&# install systemd service, user and group set to use which user and group to run ldfsdatasudo./install_systemd_service.shusergroup# startsystemctlstartlitedfs-data# stopsystemctlstoplitedfs-data# uninstall systemd servicesudo./uninstall_systemd_service.sh# test$curllocalhost:8002{"message":"LiteDFS data service"}Run ViewerThis viewer must running on your local machine, it is not a public service, it is a graphic client based on web technique.Configurationlog_level:NOSET# NOSET, DEBUG, INFO, WARNING, ERROR, CRITICALlog_path:/home/pi/litedfs_viewer/logs# log file directory, can auto generate by ldfsviewerhttp_host:0.0.0.0# viewer's http hosthttp_port:8088# viewer's http portname_http_host:192.168.199.149# name node's http hostname_http_port:9000# name node's http portdata_path:/home/pi/litedfs_viewer/data# viewer data store directory, can auto generate by ldfsviewerRun# generate configuration file & scriptsmkdir./litedfs_viewercd./litedfs_viewer# this will generate configuration.yml and other scriptsldfsviewer-g./# run manuallyldfsviewer-c./configuration.ymlornohupldfsviewer-c./configuration.yml>/dev/null2>&1&# install systemd service, user and group set to use which user and group to run ldfsviewersudo./install_systemd_service.shusergroup# startsystemctlstartlitedfs-viewer# stopsystemctlstoplitedfs-viewer# uninstall systemd servicesudo./uninstall_systemd_service.sh# test# use web browser open: http://localhost:8088Operate With LiteDFS Cluster# list root directory$ldfslocalhost:9000directorylist-r/# | id | type | size | name# create test directory$ldfslocalhost:9000directorycreate-r/test
createdirectory[/test]success# list root directory again$ldfslocalhost:9000directorylist-r/# | id | type      | size | name1||directory|0|test# create a file$ldfslocalhost:9000filecreate-r/test/test.tar.gz-l./examples.tar.gzcreatefile[/test/test.tar.gz]success# list test directory$ldfslocalhost:9000directorylist-r/test# | id                                   | type | size      | name1|878b17d4-cc11-4bba-88b0-2186b77ef552|file|110237727|test.tar.gz# create test2 directory$ldfslocalhost:9000directorycreate-r/test2
createdirectory[/test2]success# list root directory again$ldfslocalhost:9000directorylist-r/# | id | type      | size | name1||directory|0|test2||directory|0|test2# move test.tar.gz into test2 directory$ldfslocalhost:9000filemove-s/test/test.tar.gz-t/test2
movefile[/test/test.tar.gz]to/test2success# list test directory again$ldfslocalhost:9000directorylist-r/test# | id | type | size | name# list test2 directory again$ldfslocalhost:9000directorylist-r/test2# | id                                   | type | size      | name1|878b17d4-cc11-4bba-88b0-2186b77ef552|file|110237727|test.tar.gz# create file with replica 2$ldfslocalhost:9000filecreate-r/test/test.tar.gz-l./examples.tar.gz-R2createfile[/test/test.tar.gz]success# update file replica 3$ldfslocalhost:9000filecreate-r/test/test.tar.gz-l./examples.tar.gz-R3updatefile[/test/test.tar.gz]success# download /test/test.tar.gz to local file ./test.tar.gz$ldfslocalhost:9000filedownload-r/test/test.tar.gz-l./test.tar.gz
downloadfile[/test/test.tar.gz=>./test.tar.gz]success