Repository: dayyass/pydfs (language: python)

Issue:
add dfs put and get commands



Discussion / hints:

Fix patch:
diff --git a/.github/workflows/codecov.yml b/.github/workflows/codecov.yml
new file mode 100644
index 0000000..0bda999
--- /dev/null
+++ b/.github/workflows/codecov.yml
@@ -0,0 +1,37 @@
+# This workflow will install Python dependencies and run codecov
+# https://github.com/codecov/codecov-action#example-workflowyml-with-codecov-action
+
+name: codecov
+
+on:
+  push:
+    branches: [main, develop]
+  pull_request:
+    branches: [main, develop]
+
+jobs:
+  build:
+    runs-on: ${{ matrix.os }}
+    strategy:
+      matrix:
+        os: [ubuntu-latest]
+    steps:
+    - uses: actions/checkout@master
+    - name: Set up Python
+      uses: actions/setup-python@master
+      with:
+        python-version: 3.7
+    - name: Install dependencies
+      run: |
+        pip install --upgrade pip
+        pip install -r requirements.txt
+        pip install pytest pytest-cov
+    - name: Generate coverage report
+      run: pytest --cov=./ --cov-report=xml
+    # - name: Upload coverage to Codecov
+    #   uses: codecov/codecov-action@v1
+    #   with:
+    #     flags: unittests
+    #     env_vars: OS,PYTHON
+    #     fail_ci_if_error: true
+    #     verbose: true
diff --git a/.github/workflows/linter.yml b/.github/workflows/linter.yml
new file mode 100644
index 0000000..3895dc9
--- /dev/null
+++ b/.github/workflows/linter.yml
@@ -0,0 +1,37 @@
+# This workflow will install Python dependencies and run linter
+# For more information see: https://help.github.com/actions/language-and-framework-guides/using-python-with-github-actions
+
+# TODO: update linters
+
+name: linter
+
+on:
+  push:
+    branches: [main, develop]
+  pull_request:
+    branches: [main, develop]
+
+jobs:
+  build:
+    runs-on: ${{ matrix.os }}
+    strategy:
+      matrix:
+        os: [ubuntu-latest]
+    steps:
+    - uses: actions/checkout@v2
+    - name: Set up Python
+      uses: actions/setup-python@v2
+      with:
+        python-version: 3.7
+    - name: Install dependencies
+      run: |
+        pip install --upgrade pip
+        pip install isort black flake8 mypy
+    - name: Code format check with isort
+      run: isort --check-only --profile black .
+    - name: Code format check with black
+      run: black --check .
+    - name: Code format check with flake8
+      run: flake8 --ignore E501,E203,W503 .
+    - name: Type check with mypy
+      run: mypy --ignore-missing-imports .
diff --git a/.gitignore b/.gitignore
index 3844ae6..abab579 100644
--- a/.gitignore
+++ b/.gitignore
@@ -12,3 +12,8 @@
 
 venv
 .idea
+
+master.sqlite
+master.sqlite-journal
+
+pydfs.egg-info
diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
index 2685945..d992b91 100644
--- a/.pre-commit-config.yaml
+++ b/.pre-commit-config.yaml
@@ -8,6 +8,15 @@ repos:
   rev: 22.3.0
   hooks:
     - id: black
+- repo: https://github.com/pycqa/flake8
+  rev: 4.0.1
+  hooks:
+    - id: flake8
+      args: ["--ignore", "E501,E203,W503"]
+- repo: https://github.com/pre-commit/mirrors-mypy
+  rev: v0.961
+  hooks:
+    - id: mypy
 - repo: https://github.com/pre-commit/pre-commit-hooks
   rev: v4.3.0
   hooks:
@@ -20,20 +29,11 @@ repos:
     - id: trailing-whitespace
     - id: check-docstring-first
     - id: requirements-txt-fixer
-- repo: https://github.com/pycqa/flake8
-  rev: 4.0.1
-  hooks:
-    - id: flake8
-      args: ["--ignore", "E501,E203,W503"]
-- repo: https://github.com/pre-commit/mirrors-mypy
-  rev: v0.961
-  hooks:
-    - id: mypy
-- repo: local
-  hooks:
-    - id: unittest
-      name: unittest
-      entry: venv/bin/python -m unittest discover
-      language: python
-      always_run: true
-      pass_filenames: false
+# - repo: local
+#   hooks:
+#     - id: unittest
+#       name: unittest
+#       entry: venv/bin/python -m unittest discover
+#       language: python
+#       always_run: true
+#       pass_filenames: false
diff --git a/Dockerfile b/Dockerfile
new file mode 100644
index 0000000..dc23e46
--- /dev/null
+++ b/Dockerfile
@@ -0,0 +1,10 @@
+FROM python:3.7-slim-buster
+LABEL maintainer = "Dani El-Ayyass <dayyass@yandex.ru>"
+
+WORKDIR /workdir
+COPY . .
+
+RUN pip install --no-cache-dir --upgrade pip && \
+    pip install --no-cache-dir .
+
+CMD ["bash"]
diff --git a/README.md b/README.md
index c385a52..23b98ea 100644
--- a/README.md
+++ b/README.md
@@ -1,2 +1,54 @@
-# pydfs
-Distributed File System written in Python
+# PyDFS
+Distributed File System written in Python.
+
+## Installation
+```
+pip install pydfs
+```
+
+## Usage
+PyDFS is Centralized Distributed File System, which means there is a *master* and *slave* nodes.
+The current implementation assumes that the system has only one master node and many slave nodes.
+
+PyDFS supports *command line interface* (*CLI*) to interact with it.
+There are 2 groups of the commands (like Docker Management Commands):
+- `pydfs init` - to initialize, manage and sync nodes
+- `pydfs dfs` - to interact with DFS itself (put/get data to/from it)
+
+Let's take a closer look at these commands.
+
+### init commands
+With `pydfs init` command you can initialize master and slave nodes - it's pretty simple:
+- `pydfs init master`
+- `pydfs init slave --master_ip [IP]`
+
+### dfs commands
+- `pydfs dfs put --path PATH --master_ip [IP]`
+- `pydfs dfs get --path PATH --master_ip [IP]`
+
+### other commands
+- `pydfs --version`
+- `pydfs --info`
+
+## Docker
+You can use PyDFS with Docker.
+
+To build a docker image with PyDFS run:
+```docker
+docker build -t pydfs .
+```
+<!-- TODO: push image to Docker Hub -->
+
+To run a docker container with PyDFS run:
+```
+docker run -it pydfs
+```
+
+### Docker Compose
+You can also use *docker-compose* to run multi-container application with one master node and two slave nodes.
+```
+docker-compose up --build
+```
+
+## Requirements
+Python >= 3.7
diff --git a/docker-compose.yml b/docker-compose.yml
new file mode 100644
index 0000000..388218b
--- /dev/null
+++ b/docker-compose.yml
@@ -0,0 +1,41 @@
+version: "3.9"
+
+services:
+
+  master_node:
+    container_name: master_node
+    build: .
+    command: pydfs init master
+
+  slave_node_1:
+    container_name: slave_node_1
+    build: .
+    command: bash -c "./wait-for-it.sh master_node:5000 && pydfs init slave --master_ip master_node"
+    depends_on:
+      - master_node
+
+  slave_node_2:
+    container_name: slave_node_2
+    build: .
+    command: bash -c "./wait-for-it.sh master_node:5000 && pydfs init slave --master_ip master_node"
+    depends_on:
+      - master_node
+
+  user_node_1:
+    container_name: user_node_1
+    build: .
+    command: bash -c "./wait-for-it.sh master_node:5000 && ./wait-for-it.sh slave_node_1:5000 && ./wait-for-it.sh slave_node_2:5000 && pydfs dfs put --path wait-for-it.sh --master_ip master_node && tail -f /dev/null"
+    depends_on:
+      - master_node
+      - slave_node_1
+      - slave_node_2
+
+  user_node_2:
+    container_name: user_node_2
+    build: .
+    command: tail -f /dev/null
+    depends_on:
+      - master_node
+      - slave_node_1
+      - slave_node_2
+      - user_node_1
diff --git a/pydfs/__main__.py b/pydfs/__main__.py
new file mode 100644
index 0000000..ee60480
--- /dev/null
+++ b/pydfs/__main__.py
@@ -0,0 +1,132 @@
+import argparse
+import os
+import sys  # TODO: remove it
+
+sys.path.append(".")
+from pydfs import __version__  # noqa: E402
+from pydfs.arg_parse import get_argparse  # noqa: E402
+from pydfs.dfs import cmd_dfs_get_request, cmd_dfs_put_request  # noqa: E402
+from pydfs.init import cmd_init_master, cmd_init_slave  # noqa: E402
+from pydfs.logger import _logger  # noqa: E402
+
+
+def main() -> None:
+    """
+    pydfs main function (entry point)
+    """
+
+    args = get_argparse().parse_args()
+    _main(args)
+
+
+def _main(args: argparse.Namespace) -> int:
+    """
+    pydfs main function (entry point)
+
+    Args:
+        args (argparse.Namespace): CLI arguments.
+
+    Returns:
+        int: exit code.
+    """
+
+    if args.version and (not args.command):
+        _version()
+        return 0
+
+    if args.info and (not args.command):
+        _info()
+        return 0
+
+    _logger.debug(f"CLI arguments: {args}")
+
+    if args.command == "init":
+
+        if args.subcommand == "master":
+            _logger.info(f"pydfs init {args.subcommand}")
+            cmd_init_master()
+
+        elif args.subcommand == "slave":
+            _logger.info(f"pydfs init {args.subcommand} --master_ip {args.master_ip}")
+            cmd_init_slave(master_ip=args.master_ip)
+
+        else:
+            err_msg = f"unknown init subcommand: '{args.subcommand}' (use 'master' or 'slave')"
+            _logger.error(err_msg)
+            raise ValueError(err_msg)
+
+    elif args.command == "dfs":
+
+        if args.subcommand == "put":
+            _logger.info(
+                f"pydfs dfs put --path {args.path} --master_ip {args.master_ip}"
+            )
+            cmd_dfs_put_request(
+                ip=args.master_ip,
+                files={"upload_file": open(args.path, mode="rb")},
+            )
+            # TODO add error handler
+            _logger.info(f"putting file {args.path} to pydfs succeeded")
+
+        elif args.subcommand == "get":
+            _logger.info(
+                f"pydfs dfs get --path {args.path} --master_ip {args.master_ip}"
+            )
+            cmd_dfs_get_request(
+                ip=args.master_ip,
+                path=args.path,
+            )
+            # TODO add error handler
+            _logger.info(f"getting file {args.path} from pydfs succeeded")
+
+        else:
+            err_msg = (
+                f"unknown dfs subcommand: '{args.subcommand}' (use 'put' or 'get')"
+            )
+            _logger.error(err_msg)
+            raise ValueError(err_msg)
+
+    else:
+        err_msg = f"unknown command: '{args.command}' (use 'init' or 'dfs')"
+        _logger.error(err_msg)
+        raise ValueError(err_msg)
+
+    return 0
+
+
+def _version() -> None:
+    """
+    pydfs --version
+    """
+
+    # TODO: maybe not use print
+    print(f"v{__version__}")
+
+
+def _info() -> None:
+    """
+    pydfs --info
+    """
+
+    path_to_pydfs = os.path.join(os.environ["HOME"], ".pydfs")
+    path_to_master_db = os.path.join(path_to_pydfs, "master.sqlite")
+
+    # TODO: maybe not use print
+    # TODO: maybe allow master and slave node at the same time
+    if os.path.exists(path_to_master_db):
+        # TODO: add slave nodes info
+        print("pydfs master node\n")
+    elif os.path.exists(path_to_pydfs):
+        # TODO: add master node info
+        print("pydfs slave node\n")
+    else:
+        msg = (
+            "not a pydfs node\n"
+            "init it as a master node with `pydfs init master` or\n"
+            "init it as a slave node with `pydfs init slave --master_ip [IP]`\n"
+        )
+        print(msg)
+
+
+if __name__ == "__main__":
+    main()
diff --git a/pydfs/arg_parse.py b/pydfs/arg_parse.py
new file mode 100644
index 0000000..2eda912
--- /dev/null
+++ b/pydfs/arg_parse.py
@@ -0,0 +1,99 @@
+import argparse
+
+
+def get_argparse() -> argparse.ArgumentParser:
+    """
+    Function to get CLI arguments parser.
+
+    Returns:
+        argparse.ArgumentParser: CLI arguments parser.
+    """
+
+    # TODO: add --info argument
+    # TODO: add login
+    parser = argparse.ArgumentParser()
+    parser.add_argument(
+        "--version",
+        action="store_true",
+        help="pydfs version",
+    )
+    parser.add_argument(
+        "--info",
+        action="store_true",
+        help="pydfs node info",
+    )
+
+    # https://stackoverflow.com/questions/8250010/argparse-identify-which-subparser-was-used
+    subparsers = parser.add_subparsers(dest="command")
+
+    # init commands
+    subparser_init = subparsers.add_parser(
+        "init",
+        help="init commands",
+    )
+    subsubparsers_init = subparser_init.add_subparsers(dest="subcommand")
+
+    # init master command
+    subsubparsers_init.add_parser(
+        "master",
+        help="init pydfs master node",
+    )
+
+    # init slave command
+    subsubparser_init_slave = subsubparsers_init.add_parser(
+        "slave",
+        help="init pydfs slave node",
+    )
+    subsubparser_init_slave.add_argument(
+        "--master_ip",
+        type=str,
+        required=True,
+        help="pydfs master node IP",
+    )
+
+    # dfs commands
+    subparser_dfs = subparsers.add_parser(
+        "dfs",
+        help="dfs commands",
+    )
+    subsubparsers_dfs = subparser_dfs.add_subparsers(dest="subcommand")
+
+    # dfs put command
+    subsubparser_dfs_put = subsubparsers_dfs.add_parser(
+        "put",
+        help="command to put local file in pydfs",
+    )
+    subsubparser_dfs_put.add_argument(
+        "--path",
+        type=str,
+        required=True,
+        help="path to local file to put in pydfs",
+    )
+    # TODO: remove master_ip
+    subsubparser_dfs_put.add_argument(
+        "--master_ip",
+        type=str,
+        required=True,
+        help="pydfs master node IP",
+    )
+
+    # dfs get command
+    subsubparser_dfs_get = subsubparsers_dfs.add_parser(
+        "get",
+        help="command to get pydfs file back to local",
+    )
+    subsubparser_dfs_get.add_argument(
+        "--path",
+        type=str,
+        required=True,
+        help="path to pydfs file to get it locally",
+    )
+    # TODO: remove master_ip
+    subsubparser_dfs_get.add_argument(
+        "--master_ip",
+        type=str,
+        required=True,
+        help="pydfs master node IP",
+    )
+
+    return parser
diff --git a/pydfs/dfs.py b/pydfs/dfs.py
new file mode 100644
index 0000000..8a02aea
--- /dev/null
+++ b/pydfs/dfs.py
@@ -0,0 +1,43 @@
+import random
+import sys  # TODO: remove it
+from io import BufferedReader
+from typing import Dict, List
+
+import requests  # type: ignore
+
+sys.path.append(".")
+from pydfs.logger import _logger  # noqa: E402
+
+
+def cmd_dfs_put_request(ip: str, files: Dict[str, BufferedReader]):
+    response = requests.put(  # TODO: maybe post
+        url=f"http://{ip}:5000/put_file",  # TODO: add https
+        files=files,
+    )
+    if response.status_code == 200:
+        _logger.info(f"sending file {files['upload_file']} to {ip} succeeded")
+    else:
+        _logger.error(f"sending file {files['upload_file']} to {ip} failed")
+
+
+def cmd_dfs_get_request(ip: str, path: str):
+    response = requests.get(
+        url=f"http://{ip}:5000/get_file",  # TODO: add https
+        params={"path": path},
+    )
+    if response.status_code == 200:
+        _logger.info(f"requesting file {path} from {ip} succeeded")
+    else:  # TODO: add error handler
+        _logger.error(f"requesting file {path} from {ip} failed")
+
+    # TODO: standartize interface with cmd_dfs_get_request
+    # TODO: error handler
+    return response
+
+
+# TODO: add type annotation without cycle import
+def _choose_slave_node(
+    slave_nodes: List,
+):
+    slave_node_tgt = random.choice(slave_nodes)
+    return slave_node_tgt
diff --git a/pydfs/init.py b/pydfs/init.py
new file mode 100644
index 0000000..ad0f77e
--- /dev/null
+++ b/pydfs/init.py
@@ -0,0 +1,117 @@
+import os
+import sys  # TODO: remove it
+
+import requests  # type: ignore
+
+sys.path.append(".")
+from pydfs.logger import _logger  # noqa: E402
+from pydfs.utils import ping  # noqa: E402
+
+
+# TODO: make daemon from it
+def cmd_init_master() -> None:
+    """
+    pydfs init master
+    """
+
+    _mkdir_pydfs()
+
+    # TODO: maybe move it up
+    # TODO: validate if import is correct
+    from pydfs.init_master_app import app  # noqa: E402
+
+    _logger.info("master node initialized successfully")
+
+    # TODO: add WSGI (e.g. gunicorn)
+    # TODO: remove host="0.0.0.0"
+    app.run(host="0.0.0.0")
+
+
+# TODO: think how to match existed slave to master
+# TODO: check if master and slave are already connected
+def cmd_init_slave(master_ip: str) -> None:
+    """
+    pydfs init slave --master_ip ...
+
+    Args:
+        master_ip (str): pydfs master node IP.
+    """
+
+    # TODO: maybe change order
+    _mkdir_pydfs()
+    # _ping_master_node(master_ip=master_ip)  # TODO: fix with docker-compose
+    _send_slave_ip_to_master(master_ip=master_ip)
+
+    # TODO: maybe move it up
+    # TODO: validate if import is correct
+    from pydfs.init_slave_app import app  # noqa: E402
+
+    _logger.info("slave node initialized successfully")
+
+    # TODO: add WSGI (e.g. gunicorn)
+    # TODO: remove host="0.0.0.0"
+    app.run(host="0.0.0.0")
+
+
+def _mkdir_pydfs() -> None:
+    """
+    Create pydfs working directory ~/.pydfs
+    """
+
+    path = os.path.join(os.environ["HOME"], ".pydfs")
+    if not os.path.exists(path):
+        os.makedirs(path, exist_ok=False)
+        _logger.info(f"'{path}' folder created")
+    else:
+        # TODO: think about workflow (behaviour)
+        _logger.info(f"'{path}' folder has already been created")
+
+
+# TODO: save pydfs init info, for example master_ip for slave nodes
+def _ping_master_node(master_ip: str) -> None:
+    """
+    Function to ping master node by its IP.
+
+    Args:
+        master_ip (str): pydfs master node IP.
+    """
+
+    if ping(master_ip):
+        _logger.info(f"master node '{master_ip}' was found")
+    else:
+        # TODO: make more informative error message
+        err_msg = f"master node '{master_ip}' was not found"
+        _logger.error(err_msg)
+        raise ConnectionError(err_msg)
+
+
+def _send_slave_ip_to_master(master_ip: str) -> None:
+    """
+    Function to sent slave node IP to master node.
+
+    Args:
+        master_ip (str): pydfs master node IP.
+    """
+
+    # TODO: add https
+    # TODO: remove port hardcode (parametrize)
+    # TODO: maybe post request, not put
+    # TODO: send local api for validation
+    response = requests.put(
+        f"http://{master_ip}:5000/add_slave",
+        data={},  # TODO: validate
+    )
+
+    if response.status_code == 200:
+        _logger.info("successfully send ip to master")
+    else:
+        try:  # TODO: fix try/except block
+            _logger.debug(f"response json keys: {response.json().keys()}")
+            err_msg = f"HTTP {response.status_code}: {response.json()['message']}"
+        except:  # noqa: E722
+            _logger.debug("response doesn't have json in it")
+            err_msg = f"HTTP {response.status_code}"
+
+        # TODO: maybe warning, not error
+        _logger.error(f"{err_msg}\nslave node was NOT initialized successfully")
+        raise requests.HTTPError(err_msg)
diff --git a/pydfs/init_master_app.py b/pydfs/init_master_app.py
new file mode 100644
index 0000000..aaa269b
--- /dev/null
+++ b/pydfs/init_master_app.py
@@ -0,0 +1,150 @@
+import os
+import sys  # TODO: remove it
+from datetime import datetime
+
+from flask import Flask, request
+from flask_restful import Api, Resource
+from flask_sqlalchemy import SQLAlchemy
+
+sys.path.append(".")
+from pydfs.dfs import (  # noqa: E402
+    _choose_slave_node,
+    cmd_dfs_get_request,
+    cmd_dfs_put_request,
+)
+from pydfs.logger import _logger  # noqa: E402
+
+app = Flask(__name__)
+api = Api(app)
+
+# database
+# TODO: maybe add successful messages in logger
+# TODO: come up with behaviour when db already exists
+# TODO: add db admin user with password
+# TODO: create normal workflow if slave IP already in db
+_logger.info("creating master.sqlite in ~/.pydfs")
+uri = f"sqlite:///{os.path.join(os.environ['HOME'], '.pydfs', 'master.sqlite')}"
+
+app.config["SQLALCHEMY_DATABASE_URI"] = uri
+app.config["SQLALCHEMY_TRACK_MODIFICATIONS"] = False  # TODO: validate
+db = SQLAlchemy(app)
+
+
+class Slave(db.Model):  # type: ignore
+
+    __tablename__ = "slave"  # TODO: check the need
+    # __table_args__ = {"extend_existing": True}  # TODO: check it
+
+    id = db.Column(db.Integer, primary_key=True, autoincrement=True)
+    ip_address = db.Column(db.String(15), unique=True, nullable=False)
+    timestamp = db.Column(db.String(23), unique=False, nullable=False)
+
+    def __repr__(self):
+        return f"<Slave {self.ip_address}>"
+
+
+_logger.info("creating slave table in master.sqlite")
+
+
+# TODO: link (relation) with Slave table by ip_address
+class Files(db.Model):  # type: ignore
+
+    __tablename__ = "files"  # TODO: check the need
+    # __table_args__ = {"extend_existing": True}  # TODO: check it
+
+    id = db.Column(db.Integer, primary_key=True, autoincrement=True)
+    # TODO: fix 255 length
+    filename = db.Column(db.String(255), unique=True, nullable=False)
+    ip_address = db.Column(db.String(15), unique=True, nullable=False)
+    timestamp = db.Column(db.String(23), unique=False, nullable=False)
+
+    def __repr__(self):
+        return f"<File {self.filename} at {self.ip_address}>"
+
+
+_logger.info("creating files table in master.sqlite")
+
+
+db.create_all()
+
+
+class AddSlave(Resource):
+    def put(self):
+        # https://stackoverflow.com/questions/3759981/get-ip-address-of-visitors-using-flask-for-python
+        _logger.info(
+            f"inserting slave node address {request.remote_addr} in master.sqlite"
+        )
+
+        slave = Slave(
+            ip_address=request.remote_addr,
+            timestamp=datetime.now().strftime(r"%Y-%m-%d %H:%M:%S.%f")[:-3],
+        )
+        db.session.add(slave)
+        db.session.commit()
+
+        return {}  # TODO: validate
+
+
+class PutFileMaster(Resource):
+    def put(self):
+
+        _logger.debug(f"request.files: {request.files}")
+        _logger.debug(f"receive file: {request.files['upload_file']}")
+
+        slave_nodes = Slave.query.all()
+        _logger.debug(f"available slave nodes: {slave_nodes}")
+
+        slave_node_tgt = _choose_slave_node(slave_nodes=slave_nodes)
+        _logger.debug(f"chosen slave node: {slave_node_tgt}")
+
+        file = request.files["upload_file"]
+
+        # TODO: add redirect
+        # TODO: make as a transaction
+        # https://stackoverflow.com/questions/32460524/post-uploaded-file-using-requests
+        cmd_dfs_put_request(
+            ip=slave_node_tgt.ip_address,
+            files={"upload_file": (file.filename, file.stream, file.mimetype)},
+        )
+
+        # TODO add error handler
+        file_db = Files(
+            filename=file.filename,
+            ip_address=slave_node_tgt.ip_address,
+            timestamp=datetime.now().strftime(r"%Y-%m-%d %H:%M:%S.%f")[:-3],
+        )
+        db.session.add(file_db)
+        db.session.commit()
+
+        return {}  # TODO: validate
+
+
+class GetFileMaster(Resource):
+    def get(self):
+
+        _logger.debug(f"request args: {request.args}")
+
+        path = request.args["path"]  # TODO: rename filename
+        _logger.debug(f"request 'path' param: {path}")
+
+        _logger.info(f"getting slave node ip with file: {path}")
+        files = Files.query.filter_by(filename=path).all()
+        assert len(files) == 1  # TODO: remove and create good error handler
+        slave_node_tgt_ip_address = files[0].ip_address
+
+        # TODO: add redirect
+        # TODO: make as a transaction
+        # https://stackoverflow.com/questions/32460524/post-uploaded-file-using-requests
+        response = cmd_dfs_get_request(
+            ip=slave_node_tgt_ip_address,
+            path=path,
+        )
+
+        _logger.debug(f"slave response on master: {response}")
+
+        return {}  # TODO: validate
+
+
+api.add_resource(AddSlave, "/add_slave")
+api.add_resource(PutFileMaster, "/put_file")
+api.add_resource(GetFileMaster, "/get_file")
diff --git a/pydfs/init_slave_app.py b/pydfs/init_slave_app.py
new file mode 100644
index 0000000..2e1e30b
--- /dev/null
+++ b/pydfs/init_slave_app.py
@@ -0,0 +1,70 @@
+import os
+import sys  # TODO: remove it
+
+from flask import Flask, request, send_file
+from flask_restful import Api, Resource
+from flask_sqlalchemy import SQLAlchemy
+
+sys.path.append(".")
+from pydfs.logger import _logger  # noqa: E402
+
+app = Flask(__name__)
+api = Api(app)
+
+# database
+# TODO: maybe add successful messages in logger
+# TODO: come up with behaviour when db already exists
+# TODO: add db admin user with password
+# TODO: create normal workflow if slave IP already in db
+_logger.info("creating slave.sqlite in ~/.pydfs")
+uri = f"sqlite:///{os.path.join(os.environ['HOME'], '.pydfs', 'slave.sqlite')}"
+
+app.config["SQLALCHEMY_DATABASE_URI"] = uri
+app.config["SQLALCHEMY_TRACK_MODIFICATIONS"] = False  # TODO: validate
+db = SQLAlchemy(app)
+
+# TODO: add slave and master file tables
+db.create_all()
+
+
+class PutFileSlave(Resource):
+    def put(self):
+
+        _logger.debug(f"request.files: {request.files}")
+        _logger.debug(f"receive file: {request.files['upload_file']}")
+
+        request.files["upload_file"].save(
+            os.path.join(
+                os.environ["HOME"],
+                ".pydfs",
+                request.files["upload_file"].filename,  # TODO: validate
+            ),
+        )
+
+        return {}  # TODO: validate
+
+
+class GetFileSlave(Resource):
+    def get(self):
+
+        _logger.debug(f"request args: {request.args}")
+
+        path = request.args["path"]  # TODO: rename filename
+        _logger.debug(f"request 'path' param: {path}")
+
+        full_path = os.path.join(
+            os.environ["HOME"],
+            ".pydfs",
+            path,  # TODO: validate
+        )
+        _logger.debug(f"full path: {full_path}")
+
+        # TODO: try send_from_directory
+        return send_file(
+            path_or_file=full_path,
+            download_name=path,
+        )  # TODO: validate and refactor
+
+
+api.add_resource(PutFileSlave, "/put_file")
+api.add_resource(GetFileSlave, "/get_file")
diff --git a/pydfs/logger.py b/pydfs/logger.py
new file mode 100644
index 0000000..a891092
--- /dev/null
+++ b/pydfs/logger.py
@@ -0,0 +1,12 @@
+import logging
+import sys
+
+# https://stackoverflow.com/questions/14058453/making-python-loggers-output-all-messages-to-stdout-in-addition-to-log-file
+_logger = logging.getLogger()
+_logger.setLevel(logging.DEBUG)
+
+handler = logging.StreamHandler(sys.stdout)
+handler.setLevel(logging.DEBUG)
+formatter = logging.Formatter("%(asctime)s - %(name)s - %(levelname)s - %(message)s")
+handler.setFormatter(formatter)
+_logger.addHandler(handler)
diff --git a/pydfs/utils.py b/pydfs/utils.py
new file mode 100644
index 0000000..8b4d307
--- /dev/null
+++ b/pydfs/utils.py
@@ -0,0 +1,24 @@
+import os
+import sys  # TODO: remove it
+
+sys.path.append(".")
+from pydfs.logger import _logger  # noqa: E402
+
+
+# TODO: use with IP
+# TODO: validate
+# TODO: maybe rename hostname
+def ping(hostname: str) -> bool:
+    """
+    https://stackoverflow.com/questions/2953462/pinging-servers-in-python
+    """
+
+    cmd = f"ping -c 1 {hostname}"
+    _logger.debug(f"ping command: {cmd}")
+
+    response = os.system(cmd)
+    _logger.debug(f"ping response code: {response}")
+
+    success = not bool(response)
+
+    return success
diff --git a/requirements.txt b/requirements.txt
index 594f578..7c05126 100644
--- a/requirements.txt
+++ b/requirements.txt
@@ -1,1 +1,6 @@
+coverage==6.4.1  # dev
+flask==2.1.2
+flask-restful==0.3.9
+flask-sqlalchemy==2.5.1
 pre-commit==2.19.0  # dev
+requests==2.28.1
diff --git a/setup.py b/setup.py
new file mode 100644
index 0000000..d4717c7
--- /dev/null
+++ b/setup.py
@@ -0,0 +1,32 @@
+from setuptools import setup
+
+with open("README.md", mode="r", encoding="utf-8") as fp:
+    long_description = fp.read()
+
+
+# TODO: add license
+setup(
+    name="pydfs",
+    version="0.0.1",
+    description="Distributed File System written in Python",
+    long_description=long_description,
+    author="Dani El-Ayyass, Artem Fomin",
+    author_email="dayyass@yandex.ru, artfom02@gmail.com",
+    url="https://github.com/silkway-ai/pydfs",
+    packages=["pydfs"],
+    entry_points={"console_scripts": ["pydfs = pydfs.__main__:main"]},
+    install_requires=[
+        "flask==2.1.2",
+        "flask-restful==0.3.9",
+        "flask-sqlalchemy==2.5.1",
+        "requests==2.28.1",
+    ],
+    keywords=[
+        "python",
+        "distributed-systems",
+        "hadoop",
+        "filesystem",
+        "hdfs",
+        "mapreduce",
+    ],
+)
diff --git a/wait-for-it.sh b/wait-for-it.sh
new file mode 100755
index 0000000..f38b14d
--- /dev/null
+++ b/wait-for-it.sh
@@ -0,0 +1,185 @@
+#!/usr/bin/env bash
+# Use this script to test if a given TCP host/port are available
+
+# https://stackoverflow.com/questions/52699899/depends-on-doesnt-wait-for-another-service-in-docker-compose-1-22-0
+# https://github.com/vishnubob/wait-for-it
+
+WAITFORIT_cmdname=${0##*/}
+
+echoerr() { if [[ $WAITFORIT_QUIET -ne 1 ]]; then echo "$@" 1>&2; fi }
+
+usage()
+{
+    cat << USAGE >&2
+Usage:
+    $WAITFORIT_cmdname host:port [-s] [-t timeout] [-- command args]
+    -h HOST | --host=HOST       Host or IP under test
+    -p PORT | --port=PORT       TCP port under test
+                                Alternatively, you specify the host and port as host:port
+    -s | --strict               Only execute subcommand if the test succeeds
+    -q | --quiet                Don't output any status messages
+    -t TIMEOUT | --timeout=TIMEOUT
+                                Timeout in seconds, zero for no timeout
+    -- COMMAND ARGS             Execute command with args after the test finishes
+USAGE
+    exit 1
+}
+
+wait_for()
+{
+    if [[ $WAITFORIT_TIMEOUT -gt 0 ]]; then
+        echoerr "$WAITFORIT_cmdname: waiting $WAITFORIT_TIMEOUT seconds for $WAITFORIT_HOST:$WAITFORIT_PORT"
+    else
+        echoerr "$WAITFORIT_cmdname: waiting for $WAITFORIT_HOST:$WAITFORIT_PORT without a timeout"
+    fi
+    WAITFORIT_start_ts=$(date +%s)
+    while :
+    do
+        if [[ $WAITFORIT_ISBUSY -eq 1 ]]; then
+            nc -z $WAITFORIT_HOST $WAITFORIT_PORT
+            WAITFORIT_result=$?
+        else
+            (echo -n > /dev/tcp/$WAITFORIT_HOST/$WAITFORIT_PORT) >/dev/null 2>&1
+            WAITFORIT_result=$?
+        fi
+        if [[ $WAITFORIT_result -eq 0 ]]; then
+            WAITFORIT_end_ts=$(date +%s)
+            echoerr "$WAITFORIT_cmdname: $WAITFORIT_HOST:$WAITFORIT_PORT is available after $((WAITFORIT_end_ts - WAITFORIT_start_ts)) seconds"
+            break
+        fi
+        sleep 1
+    done
+    return $WAITFORIT_result
+}
+
+wait_for_wrapper()
+{
+    # In order to support SIGINT during timeout: http://unix.stackexchange.com/a/57692
+    if [[ $WAITFORIT_QUIET -eq 1 ]]; then
+        timeout $WAITFORIT_BUSYTIMEFLAG $WAITFORIT_TIMEOUT $0 --quiet --child --host=$WAITFORIT_HOST --port=$WAITFORIT_PORT --timeout=$WAITFORIT_TIMEOUT &
+    else
+        timeout $WAITFORIT_BUSYTIMEFLAG $WAITFORIT_TIMEOUT $0 --child --host=$WAITFORIT_HOST --port=$WAITFORIT_PORT --timeout=$WAITFORIT_TIMEOUT &
+    fi
+    WAITFORIT_PID=$!
+    trap "kill -INT -$WAITFORIT_PID" INT
+    wait $WAITFORIT_PID
+    WAITFORIT_RESULT=$?
+    if [[ $WAITFORIT_RESULT -ne 0 ]]; then
+        echoerr "$WAITFORIT_cmdname: timeout occurred after waiting $WAITFORIT_TIMEOUT seconds for $WAITFORIT_HOST:$WAITFORIT_PORT"
+    fi
+    return $WAITFORIT_RESULT
+}
+
+# process arguments
+while [[ $# -gt 0 ]]
+do
+    case "$1" in
+        *:* )
+        WAITFORIT_hostport=(${1//:/ })
+        WAITFORIT_HOST=${WAITFORIT_hostport[0]}
+        WAITFORIT_PORT=${WAITFORIT_hostport[1]}
+        shift 1
+        ;;
+        --child)
+        WAITFORIT_CHILD=1
+        shift 1
+        ;;
+        -q | --quiet)
+        WAITFORIT_QUIET=1
+        shift 1
+        ;;
+        -s | --strict)
+        WAITFORIT_STRICT=1
+        shift 1
+        ;;
+        -h)
+        WAITFORIT_HOST="$2"
+        if [[ $WAITFORIT_HOST == "" ]]; then break; fi
+        shift 2
+        ;;
+        --host=*)
+        WAITFORIT_HOST="${1#*=}"
+        shift 1
+        ;;
+        -p)
+        WAITFORIT_PORT="$2"
+        if [[ $WAITFORIT_PORT == "" ]]; then break; fi
+        shift 2
+        ;;
+        --port=*)
+        WAITFORIT_PORT="${1#*=}"
+        shift 1
+        ;;
+        -t)
+        WAITFORIT_TIMEOUT="$2"
+        if [[ $WAITFORIT_TIMEOUT == "" ]]; then break; fi
+        shift 2
+        ;;
+        --timeout=*)
+        WAITFORIT_TIMEOUT="${1#*=}"
+        shift 1
+        ;;
+        --)
+        shift
+        WAITFORIT_CLI=("$@")
+        break
+        ;;
+        --help)
+        usage
+        ;;
+        *)
+        echoerr "Unknown argument: $1"
+        usage
+        ;;
+    esac
+done
+
+if [[ "$WAITFORIT_HOST" == "" || "$WAITFORIT_PORT" == "" ]]; then
+    echoerr "Error: you need to provide a host and port to test."
+    usage
+fi
+
+WAITFORIT_TIMEOUT=${WAITFORIT_TIMEOUT:-15}
+WAITFORIT_STRICT=${WAITFORIT_STRICT:-0}
+WAITFORIT_CHILD=${WAITFORIT_CHILD:-0}
+WAITFORIT_QUIET=${WAITFORIT_QUIET:-0}
+
+# Check to see if timeout is from busybox?
+WAITFORIT_TIMEOUT_PATH=$(type -p timeout)
+WAITFORIT_TIMEOUT_PATH=$(realpath $WAITFORIT_TIMEOUT_PATH 2>/dev/null || readlink -f $WAITFORIT_TIMEOUT_PATH)
+
+WAITFORIT_BUSYTIMEFLAG=""
+if [[ $WAITFORIT_TIMEOUT_PATH =~ "busybox" ]]; then
+    WAITFORIT_ISBUSY=1
+    # Check if busybox timeout uses -t flag
+    # (recent Alpine versions don't support -t anymore)
+    if timeout &>/dev/stdout | grep -q -e '-t '; then
+        WAITFORIT_BUSYTIMEFLAG="-t"
+    fi
+else
+    WAITFORIT_ISBUSY=0
+fi
+
+if [[ $WAITFORIT_CHILD -gt 0 ]]; then
+    wait_for
+    WAITFORIT_RESULT=$?
+    exit $WAITFORIT_RESULT
+else
+    if [[ $WAITFORIT_TIMEOUT -gt 0 ]]; then
+        wait_for_wrapper
+        WAITFORIT_RESULT=$?
+    else
+        wait_for
+        WAITFORIT_RESULT=$?
+    fi
+fi
+
+if [[ $WAITFORIT_CLI != "" ]]; then
+    if [[ $WAITFORIT_RESULT -ne 0 && $WAITFORIT_STRICT -eq 1 ]]; then
+        echoerr "$WAITFORIT_cmdname: strict mode, refusing to execute subprocess"
+        exit $WAITFORIT_RESULT
+    fi
+    exec "${WAITFORIT_CLI[@]}"
+else
+    exit $WAITFORIT_RESULT
+fi


Test patch:
diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml
new file mode 100644
index 0000000..0dbb421
--- /dev/null
+++ b/.github/workflows/tests.yml
@@ -0,0 +1,30 @@
+# This workflow will install Python dependencies and run tests with a variety of Python versions
+# For more information see: https://help.github.com/actions/language-and-framework-guides/using-python-with-github-actions
+
+name: tests
+
+on:
+  push:
+    branches: [main, develop]
+  pull_request:
+    branches: [main, develop]
+
+jobs:
+  build:
+    runs-on: ${{ matrix.os }}
+    strategy:
+      matrix:
+        python-version: ['3.7', '3.8', '3.9', '3.10']
+        os: [ubuntu-latest, macOS-latest]  # TODO: add windows-latest
+    steps:
+    - uses: actions/checkout@v2
+    - name: Set up Python
+      uses: actions/setup-python@v2
+      with:
+        python-version: ${{ matrix.python-version }}
+    - name: Install dependencies
+      run: |
+        pip install --upgrade pip
+        pip install -r requirements.txt
+    - name: Unittests
+      run: python -m unittest discover
diff --git a/tests/__init__.py b/tests/__init__.py
new file mode 100644
index 0000000..e69de29
diff --git a/tests/test_info.py b/tests/test_info.py
new file mode 100644
index 0000000..19c937f
--- /dev/null
+++ b/tests/test_info.py
@@ -0,0 +1,31 @@
+import sys  # TODO: remove it
+import unittest
+
+sys.path.append(".")
+from pydfs.__main__ import _main  # noqa: E402
+from pydfs.arg_parse import get_argparse  # noqa: E402
+
+
+class TestInfo(unittest.TestCase):
+    """
+    Class for testing pydfs --info and pydfs --version.
+    """
+
+    def setUp(self):
+        # https://stackoverflow.com/questions/18160078/how-do-you-write-tests-for-the-argparse-portion-of-a-python-module
+        self.parser = get_argparse()
+
+    def test_version(self):
+        args = self.parser.parse_args(["--version"])
+        exit_code = _main(args)
+        self.assertEqual(exit_code, 0)
+
+    # TODO: test different workflows
+    def test_info(self):
+        args = self.parser.parse_args(["--info"])
+        exit_code = _main(args)
+        self.assertEqual(exit_code, 0)
+
+
+if __name__ == "__main__":
+    unittest.main()
diff --git a/tests/test_init.py b/tests/test_init.py
new file mode 100644
index 0000000..943500e
--- /dev/null
+++ b/tests/test_init.py
@@ -0,0 +1,47 @@
+import sys  # TODO: remove it
+import threading
+import time
+import unittest
+
+sys.path.append(".")
+from pydfs.__main__ import _main  # noqa: E402
+from pydfs.arg_parse import get_argparse  # noqa: E402
+
+
+# TODO: test different workflows
+# TODO: maybe use flask-testing
+class TestInit(unittest.TestCase):
+    """
+    Class for testing pydfs init commands.
+    """
+
+    def setUp(self):
+        # https://stackoverflow.com/questions/18160078/how-do-you-write-tests-for-the-argparse-portion-of-a-python-module
+        self.parser = get_argparse()
+
+    # TODO: test different workflows
+    def test_init_master_and_slave(self):
+
+        # init master
+        args_master = self.parser.parse_args(["init", "master"])
+
+        # create a thread that will contain running server
+        # https://gist.github.com/prschmid/4643738
+        master_node_thread = threading.Thread(
+            target=_main,
+            args=(args_master,),
+            daemon=True,  # TODO: check correctness
+        )
+        master_node_thread.start()
+        time.sleep(5)  # TODO: fix hardcode
+
+        # init slave
+        args_slave = self.parser.parse_args(
+            ["init", "slave", "--master_ip", "127.0.0.1"]
+        )
+        exit_code = _main(args_slave)
+        self.assertEqual(exit_code, 0)
+
+
+if __name__ == "__main__":
+    unittest.main()
diff --git a/tests/test_utils.py b/tests/test_utils.py
new file mode 100644
index 0000000..37edfa0
--- /dev/null
+++ b/tests/test_utils.py
@@ -0,0 +1,20 @@
+import sys  # TODO: remove it
+import unittest
+
+sys.path.append(".")
+from pydfs.utils import ping  # noqa: E402
+
+
+class TestUtils(unittest.TestCase):
+    """
+    Class for testing utils.
+    """
+
+    def test_ping(self):
+        self.assertTrue(ping("localhost"))
+        self.assertTrue(ping("127.0.0.1"))
+        self.assertFalse(ping("127001"))
+
+
+if __name__ == "__main__":
+    unittest.main()