Package: detti-db-server
Lightweight Json based key-value DB and/or server.IntroductionWhy:Small, Fast, Efficient, Easy, FunnyImportant:The Detti DB handles string key, and many value types.The Detti Server handles only string type as key and value (Dict[str, str]).LinksPyPiGitHubPePyExample DB usage fileExample Server usage fileEasy install with PIP>>>pipinstalldetti-db-serverCreate environment and use virtual environmentCreate a project folder and a venv folder within:>>>mkdir-pdetti
>>>cddettiClone the source code from Git:>>>gitclonehttps://github.com/milanbalazs/detti_db_server.git
>>>cddetti_db_serverCreate and activate the virtual environment:>>>python3-mvenvvenv
>>>sourcevenv/bin/activateInstall the required modules from requirements.txt:>>>pipinstall-rrequirements.txt
OR
>>>pythonsetup.pyinstallSystemRequirements:InterpreterPython3.6.x <Python packagesThey can find in the requirements.txt file (pipreqs).The required packages can be installed withpip.Tested system:Interpreter:Python 3.6.9Operation system:Linux Mint 19.1 TessaBash:4.4.20(1)-releaseCurlcurl 7.58.0 (x86_64-pc-linux-gnu)detti DBConfigurationThe default config isdetti_conf.ini.It is in the root folder (next to thedetti_db.pyfile).The configuration file is a standardINI file format.Default config:[DETTI_DB]# Path of the DB file. Recommended to define full path.path_of_db=test.db# Maximum length of the keys in DB (Avoid memory overload).len_of_key=100# Maximum length of the values in DB (Avoid memory overload).len_of_val=100# Level of the logger. Possible: DEBUG, INFO, WARNING, ERROR, CRITICAL# IMPORTANT: The generated log file will contain all log level messages!log_level=WARNINGNote:The defaultdetti_conf.inifile contains more sections but only theDETTI_DBsection is
related to the DB. Other sections are not used in case of DB. It is not problem if other
(default) sections are not in the config file.Use own config file:There isconfig_fileargument (str) ofDettiDBclass which set the used config file path.Example:detti_db=DettiDB(config_file="own_config.ini")Owerwrite config file parameters:You can overwrite all config file parameters as instance variables.Example:detti_db=DettiDB(len_of_val=50)# The parameter is set to 100 in config file but it will be overwrite to 50.UsageImportDettiDBclass from thedetti_dbmodule:fromdetti_dbimportDettiDBCreate instance fromDettiDBclass (Using the default init values):detti_db=DettiDB()Supported types in DB:KeystrValuestrfloatintlistdict:arrow_right:Setters:Key:str, Value:str:With dictionary like solution:detti_db["test_str_key"]="test_val"# Set the value as "test_val" (str):Return:Trueif the setting is successful elseFalseNote:If you want to set a non-supported value type, you will get a warning message, and
the value won't be store to DB. Eg.:detti_db["test_key"]=(1,2,3)# Try to store Tuple type>>[detti_db.py][WARNING]Thegettingvaluetypeisnotsupported(<class'tuple'>). The value won'tbestored.With method usage:detti_db.set("test_str_key_2","test_val_2")# Set the value as "test_val_2" (str):Return:Trueif the setting is successful elseFalseNote:Theset()method tries to cast the getting value to string. Eg.:detti_db.set("test_str_key_3",123)# Set the value as "123" (str)Key:str, Value:int:With dictionary like solution:detti_db["test_int_key"]=8# Set the value as 8 (int):Return:Trueif the setting is successful elseFalseWith method usage:detti_db.set_int("test_int_key_2",9)# Set the value as 9 (int):Return:Trueif the setting is successful elseFalseNote:Theset_int()method tries to cast the getting value to integer. Eg.:detti_db.set_int("test_int_key_3",123.123)# Set the value as 123 (int)detti_db.set_int("test_int_key_4","888")# Set the value as 888 (int)Key:str, Value:float:With dictionary like solution:detti_db["test_float_key"]=8.8# Set the value as 8.8 (float):Return:Trueif the setting is successful elseFalseWith method usage:detti_db.set_float("test_float_key_2",9.9)# Set the value as 9.9 (float):Return:Trueif the setting is successful elseFalseNote:Theset_float()method tries to cast the getting value to float. Eg.:detti_db.set_float("test_float_key_3",123)# Set the value as 123.0 (float)detti_db.set_float("test_float_key_4","888.888")# Set the value as 888.888 (float)Key:str, Value:List[Any]:With dictionary like solution:detti_db["test_list_key"]=["a",1]# Set the value as ["a", 1] (list):Return:Trueif the setting is successful elseFalseWith method usage:detti_db.set_list("test_list_key_2",["a",1])# Set the value as ["a", 1] (list):Return:Trueif the setting is successful elseFalseNote:Theset_list()method tries to cast the getting value to float. Eg.:detti_db.set_list("test_list_key_3",("a",2))# Set the value as ["a", 2] (list)detti_db.set_list("test_list_key_4","abc")# Set the value as ["a", "b", "c"] (list))AppendAnytolist:detti_db.append_list("key","value")Example:detti_db.set_list("test_list",["a",1])detti_db.append_list("test_list",666)print(detti_db["test_list"])>>>["a",1,666]:Return:Trueif the appending is successful elseFalseNote:The return value isFalseif you try to append a new element to a not existing key in DB.The return value isFalseif you try to append a new element to a key which value is not list type.Key:str, Value:dict:With dictionary like solution:detti_db["test_dict_key"]={"a":1}# Set the value as {"a": 1} (dict):Return:Trueif the setting is successful elseFalseWith method usage:detti_db.set_dict("test_dict_key_2",{"a":1})# Set the value as {"a": 1} (dict):arrow_right:Getters:Get elementWith dictionary like solution:detti_db["test_key"]# Return: "test_val":Return: The requested value if it exists in DB elseNoneWith method usage:detti_db.get("test_key_2")# Return: "test_val_2":Return: The requested value if it exists in DB elseNoneNote:The above getter solutions can return any types.Using default value in get:detti_db.get("not_exist_key",default_value=5)# Return: 5 (Due to the "not_exist_key" key is not in DB.):Return: The requested value if it exists in DB else the set default value.Note:The above getter solutions can return any types.Any type can be set as default parameter.Get all elements:Theget_all()method provides the all key-value pairs from DB in Json format. It supports any types.detti_db.get_all()# Return: {'test_key': 'test_val', 'test_key_2': 'test_val_2'}:Return: The requested items in dict if any exists in DB else empty dictCheck if key exists in DB:Theis_exist()method returnsTrueif the key is in DB elseFalse.detti_db.is_exist("elem_key"):Return:Trueif the key is in DB elseFalse.Deletion:With dictionary like solution:deldetti_db["test_key"]:Return:Trueif the setting is successful elseFalseWith method usage:detti_db.delete("test_key_2")# Return True if it's success else False:Return:Trueif the setting is successful elseFalseSearching:Search keys based on provided prefix (Returning aDict[str, str]):detti_db.search_keys_in_db("my_")# Return: {'my_test_key_4': 'test_val_4'}:Return: The requested items in dict if any exists in DB else empty dictSearch values based on provided prefix (Returning aDict[str, str]):detti_db.search_values_in_db("my_")# Return: {'test_key_4': 'my_test_val_4'}:Return: The requested items in dict if any exists in DB else empty dictGet size of DB:detti_db.size_of_db()# Return: 666 (int):Return: Size of the DB is bytes (int).Get all keys of DB:detti_db.get_all_keys()# Return: ["test_key", "test"key_2"] (List[str]):Return: All keys of the DB (List[str]).Dump the current loaded DB to Json file:detti_db.dump_to_json("dump.json")Possible parameters:file_path: Path of the destination file.force: If it is given the existing file will be overwritten.Default: Falsepermissions: Set the permission of file. It is true for overwritten files! Octal!Default: 0o600Eg.:detti_db.dump_to_json(file_path="dump.json",force=True,permissions=0o664)Complete example code (With not existing DB):importosimportsys# Get the path of the directory of the current file.PATH_OF_FILE_DIR:str=os.path.realpath(os.path.dirname(__file__))# Append the path of the tools folder to find modules.sys.path.append(os.path.join(PATH_OF_FILE_DIR,".."))fromdetti_dbimportDettiDB# noqa: E40detti_db:DettiDB=DettiDB()# settersdetti_db["test_key"]="test_val"detti_db.set("test_key_2","test_val_2")detti_db.set_int("test_int_key",123)detti_db.set_float("test_float_key",123.123)detti_db.set_list("test_list_key",["a",1])detti_db.set_dict("test_dict_key",{"b":2})# gettersprint("test_key ->{}".format(detti_db["test_key"]))print("test_int_key ->{}".format(detti_db.get("test_int_key")))print("test_list_key ->{}".format(detti_db["test_list_key"]))print("test_dict_key ->{}".format(detti_db["test_dict_key"]))print("All content:{}".format(detti_db.get_all()))print("Number of elements in DB:{}".format(detti_db.get_number_of_elements()))print("Size of DB:{}".format(detti_db.size_of_db()))print("All keys in DB:{}".format(detti_db.get_all_keys()))# deletionsdeldetti_db["test_key"]detti_db.delete("test_key_2")# Set some new items for searchingdetti_db["test_key_3"]="my_test_val_3"detti_db["my_test_key_4"]="test_val_4"# Searchingprint("'my_' key prefixes ->{}".format(detti_db.search_keys_in_db("my_")))print("'my_' value prefixes ->{}".format(detti_db.search_values_in_db("my_")))Output:>>>python3examples/db_example_1.py
test_key->test_val
test_int_key->123test_list_key->['a',1]test_dict_key->{'b':2}Allcontent:{'test_key':'test_val','test_key_2':'test_val_2','test_int_key':123,'test_float_key':123.123,'test_list_key':['a',1],'test_dict_key':{'b':2}}NumberofelementsinDB:6SizeofDB:216AllkeysinDB:['test_key','test_key_2','test_int_key','test_float_key','test_list_key','test_dict_key']'my_'keyprefixes->{'my_test_key_4':'test_val_4'}'my_'valueprefixes->{'test_key_3':'my_test_val_3'}detti Server (with RESTful API)ConfigurationThe default config isdetti_conf.ini.It is in the root folder (next to thedetti_db.pyfile).The configuration file is a standardINI file format.Default config:[DETTI_DB]# Path of the DB file. Recommended to define full path.path_of_db=test.db# Maximum length of the keys in DB (Avoid memory overload).len_of_key=100# Maximum length of the values in DB (Avoid memory overload).len_of_val=100# Level of the logger. Possible: DEBUG, INFO, WARNING, ERROR, CRITICAL# IMPORTANT: The generated log file will contain all log level messages!log_level=WARNING[SERVER]host=localhostport=5000debug=True# Setting the request limits in different unites. The most strict will be used!sec_limit=5min_limit=300hour_limit=18000day_limit=432000# IMPORTANT# If you set the user and password parameter the DB will be accessed with JWT Token!user=password=Note:The defaultdetti_conf.inifile contains more sections but theSERVERandDETTI_DBsections are related (and mandatory) to the Server running.
Other sections are not used in case of DB. It is not problem if other
(default) sections are not in the config file.Use own config file:There isconfig_fileCLI argument (str).Force to use a config file. Default:detti_conf.ini(In root folder)Example:>>> python3 detti_server --config_file my_own_config.iniUsageStart the server:>>>python3detti_server.pyOutput in case of successful starting (with the default config):* Serving Flask app "detti_server" (lazy loading)
 * Environment: production
   WARNING: This is a development server. Do not use it in a production deployment.
   Use a production WSGI server instead.
 * Debug mode: on
 * Running on http://localhost:5000/ (Press CTRL+C to quit)
 * Restarting with stat
 * Debugger is active!
 * Debugger PIN: 217-599-780Test server statusThe server status can be checked to send a GET request to/pingend-point of the server.Response if the server is up and running:Curl:>>>curlhttp://localhost:5000/ping
>"PONG"Python:importrequestsrequests.get("http://localhost:5000/ping")# Return: "PONG"Response if the server is down (Status code: 7):>>>curlhttp://localhost:5000/ping
>curl:(7)Failedtoconnecttolocalhostport5000:KapcsolatelutasítvaEnd-points (RESTful APIs)/get/<string:db_key>Providing the key-value pair based on getting key. The status code is 201 in case of error.Curl:>>>curlhttp://localhost:5000/set-d"exist=value_of_exist_key"-XPUT
>{"STATUS":"OK"}>>>curlhttp://localhost:5000/get/exist
>{"exist":"value_of_exist_key"}>>>curlhttp://localhost:5000/get/doesnt_exist
>{"doesnt_exist":"The key doesn't exist in DB."}Python:importrequests# Set an element in the DBput_resp=requests.put("http://localhost:5000/set",data={"exist":"value_of_exist_key"})print(put_resp.json())# Return: {"STATUS": "OK"}# Get the element from DBresp=requests.get("http://localhost:5000/get/exist")print(resp.json())# Return: {"exist": "value_of_exist_key"}/setSetting/updating key-value pair in the DB.Curl:>>>curlhttp://localhost:5000/set-d"test_key=test_val"-XPUT
>{"STATUS":"OK"}>>>curlhttp://localhost:5000/get/test_key
>{"test_key":"test_val"}Python:importrequests# Set an element in the DBput_resp=requests.put("http://localhost:5000/set",data={"test_key":"test_val"})print(put_resp.json())# Return: {"STATUS": "OK"}# Get the element from DBresp=requests.get("http://localhost:5000/get/test_key")print(resp.json())# Return: {"test_key": "test_val"}/search_key/<string:key_prefix>Searching keys in the DB based on provided key prefix.Curl:>>>curlhttp://localhost:5000/set-d"test_key=test_val"-XPUT
>{"STATUS":"OK"}>>>curlhttp://localhost:5000/set-d"prod_key_1=prod_val_1"-XPUT
>{"STATUS":"OK"}>>>curlhttp://localhost:5000/set-d"prod_key_2=prod_val_2"-XPUT
>{"STATUS":"OK"}>>>curlhttp://localhost:5000/search_key/prod_
>{"prod_key_1":"prod_val_1","prod_key_2":"prod_val_2"}>>>curlhttp://localhost:5000/search_key/not_exist
>{"not_exist":"Cannot find keys for prefix"}Python:importrequests# Set some elements in DB.requests.put("http://localhost:5000/set",data={"dev_data":"dev"})requests.put("http://localhost:5000/set",data={"prod_key_1":"prod_val_1"})requests.put("http://localhost:5000/set",data={"prod_key_2":"prod_val_2"})# Get the elements based on provided KEY prefix.resp=requests.get("http://localhost:5000/search_key/prod_")print(resp.json())# Return: {"prod_key_1": "prod_val_1", "prod_key_2": "prod_val_2"}# Try a KEY prefix which is not contained in keys.resp=requests.get("http://localhost:5000/search_key/nonono")print(resp.json())# Return: {"nonono": "Cannot find keys for prefix"}/search_val/<string:value_prefix>Searching values in the DB based on provided value prefix.Curl:>>>curlhttp://localhost:5000/set-d"test_key=test_val"-XPUT
>{"STATUS":"OK"}>>>curlhttp://localhost:5000/set-d"prod_key_1=prod_val_1"-XPUT
>{"STATUS":"OK"}>>>curlhttp://localhost:5000/set-d"prod_key_2=prod_val_2"-XPUT
>{"STATUS":"OK"}>>>curlhttp://localhost:5000/search_val/prod_
>{"prod_key_1":"prod_val_1","prod_key_2":"prod_val_2"}>>>curlhttp://localhost:5000/search_val/not_exist
>{"not_exist":"Cannot find values for prefix"}Python:importrequests# Set some elements in DB.requests.put("http://localhost:5000/set",data={"dev_data":"dev"})requests.put("http://localhost:5000/set",data={"prod_key_1":"prod_val_1"})requests.put("http://localhost:5000/set",data={"prod_key_2":"prod_val_2"})# Get the elements based on provided VALUE prefix.resp=requests.get("http://localhost:5000/search_val/prod_")print(resp.json())# Return: {"prod_key_1": "prod_val_1", "prod_key_2": "prod_val_2"}# Try a VALUE prefix which is not contained in keys.resp=requests.get("http://localhost:5000/search_val/nonono")print(resp.json())# Return: {"nonono": "Cannot find values for prefix"}/delete/<string:db_key>Deleting an element from the DB.Curl:>>curlhttp://localhost:5000/set-d"test_key=test_val"-XPUT
>{"STATUS":"OK"}>>curlhttp://localhost:5000/get/test_key
>{"test_key":"test_val"}>>curlhttp://localhost:5000/delete/test_key-XDELETE
>{"STATUS":"OK"}>>curlhttp://localhost:5000/get/test_key
>{"test_key":"The key doesn't exist in DB."}Python:importrequests# Set some elements in DB.requests.put("http://localhost:5000/set",data={"to_be_deleted":"dummy"})# Delete the created itemdel_resp=requests.delete("http://localhost:5000/delete/to_be_deleted")print(del_resp.json())# Return: {"STATUS": "OK"}# Try to get the deleted itemresp=requests.get("http://localhost:5000/get/to_be_deleted")print(resp.status_code)# Return: 201print(resp.json())# Return: {"to_be_deleted": "The key doesn't exist in DB."}/pingChecking if the server is running.Success example:>>>curlhttp://localhost:5000/ping
>"PONG"Failed example (Status code: 7):>>>curlhttp://localhost:5000/ping
>curl:(7)Failedtoconnecttolocalhostport5000:Kapcsolatelutasítva/getallProviding all elements from the DB. {key: value, key: value}Curl:>>>curlhttp://localhost:5000/set-d"test_key=test_val"-XPUT
>{"STATUS":"OK"}>>>curlhttp://localhost:5000/set-d"test_key_1=test_val_1"-XPUT
>{"STATUS":"OK"}>>>curlhttp://localhost:5000/getall
>{"test_key":"test_val","test_key_1":"test_val_1"}Python:importrequests# Set some elements in DB.requests.put("http://localhost:5000/set",data={"get_all_1":"dummy"})requests.put("http://localhost:5000/set",data={"get_all_2":"dummy"})# Getting all elements from DB.resp=requests.get("http://localhost:5000/getall")print(resp.json())# Return: {"get_all_1": "dummy", "get_all_2": "dummy"}JWT AuthenticationOfficial page of JWT:JSON Web TokensImportant:The JWT authentication is not active with the default configuration.The "user" and "password" parameters have to be set in the configuration file
to activate the JWT Authentication.For example:user=test_userpassword=test_passwordGet the token from the server:>>>curl-i-XPOST-H"Content-Type: application/json"-d'{"username":"test_user","password":"test_password"}'http://localhost:5000/auth
>{"access_token":"eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJleHAiOjE2MTE4...}Use the JWT authentication for APIs:>>>curl-H"Authorization: jwt eyJ0eXAiOiJKV..."http://localhost:5000/get/exist
>{"exist":"exist_val"}If the token is not used, the APIs provide error message with 401 status code:>>>curlhttp://localhost:5000/get/exist
>{"description":"Request does not contain an access token","error":"Authorization Required","status_code":401}Production lineCurrently, the production line support is not implemented in this repo (But it is in the road-map)!
You can run the server on the production line with Nginx and Gunicorn.Tutorial:How To Serve Flask Applications with Gunicorn and Nginx on Ubuntu 18.04FutureIntroduce the multithreading/multiprocessing in searching methodsIn case of big data the multithreading/multiprocessing can reduce the execution timeAdd support for more data types on server side.Change log1.2.1Pass the**optionsparameter inrun_server()function.The options to be forwarded to the underlying Werkzeug server.
See :func:werkzeug.serving.run_simplefor more information.Add thedump_to_json()method to dump the current DB to Json file.AddPython BlackandFlake8pre-commit hooks to repo for better contributions.Fix TypeError in case ofdicttype setting.Add Cross Origin Resource Sharing (CORS) for server to make cross-origin AJAX possible.1.1.1Addget_number_of_elements()method to get number of elements of DB.Addget_all_keys()method to get all keys of DB.Addset_dict()method asdicttype setter.Addrun_server()function to server part for better integration.Now thedetti_server.pyfile can be imported and configurable before starting the server.Eg.:import detti_server; detti_server.run_server()1.1.0Get size of DB withsize_of_db()method.Implement__contains__magic method (Eg.:"a" in detti_db).Adddefault_valueoption togetmethod (Eg.:detti_db.get("not_exist_key", default_value=5))All parameters from config file can be overwritten as__init__argument (Eg.:detti_db = DettiDB(len_of_val=50))