Package: luxdb
LuxDBThis is a simple database for multidimensional vectors. It basically provides persistance and connectivity with asyncio toHnswlib. The project contains the server and also a simple client.Still under development, there will be breaking changes and you will loose data if you only store it in this database. So don't use it for anything that you want to keep.TODOLanguage agnostic transport layer =>Performance?Rollbacks, transactions, ...FeaturesIn LuxDB you can store your vectors and query them for their nearest neighbors. Vectors are stored inindexesthat have a name, a dimension and a metric. After you created the index you
need to initialize it with a maximum size. (See the code below for some usage examples.) And luxdb/client.py or luxdb/sync_client.py for a list of all available operations.Changes are stored on the disk and can be used after the database is shutdown. This is achieved withZODB, each index is stored seperatly in aOOBTree. The store can be created with a path, in that case aFileStoragewill be created there. You can also provide a Storage in the constructor of the storage. For testing you can omitpathandstorage, in that case the data will be stored in memory only.Authentication and encryption happens withFernet, server and client need a shared secret in order to communicate. From this secret a key is derived, and this key is used to encrypt commands that are sent to the server. This guarantees, that only clients with the secret can execute commands on the server. The client needs the secret in the constructor, the server can either take it from the command line (--secret) or from an enviroment variable (LUXDB_SECRET).UsageStart the server, either with docker:dockerrun-p8484:8484registry.gitlab.com/sacovo/luxdb
dockerrun-p8484:8484registry.gitlab.com/sacovo/luxdb--loglevel=info--port8484--host0.0.0.0Or directly (after installing the dependencies in requirements.txt)./luxdb-server --port 8484 --loglevel debug path/to/storage.dbThe docker container stores the database in/data/so you can mount something there in order to store data.Look into thesnippetsto see some example configurations and code snippets that show how to use the client.You can then use the client to connect to the server and add or retrieve data.fromluxdb.clientimportconnect# Connect to the serverasyncwithconnect(host,port,secret)asclient:name='my-index'# Create an index for 12 dimensional vectorsawaitclient.create_index(name,'l2',12)# Initialize the indexawaitclient.init_index(name,max_elements)# Add some datadata=np.float32(np.random.random((1000,12)))labels=np.arange(1000)awaitclient.add_items(name,data,labels)# Search the nearest neighbors of data[0]found,distances=awaitclient.query_index(name,data[0],k=5)# Or the nearest neighbors of all elementsfound,distances=awaitclient.query_index(name,data,k=2)# You can resize the index after it was createawaitclient.resize_index(name,new_size)# Get all ids that are storedids=client.get_ids(name)# Get the element with specific idselements=client.get_elements(name,[1,3,5])For more usage examples you can check the tests intests/test_client.pyEnvironment variablesNameDescriptonDefaultLUXDB_SECRETSecret used to encrypt and authenticate communication with clients.''LUXBD_SALTSalt used in key derivation from secret. Needs to be the same on client and server.'wYfJIy4Nx1hPcxiljwg'KDF_ITERATIONSIterations to use in key derivation1 << 18FERNET_TTLTime in seconds that messages are valid after they are encrypted.60Project structureThe project consists of a wrapper around a collection ofhnswlib.Indexobjects, a server that performs modifications and lookups and a client. Communication between the server and the client happen through Command objects.DevelopmentTestsThe iteration count of the key derivation function can be changed through the environment variableKDF_ITERATIONS. To speed up tests you can set it to a low value. Don't set it to a low value in any other context!